system

Abstract

Provide a system. A means for analyzing user emotions in real time; a means for customizing the virtual shopping experience based on the results of that analysis; and a means of collecting user feedback and updating the analysis results; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Online shopping requires the ability to analyze user emotions in real time and personalize the shopping experience based on that information. However, current systems struggle to properly understand user emotions and customize the shopping experience in real time. Furthermore, they lack an effective way to adapt the system based on user feedback. Therefore, a new solution that integrates emotion analysis and shopping experience customization is needed to improve user satisfaction. [Means for solving the problem]

[0005] This invention solves the above-mentioned problems by providing a system that analyzes a user's emotions in real time and customizes a virtual shopping experience based on the analysis results. Specifically, the system includes a terminal for capturing the user's facial expressions and voice data and a communication means for transmitting the data to a server. The server then analyzes the user's emotions based on the received data and generates product suggestions and special offers based on the analysis results. The system further displays the generated suggestions and offers on the user's terminal, collects user feedback, and updates the analysis results to improve the accuracy of future suggestions.

[0006] "Users" are people who shop online.

[0007] "Emotion analysis" is the process of analyzing a user's facial expressions and voice data to identify their emotions in real time.

[0008] A "virtual shopping experience" is a virtual environment where users can enjoy shopping online just as they would in a real store.

[0009] "Customization" means providing a different shopping experience for each user based on the analysis results.

[0010] "Feedback" refers to the reactions and actions that users take in response to suggestions and offers.

[0011] A "terminal" is a device operated by a user (such as a PC, smartphone, or VR headset).

[0012] "Capture" refers to capturing a user's facial expressions and voice data using a camera and microphone.

[0013] A "server" is a computer system that has an emotion analysis AI and a database and processes and stores data.

[0014] "Communication means" refers to the network infrastructure that allows the terminal to transmit captured data to the server.

[0015] "Product suggestions" are product information recommended based on the user's interests and emotions.

[0016] "Special Offers" refers to providing users with special promotions and discount information.

[0017] "Updating analysis results" is the process of modifying the emotion analysis model based on user feedback to improve the accuracy of future suggestions. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0019] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0022] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0023] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0024] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0026] [First embodiment]

[0027] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0028] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0031] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0034] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0038] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0039] This invention relates to a system that analyzes a user's emotions in real time and customizes a virtual shopping experience based on the analysis results. Specifically, the system includes a terminal for capturing the user's facial expression and voice data and a communication means for transmitting the data to a server. The server analyzes the user's emotions based on the received data and generates product suggestions and special offers based on the analysis results. The generated suggestions and offers are displayed on the user's terminal, and the analysis results are updated based on the user's feedback to improve the accuracy of future suggestions.

[0040] System configuration and processing flow

[0041] composition

[0042] The system consists of the following main components:

[0043] 1. Server

[0044] Emotion analysis AI

[0045] Database

[0046] 2. Terminal

[0047] Camera and microphone

[0048] Data Transmission Module

[0049] Virtual shopping environment interface

[0050] 3. Means of communication

[0051] Network Infrastructure

[0052] Program processing

[0053] 1. Initialize the server

[0054] The server loads the sentiment analysis model and the virtual shopping environment, which are generated from historical user data.

[0055] 2. Initialize the device

[0056] The device checks the camera and microphone and displays the message "Do you want to start shopping online?" to the user.

[0057] 3. Collection of User Data

[0058] When a user allows access to the camera and microphone, the device captures the user's facial expressions and voice data and sends it to the server.

[0059] 4. Emotion analysis

[0060] The server analyzes the facial expression and voice data sent from the device and infers the user's emotions in real time. The analysis results are recorded in a database.

[0061] 5. Customize your experience

[0062] The server generates personalized product suggestions and special offers for the user based on the results of sentiment analysis and past shopping history.

[0063] The generated proposals and offers are sent to the terminal and displayed on the user's screen.

[0064] 6. User Feedback

[0065] The user responds to the suggested product or offer (e.g., clicks, tries on, purchases).

[0066] The terminal records this response and sends it to the server as feedback data.

[0067] 7. Update analysis results

[0068] The server updates the sentiment analysis model based on the user's feedback data to improve the accuracy of suggestions for the next shopping session.

[0069] Specific examples

[0070] Scenario: User wants to try on a virtual item

[0071] 1. Initialize the server

[0072] The server loads the avatar and clothing data for virtual try-on.

[0073] 2. Initialize the device

[0074] The device displays a virtual fitting room interface to users wearing a VR headset.

[0075] 3. Collection of User Data

[0076] While the user looks at themselves in the mirror, a camera and microphone capture their facial expressions and voice, sending the data to a server.

[0077] 4. Emotion analysis

[0078] The server analyzes the user's emotions (e.g., surprise, satisfaction) in real time based on facial expression data and voice data.

[0079] 5. Customize your experience

[0080] If the user is satisfied, the server generates data for suggesting clothes of a similar style and transmits it to the terminal.

[0081] The terminal displays these suggestions to the user.

[0082] 6. User Feedback

[0083] Users try on the suggested clothes and respond by rejecting them or purchasing them.

[0084] The device records this response and sends it to the server.

[0085] 7. Update analysis results

[0086] The server integrates user feedback to update the analytical model, improving the accuracy of future proposals.

[0087] In this way, users can enjoy a real-time, personalized online shopping experience, and the system continuously learns and optimizes the user experience.

[0088] The processing flow will be explained below.

[0089] Step 1:

[0090] The server loads the sentiment analysis model and the virtual shopping environment, including the machine learning models and connections to the necessary databases.

[0091] Step 2:

[0092] The device checks the camera and microphone and displays the message "Do you want to start shopping online?" The user allows the use of the camera and microphone.

[0093] Step 3:

[0094] The device periodically captures the user's facial expressions with a camera and collects audio data with a microphone, which is then sent to a server every few seconds.

[0095] Step 4:

[0096] The server analyzes the received facial expression and voice data to infer the user's emotions (e.g., joy, excitement, dissatisfaction) in real time. The analysis results are recorded in a database.

[0097] Step 5:

[0098] The server uses the analysis results and past shopping history to generate personalized product suggestions and special offers for the user, using machine learning algorithms.

[0099] Step 6:

[0100] The server sends the generated proposals and offers to the terminal, which displays this information on the user's screen. The user then confirms the proposed products and offers.

[0101] Step 7:

[0102] The user's reactions to the proposed products and offers (e.g., clicking, trying on, purchasing) are recorded by the device and transmitted to the server.

[0103] Step 8:

[0104] The server analyzes the user's reaction data sent from the terminal and creates a new database entry as feedback data.

[0105] Step 9:

[0106] The server updates the sentiment analysis model and product recommendation algorithm based on the feedback data, improving the accuracy of future recommendations.

[0107] Step 10:

[0108] The device will continue to present new sentiment-based suggestions and offers to the user, and the whole process will be repeated in real time to optimize the user experience.

[0109] Example 1

[0110] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0111] In conventional online shopping, it was difficult to analyze user emotions in real time and personalize the shopping experience based on that. Furthermore, there was a lack of a mechanism for collecting user feedback and updating the analysis results, which meant that product recommendations did not adequately reflect the user's interests and satisfaction. This resulted in a poor user experience and an inability to fully stimulate purchasing motivation.

[0112] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0113] In this invention, the server includes means for loading and initializing an emotion analysis model, means for collecting facial expression and voice data of a user, means for transmitting the collected data to the server, means for analyzing the user's emotions in real time based on the data received by the server, means for recording the analysis results in a database, means for generating personalized product suggestions and special offers based on the emotion analysis results and past shopping history, means for displaying the generated suggestions and offers on the user terminal, means for collecting user feedback and transmitting the feedback data to the server, and means for updating the emotion analysis model based on the feedback data. This makes it possible to analyze the user's emotions in real time and provide a highly personalized shopping experience that reflects the results.

[0114] An "emotion analysis model" is an algorithm or program that analyzes a user's facial expressions and voice data to infer their emotional state.

[0115] A "server" is a computer system that processes and stores data and provides that data to other computers or devices over a network.

[0116] "User's facial expression and voice data" refers to digital data of human facial movements and voice signals captured using a camera and microphone.

[0117] A "database" is a structured collection of data for efficiently storing, retrieving, and updating information.

[0118] "Product suggestion" refers to the act of introducing products that are likely to interest a user based on the results of an emotional analysis of the user and their past shopping history, and the data used to do so.

[0119] "Special Offers" are commercial offers such as special prices or promotions based on the user's interests.

[0120] "Feedback data" refers to data based on user behavior and reactions that indicate interest in and intent to purchase product suggestions and special offers.

[0121] "Personalization" means customizing product proposals and services to suit the characteristics and preferences of individual users.

[0122] "Real-time" is a concept that refers to processing and reactions occurring almost simultaneously with the collection of data.

[0123] This invention relates to a system that analyzes a user's emotions in real time and customizes a virtual shopping experience based on the analysis results. Specifically, the system includes a terminal for capturing the user's facial expression and voice data and a communication means for transmitting the data to a server. The server analyzes the user's emotions based on the received data and generates product suggestions and special offers based on the analysis results. The generated suggestions and offers are displayed on the user's terminal, and the analysis results are updated based on user feedback, thereby improving the accuracy of future suggestions.

[0124] The main components of the system are as follows:

[0125] 1. Server

[0126] Sentiment analysis model (e.g., AI model using TENSORFLOW (registered trademark))

[0127] Database (e.g. MySQL (registered trademark))

[0128] 2. Terminal

[0129] Camera and microphone (e.g., a standard webcam or microphone)

[0130] Data transmission module (e.g. Wi-Fi or Bluetooth)

[0131] Virtual shopping environment interface (e.g., built with Unity)

[0132] 3. Means of communication

[0133] Network infrastructure (e.g., 5G, fiber optic connections)

[0134] Specifically, the server first loads the emotion analysis model and virtual shopping environment. This requires initializing an AI model using TensorFlow and establishing a connection to a database of products taken from past user data. The device then diagnoses the camera and microphone, checks their operation, and displays a message to the user asking, "Do you want to start online shopping?" If the user allows access to the camera and microphone, the device captures the user's facial expressions and voice data and sends it to the server.

[0135] The server analyzes the facial expression and voice data sent from the device and infers the user's emotions in real time. The analysis results are recorded in a database. The server then generates personalized product suggestions and special offers for the user based on the emotion analysis results and past shopping history. The generated suggestions and offers are sent to the device and displayed on the user's screen. When the user responds to the suggested products or offers (e.g., clicking, trying them on, purchasing), the device records this response and sends it to the server as feedback data. Finally, the server updates the emotion analysis model based on the user's feedback data to improve the accuracy of suggestions for the next shopping session.

[0136] As a specific example, consider a user using a virtual fitting service. The server loads the virtual fitting avatar and clothing data, and the device displays a virtual fitting room interface to the user wearing a VR headset. While the user looks at themselves in the mirror, a camera and microphone capture their facial expressions and voice, and send the data to the server. The server analyzes the user's emotions (e.g., surprise, satisfaction) in real time based on the facial expression and voice data. The server then suggests more similar styles of clothing based on the emotion analysis results, and the device displays these suggestions to the user. When the user tries on the suggested clothing and responds by rejecting it or purchasing it, the device records this response and sends it to the server. The server integrates the user's feedback to update the analysis model, improving the accuracy of future suggestions.

[0137] Examples of prompt statements

[0138] Prompt: "Analyze the user's facial expressions and voice in a virtual fitting room to suggest outfits that match their emotions."

[0139] Input data example: Facial expression data [image file], voice data ["I like this outfit!"]

[0140] In this way, users can enjoy a real-time, personalized online shopping experience, and the system can continuously learn and optimize the user experience.

[0141] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0142] Step 1:

[0143] Initializing the Server

[0144] The server loads the sentiment analysis model and the virtual shopping environment. Specifically, it first loads the sentiment analysis model parameters into memory using TensorFlow. Next, it establishes a connection to a product database (e.g., MySQL) based on past user data and loads the necessary product information into memory. This allows the server to perform sentiment analysis and product suggestions in real time.

[0145] Input: TensorFlow model parameter file, product database connection information

[0146] Output: Initialized sentiment analysis model, product data loaded into memory.

[0147] Step 2:

[0148] Initializing the device

[0149] The device diagnoses the camera and microphone to ensure they are working properly. It then displays a message to the user asking, "Do you want to start online shopping?" If the user responds, the device prepares to send the input to the server.

[0150] Specific actions include checking the camera video feed and checking the volume level of the microphone input.

[0151] Input: User actions, device camera and microphone diagnostic results

[0152] Output: User camera and microphone permissions, confirmation to start shopping

[0153] Step 3:

[0154] User Data Collection

[0155] Once the user allows the use of the camera and microphone, the device will activate them and continuously capture the user's facial expressions and voice data, which will be transmitted to the server in real time. The camera will capture high-resolution images, and the microphone will record clear audio.

[0156] Input: Video data from camera, audio data from microphone

[0157] Output: Facial expression data and voice data sent to the server

[0158] Step 4:

[0159] Emotion analysis

[0160] The server receives and analyzes the facial expression and voice data sent from the device. It passes the facial expression images and voice data as input to an emotion analysis model, which then infers the user's emotions in real time. The analysis results are then recorded in a database.

[0161] Input: Emotion analysis model loaded from database, received facial expression data and speech data

[0162] Output: User's emotional state, analysis results recorded in a database

[0163] Step 5:

[0164] Customize your experience

[0165] The server generates personalized product suggestions and special offers for the user based on the results of sentiment analysis and past shopping history. The generated suggestions and offers are sent in JSON format to the device, which then displays them on the user's screen.

[0166] Input: Sentiment analysis results, past shopping history, product database

[0167] Output: Generated product suggestions, special offers, and display data sent to the device

[0168] Step 6:

[0169] User Feedback

[0170] When users interact with the proposed products or offers (e.g., click, try on, purchase), the device records these interactions, and the recorded feedback data is sent to the server in real time.

[0171] Input: User interaction data

[0172] Output: Feedback data sent to the server

[0173] Step 7:

[0174] Update analysis results

[0175] The server receives user feedback data and updates the sentiment analysis model based on that data. This update improves the accuracy of suggestions for the next shopping session by optimizing the model parameters and algorithms.

[0176] Input: Feedback data, sentiment analysis model

[0177] Output: Updated sentiment analysis model, improving the accuracy of the next proposal

[0178] Through these steps, the system can analyze user sentiment in real time, generate and present product suggestions and special offers tailored to each user, and continuously learn to optimize the user experience.

[0179] (Application example 1)

[0180] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0181] Conventional virtual shopping experiences have had the challenge of being difficult to flexibly respond to individual users' emotions and preferences. In particular, there has been a lack of systems that can analyze a user's emotional state in real time and make product recommendations based on the results. This means that users are often left feeling dissatisfied because they are rarely presented with products that suit them at the right time. Furthermore, there has been a lack of mechanisms for effectively utilizing user feedback to improve the shopping experience.

[0182] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0183] In this invention, the server includes means for analyzing user emotions in real time, means for collecting user feedback and updating the analysis results, and means for capturing the user's facial expression and voice data using the smart glasses, which enables the generation of personalized product suggestions and special offers based on the user's emotions and feedback and displaying these suggestions on the display of the smart glasses, thereby providing the user with a more comfortable and efficient shopping experience.

[0184] "Means for analyzing a user's emotions in real time" refers to a technology that instantly determines the user's emotional state at any given time based on the facial expressions and voice data they show.

[0185] "Means for customizing virtual shopping experiences" refers to technology that individually adjusts and optimizes the shopping experience based on user sentiment analysis and feedback.

[0186] The "means of collecting user feedback and updating analysis results" refers to a technology that continuously records the reactions and behaviors of users and uses them to improve the accuracy of emotion analysis models.

[0187] "Means for capturing a user's facial expression and voice data using smart glasses" refers to a technology that uses smart glasses with built-in cameras and microphones to obtain a user's facial expression and voice information.

[0188] The "communication means for transmitting the data to the server" refers to a technique for transmitting the captured data to the server via the Internet or other communication network.

[0189] The "means for generating personalized product proposals based on analysis results" refers to a technology that selects and proposes optimal products and offers individually based on the user's emotional state and past shopping history.

[0190] The "means for displaying the generated proposals on the display of the smart glasses" is a technology that uses the display of the smart glasses to visually present the generated product proposals or offers to the user.

[0191] "Means for recording user responses and transmitting them to a server" refers to technology that records the actions and responses that users show to displayed suggestions and offers and transmits them to a server via the Internet or other communications network.

[0192] This invention is a system that analyzes users' emotions in real time and customizes the virtual shopping experience based on the analysis results. Specifically, it uses smart glasses to capture the user's facial expressions and voice data, transmits them to a server for analysis, and generates individually optimized product recommendations.

[0193] System Configuration

[0194] server

[0195] Emotion analysis AI model: A model for analyzing a user's facial expressions and voice data to infer emotions. Models created with TensorFlow and PyTorch are used.

[0196] Database: A database for storing user data, suggestion data, and feedback data. MySQL or PostgreSQL can be used.

[0197] Communication method: Protocol used to send and receive data. WebSocket or gRPC is used.

[0198] Terminal (smart glasses)

[0199] Camera and microphone: Devices required to capture the user's facial expressions and voice data.

[0200] Display: A device used to display suggested products and offers to the user.

[0201] Data transmission module: A module for transmitting captured data to the server.

[0202] Program processing overview

[0203] 1. Initialize the server:

[0204] When the server starts, it loads the sentiment analysis model and resources of the virtual shopping environment, including the model generated based on historical user data.

[0205] 2. Initialize the device:

[0206] When the smart glasses start up, the user is prompted, "Would you like to enter the virtual store?"

[0207] If the user agrees, the camera and microphone will be enabled.

[0208] 3. Collection of User Data:

[0209] The smart glasses capture the user's facial expressions and voice and transmit them to a server in real time.

[0210] 4. Emotion analysis:

[0211] The server analyzes the user's emotions based on the received data, using an emotion analysis AI model, and stores the results in a database.

[0212] 5. Product proposal generation:

[0213] Based on the sentiment analysis results and past shopping history, the server generates personalized product suggestions.

[0214] 6. Displaying Proposals:

[0215] The generated suggestions are displayed on the smart glasses' display, and the user can refer to them while shopping.

[0216] 7. Collecting User Feedback:

[0217] The smart glasses record the user's response to the suggested products (e.g., whether they purchase or try them on) and send it to the server.

[0218] 8. Update analysis results:

[0219] The server updates the emotion analysis model based on user feedback data, which improves the accuracy of future suggestions.

[0220] Specific examples

[0221] When a user is searching for clothes in a virtual store, the smart glasses analyze the user's emotions and, if they are satisfied, suggest, "Would you like to see more similar styles?" Conversely, if the user appears dissatisfied, they suggest, "Would you like to see other styles?"

[0222] Prompt Sentence Examples

[0223] "Analyzing the emotions users show during their virtual store experiences."

[0224] "Analyzes the user's emotions in real time based on facial expressions and voice data, and makes appropriate product recommendations."

[0225] This system allows users to receive personalized product suggestions in real time, and the system continuously learns based on feedback data, further improving the shopping experience in future purchases.

[0226] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0227] Step 1:

[0228] Initializing the Server

[0229] When the server starts up, it loads the emotion analysis model and resources for the virtual shopping environment. At this time, the emotion analysis AI model built based on past user data is loaded. The input is the server startup signal, and the output is that the emotion analysis model and resources are ready for use. Specifically, it retrieves the necessary configuration information from the database and loads the model and resources into memory.

[0230] Step 2:

[0231] Initializing the device

[0232] When the smart glasses start up, they prompt the user, "Would you like to enter the virtual store?" The input is the smart glasses' startup signal, and the output is the display of the prompt. Specifically, the system performs a startup check and then displays the prompt on the display.

[0233] Step 3:

[0234] User Data Collection

[0235] Once the user agrees to the prompt, the smart glasses' camera and microphone are enabled and begin capturing the user's facial expressions and voice data, which are then transmitted to the server in real time. The input is the user's consent and the captured data, and the output is the data transmitted to the server. Specifically, the glasses enable the camera and microphone, continuously capture data, and transmit it using a network protocol.

[0236] Step 4:

[0237] Emotion analysis

[0238] The server analyzes the user's emotions in real time using an emotion analysis AI model based on the received facial expression and voice data. The analysis results are saved in a database. The input is the received data, and the output is the analysis results. Specifically, the facial expression data is passed to an image recognition model, and the voice data is passed to a voice recognition model to determine the emotion.

[0239] Step 5:

[0240] Generate product suggestions

[0241] The server generates individually optimized product recommendations based on the sentiment analysis results and the user's past shopping history. The inputs are the analysis results and shopping history, and the output is the generated product recommendations. Specifically, the server retrieves past history from the database and inputs it, along with the sentiment analysis results, into the product recommendation algorithm.

[0242] Step 6:

[0243] View Suggestions

[0244] The generated product suggestions are displayed on the smart glasses' display. The input is the generated product suggestions, and the output is the visual information displayed on the display. Specifically, the suggestion data is converted into a display format and sent to the display for display.

[0245] Step 7:

[0246] Collecting user feedback

[0247] The smart glasses record the user's reaction to the suggested products (e.g., whether they purchase or try them on), and then send the feedback data to the server. The input is the user's reaction, and the output is the feedback data sent to the server. Specifically, the glasses record the user's interaction with sensors and send the data to the server using a network protocol.

[0248] Step 8:

[0249] Update analysis results

[0250] The server updates the sentiment analysis AI model based on the user's feedback data. This improves the accuracy of future suggestions. The input is the feedback data, and the output is an updated sentiment analysis model. Specifically, the server analyzes the feedback data and partially retrains the existing model.

[0251] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0252] This invention relates to a system that analyzes a user's emotions in real time using an emotion engine and customizes a virtual shopping experience based on the analysis results. Specifically, this system includes a terminal that captures the user's facial expressions and voice data, a communication means for transmitting the data to a server, and a function for recognizing the user's emotions using the emotion engine.

[0253] System configuration and processing flow

[0254] composition

[0255] The system consists of the following main components:

[0256] 1. Server

[0257] Emotion analysis AI

[0258] Database

[0259] Emotion Engine

[0260] 2. Terminal

[0261] Camera and microphone

[0262] Data Transmission Module

[0263] Virtual shopping environment interface

[0264] 3. Means of communication

[0265] Network Infrastructure

[0266] Program processing

[0267] 1. Initialize the server

[0268] The server loads the sentiment analysis model, emotion engine, and virtual shopping environment, including connections to machine learning models and necessary databases.

[0269] 2. Initialize the device

[0270] The device checks the camera and microphone and displays the message "Do you want to start shopping online?" The user allows the use of the camera and microphone.

[0271] 3. Collection of User Data

[0272] When a user allows access to the camera and microphone, the device captures the user's facial expressions and voice data in real time and sends it to the server.

[0273] 4. Emotion analysis

[0274] The server's emotion engine analyzes the received facial expression and voice data to infer the user's emotions (e.g., joy, excitement, frustration) in real time. The analysis results are recorded in a database.

[0275] 5. Customize your experience

[0276] The server uses the sentiment analysis results and past shopping history to generate personalized product suggestions and special offers for the user, using machine learning algorithms.

[0277] 6. Display of proposals

[0278] The server sends the generated proposals and offers to the terminal, which displays this information on the user's screen. The user then confirms the proposed products and offers.

[0279] 7. User Feedback

[0280] The device records the user's reactions to the proposed products and offers (e.g., clicks, try-ons, purchases) and sends them to the server.

[0281] 8. Update analysis results

[0282] The server analyzes the user's reaction data sent from the device and stores it in a database as feedback. It then updates the sentiment analysis model and product recommendation algorithm. This update improves the accuracy of future recommendations.

[0283] Specific examples

[0284] Scenario: User wants to try on a virtual item

[0285] 1. Initialize the server

[0286] The server loads the avatar and clothing data for virtual try-on.

[0287] 2. Initialize the device

[0288] The device displays a virtual fitting room interface to users wearing a VR headset.

[0289] 3. Collection of User Data

[0290] While the user looks at themselves in the mirror, a camera and microphone capture their facial expressions and voice, sending the data to a server.

[0291] 4. Emotion analysis

[0292] The server's emotion engine analyzes the user's facial expression data and voice data to analyze emotions (e.g., surprise, satisfaction) in real time. The analysis results are recorded in a database.

[0293] 5. Customize your experience

[0294] If the user is satisfied, the server generates and sends data to the terminal to suggest clothes of a similar style, which the terminal then displays to the user.

[0295] 6. User Feedback

[0296] The user tries on the suggested clothes and responds by commenting or purchasing them, and the device records these responses and sends them to the server.

[0297] 7. Update analysis results

[0298] The server analyzes the user's feedback data, creates a new database entry as feedback, and updates the sentiment analysis model and suggestion algorithm to improve the accuracy of future suggestions.

[0299] In this way, users can enjoy a real-time, personalized online shopping experience, and the system continuously learns to provide a more optimized user experience.

[0300] The processing flow will be explained below.

[0301] Step 1:

[0302] The server loads the sentiment analysis model, emotion engine, and virtual shopping environment, a process that includes setting the parameters of the machine learning model and connecting to the relevant databases.

[0303] Step 2:

[0304] The device checks the camera and microphone and displays the message "Do you want to start shopping online?" The user allows the use of the camera and microphone.

[0305] Step 3:

[0306] The device captures the user's facial expressions with a camera and collects audio data with a microphone. This capture process is carried out periodically and the data is sent to a server in real time.

[0307] Step 4:

[0308] The server's emotion engine analyzes the received facial expression data and voice data to infer the user's emotions (e.g., joy, excitement, dissatisfaction) in real time. The analysis results are recorded in a database.

[0309] Step 5:

[0310] The server uses the results of the sentiment analysis and past shopping history to generate personalized product suggestions and special offers for the user, using machine learning algorithms.

[0311] Step 6:

[0312] The server sends the generated proposals and offers to the device, which then displays the information on the user's screen. The user can then review the proposed products and offers and take action as necessary.

[0313] Step 7:

[0314] The user's reactions to the proposed products and offers (e.g., clicks, try-ons, purchases) are recorded by the device and transmitted to the server in real time.

[0315] Step 8:

[0316] The server analyzes the user's response data and creates new database entries as feedback. This feedback data is an important factor in future analysis and proposal generation.

[0317] Step 9:

[0318] The server updates the sentiment analysis model and product recommendation algorithm based on the feedback data, improving the accuracy of future recommendations and optimizing the user experience.

[0319] Step 10:

[0320] The terminal will continue to present new suggestions and offers to the user based on their emotions, and the whole process will be repeated in real time, allowing the user to continuously enjoy the virtual shopping experience.

[0321] Example 2

[0322] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0323] Conventional virtual shopping systems have struggled to provide a personalized experience based on users' emotions and preferences. They also lacked the means to collect real-time user feedback and improve the system based on it. This resulted in a uniform shopping experience, making it difficult to improve user satisfaction.

[0324] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for loading an emotion analysis model, means for checking the camera and microphone and confirming with the user whether to start online shopping, means for capturing the user's facial expression and voice data in real time and sending it to the server, means for analyzing the user's emotions in real time, means for generating personalized product suggestions and special offers based on the analysis results and past shopping history, means for displaying the generated suggestions and offers on the user terminal, and means for collecting user feedback and updating the analysis results and the suggestion algorithm. This makes it possible to provide a personalized shopping experience based on the user's emotions and preferences and to continuously improve the system.

[0325] An "emotion analysis model" is a machine learning model that predicts a user's emotions in real time based on their facial expressions and voice data.

[0326] A "camera" is a photographing device for capturing the user's facial expressions.

[0327] A "microphone" is a voice recording device for capturing the user's voice data.

[0328] "Server" refers to a computer system that manages and operates the emotion analysis model, emotion engine, and virtual shopping environment.

[0329] An "emotion engine" is software that analyzes facial expressions and voice data to predict emotions in real time.

[0330] The "database" is a system for storing and managing user sentiment analysis results, shopping history, feedback data, etc.

[0331] A "virtual shopping environment" is a virtual environment in which users can experience shopping online.

[0332] "Communication means" refers to the network infrastructure for transmitting the user's facial expressions and voice data from the terminal to the server.

[0333] "Feedback" refers to the reactions and actions that users take in response to proposed products or offers.

[0334] "Product suggestions" refers to suggesting personalized products to users based on the results of emotional analysis of the user and their past shopping history.

[0335] "Special Offer" refers to a specific discount or promotion that is customized for you.

[0336] A "suggestion algorithm" is a computational means for generating optimal product suggestions and special offers based on the results of a user's sentiment analysis and past history.

[0337] The present invention relates to a system for analyzing a user's emotions in real time and customizing a virtual shopping experience based on the analysis results. A specific embodiment of this system will be described below.

[0338] System configuration

[0339] This system mainly consists of three main components: a server, a terminal, and a communication means.

[0340] 1. Server

[0341] Sentiment analysis model: Refers to a neural network model built using a machine learning library such as TensorFlow.

[0342] Emotion engine: Software that uses OpenAI (registered trademark) APIs and other technologies to analyze a user's facial expressions and voice data.

[0343] Database: We use a database management system such as MySQL to store and manage user sentiment analysis results, shopping history, and feedback data.

[0344] 2. Terminal

[0345] Camera: A photographic device for capturing the user's facial expressions.

[0346] Microphone: A voice recording device for capturing the user's voice data.

[0347] Data transmission module: A communication module for transmitting facial expression data and voice data to the server.

[0348] Virtual shopping environment interface: The interface through which the user operates, for example, using a VR headset or touchscreen.

[0349] 3. Means of communication

[0350] Network infrastructure: A means of transmitting data from a terminal to a server using communication infrastructure such as the Internet.

[0351] Program processing

[0352] The programs in this system perform the following basic tasks:

[0353] 1. Loading the sentiment analysis model

[0354] The server uses TensorFlow to load a sentiment analysis model, which includes a pre-trained neural network.

[0355] 2. Check your camera and microphone

[0356] The device checks whether the camera and microphone are properly connected, displays a message to the user asking "Do you want to start shopping online?" and asks for permission.

[0357] 3. Data Capture

[0358] The camera captures the user's facial expressions and the microphone collects audio data, which is sent to a server in real time.

[0359] 4. Emotion analysis

[0360] The server's emotion engine analyzes the received facial expression and voice data to infer the user's emotion in real time. For example, it can infer the emotion "satisfied" by analyzing a smiling face and a calm voice.

[0361] 5. Product proposal generation

[0362] The server generates personalized product suggestions and special offers based on the sentiment analysis results and past shopping history, using machine learning algorithms such as Scikit-learn.

[0363] 6. Display of proposals

[0364] The server sends the generated recommendations to the device, which displays them on the user's screen, allowing the user to view the recommendations, click, and purchase.

[0365] 7. Gathering Feedback

[0366] The device records the user's reactions (clicks, trying on, purchases, etc.) and sends them to the server, which analyzes them and updates the sentiment analysis model and proposed algorithm.

[0367] Specific examples

[0368] For example, in the scenario where a user is trying on new clothes while shopping online:

[0369] 1. Initialize the server

[0370] Load the TensorFlow model and emotion engine and connect to the database.

[0371] 2. Initialize the device

[0372] It checks your camera and microphone and displays the message "Do you want to start shopping online?"

[0373] 3. Data Capture

[0374] As the user tries on new clothes, the camera and microphone capture facial expressions and voice data, which are then sent to a server.

[0375] 4. Emotion analysis

[0376] The server receives the captured data and uses an emotion analysis model to infer the emotion "satisfied."

[0377] 5. Product proposal generation

[0378] The server suggests clothes with a similar style to the user based on the emotion of "satisfaction" and past purchase history.

[0379] 6. Display of proposals

[0380] The server sends the proposal to the terminal, which displays it on the user's screen.

[0381] 7. Gathering Feedback

[0382] The user clicks on the suggested clothing and reacts by trying it on or purchasing it, and the device records the reaction and sends it to the server.

[0383] Prompt Sentence Examples

[0384] The following prompt sentences are fed into the generative AI model to perform sentiment analysis:

[0385] A user is trying on new clothes and smiling as he looks at himself in the mirror. His voice is calm and relaxed. How should the server's emotion engine interpret this data and what product recommendations should it make?

[0386] In this way, the system can analyze user sentiment and provide a personalized shopping experience, and it also makes continuous improvements through feedback to provide a more optimized user experience.

[0387] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0388] Step 1: Initialize the server

[0389] explanation

[0390] At system startup, the server does the following:

[0391] Specific actions

[0392] 1. Loading the sentiment analysis model

[0393] Load a sentiment analysis model using TensorFlow. As input, a pre-trained model file is required. As output, the model is loaded into memory and ready for analysis.

[0394] Example: Read the file path of a model and load it using a TensorFlow function.

[0395] 2. Activating the Emotional Engine

[0396] Starts the emotion engine and waits for a connection. It uses the emotion engine's configuration file as input and the engine is started and starts accepting requests as output.

[0397] Example: Initialize the engine by reading the OpenAI API key and necessary configuration information.

[0398] 3. Setting up a virtual shopping environment

[0399] It connects to the required databases and sets up a virtual shopping environment, taking database configuration information and connection credentials as input, and an established database connection ready for operation as output.

[0400] Example: Use MySQL connection information to connect to a database and cache product information and user history.

[0401] Step 2: Reset the device

[0402] explanation

[0403] To provide an interface with the user, the terminal:

[0404] Specific actions

[0405] 1. Hardware check

[0406] Checks whether the camera and microphone are properly connected. It takes hardware status information as input and displays the connection status as output.

[0407] Example: Display a message to the user asking "Do you want to turn on your camera?" and ask for permission.

[0408] 2. Interface display

[0409] Prompts the user with the message "Start shopping online?" Uses user interface configuration information as input and provides the message as output.

[0410] Example: Asking the user for permission using a GUI or VR interface.

[0411] Step 3: Collect user data

[0412] explanation

[0413] The device collects user data in real time and sends it to the server:

[0414] Specific actions

[0415] 1. Data Capture

[0416] The camera continuously captures the user's facial expressions, using real-time video data as input and generating facial expression data as output.

[0417] Example: Activating the camera module to analyze facial expressions frame by frame.

[0418] 2. Data Transmission

[0419] The microphone collects the user's voice data and sends it to the server. It uses real-time voice data as input and sends voice data to the server as output.

[0420] Example: Start an audio capture module and periodically upload audio data to a server.

[0421] Step 4: Sentiment Analysis

[0422] explanation

[0423] The server analyzes the user's emotions based on the received data:

[0424] Specific actions

[0425] 1. Data Reception

[0426] It receives facial expression data and voice data from the terminal. It receives data sent from the terminal and data packets from the relay server as input, and stores the data in an analysis buffer as output.

[0427] Example: Receive data at a server API endpoint and pass it to an analytics engine.

[0428] 2. Emotional inference

[0429] The emotion engine analyzes facial expression and voice data to infer emotions in real time, using the received data as input and generating inferred emotions as output.

[0430] Example: Transferring a smiling face and a calm voice to infer the emotion "satisfied."

[0431] 3. Data Recording

[0432] Record the analysis results in a database. The sentiment analysis results are used as input and a new record is added to the database as output.

[0433] Example: Inserting inferred sentiment into MySQL.

[0434] Step 5: Customize the experience

[0435] explanation

[0436] The server generates personalized suggestions for the user based on the analysis results:

[0437] Specific actions

[0438] 1. Data analysis

[0439] Analyzes the sentiment analysis results and past shopping history. The historical information retrieved from the database and the analysis results are used as input, and the analysis results are generated as output.

[0440] Example: Inferring user preferences by combining past purchase history and current sentiment analysis results.

[0441] 2. Proposal generation

[0442] Generate personalized product recommendations and special offers, using analysis results as input and recommendations as output.

[0443] Example: Using Scikit-learn algorithms to make optimal product recommendations.

[0444] Step 6: View the proposal

[0445] explanation

[0446] Display the suggestions on the device so the user can review them:

[0447] Specific actions

[0448] 1. Submit a proposal

[0449] Send the generated proposal or offer to the device, using the proposal as input and successful transmission as output.

[0450] Example: Send proposal data in JSON format to the device.

[0451] 2. User Interface

[0452] The proposal is displayed on the user's screen. The proposal is used as input, and a visual display is produced as output that the user can confirm.

[0453] Example: Displaying product suggestions using a GUI or VR interface.

[0454] Step 7: User feedback

[0455] explanation

[0456] Collect the user's responses to the suggestions and send them to the server:

[0457] Specific actions

[0458] 1. Feedback Collection

[0459] Record user responses (clicks, try-ons, purchases, etc.) Uses user action data as input and generates feedback data as output.

[0460] Example: Storing a user's click history or purchase history.

[0461] 2. Data Transmission

[0462] The collected feedback data is sent to the server, which uses the feedback data as input and completes sending it to the server as output.

[0463] Example: Uploading feedback data to a server using an API.

[0464] Step 8: Update the analysis results

[0465] explanation

[0466] The server uses the feedback to update the model and algorithms to improve accuracy:

[0467] Specific actions

[0468] 1. Data Analysis

[0469] Analyze the received feedback data, using the feedback data as input and producing the analysis results as output.

[0470] Example: Statistical analysis of user feedback data to extract trends.

[0471] 2. Model Update

[0472] Update sentiment analysis models and product recommendation algorithms: new models and algorithms are generated using the analysis results as input and output.

[0473] Example: Retraining the sentiment analysis model and optimizing the proposed algorithm.

[0474] Through the above processing steps, the system can analyze user sentiment in real time and provide a personalized shopping experience.

[0475] (Application example 2)

[0476] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0477] Conventional online shopping systems have the problem that they do not provide product recommendations that fully understand the user's preferences, resulting in poor user experience. They also have been unable to understand the user's emotions and personalize the shopping experience in real time based on those emotions. Therefore, there is a need to provide a personalized shopping experience that reflects the user's interests.

[0478] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0479] In this invention, the server includes means for analyzing user emotions in real time, means for customizing the virtual shopping experience based on the analysis results, means for collecting user feedback and updating the analysis results, means for recognizing the user's emotions from the captured data and generating emotion-based product suggestions and special offers, and means for displaying the generated suggestions and offers on the user's terminal, thereby enabling personalized product suggestions based on the user's emotions.

[0480] "Means for analyzing user emotions in real time" refers to a function that analyzes the user's facial expressions and voice data obtained using a camera or microphone, etc., and instantly recognizes the user's emotions at that time.

[0481] "Means to customize the virtual shopping experience" refers to the ability to optimize the shopping environment and recommendations for individual users based on the results of sentiment analysis.

[0482] "Means of collecting user feedback and updating analysis results" refers to the function of recording user reactions and behavioral data and using it to improve emotion analysis models and product recommendation algorithms.

[0483] "Means of recognizing a user's emotions from captured data and generating emotion-based product suggestions and special offers" refers to a function that selects and suggests products and services suitable for the user based on emotional data obtained by analyzing collected facial expressions and voice data.

[0484] "Means for displaying generated suggestions and offers on the user's device" refers to the function of displaying product suggestions and special offers generated through sentiment analysis on the user's device, such as a smartphone, smart glasses, or head-mounted display.

[0485] A "terminal for displaying a virtual fitting room interface" is a device that provides an interface for users to try on clothes in a virtual space, and in many cases a head-mounted display is used.

[0486] "Means of recording specific user responses from captured data and using them as feedback" refers to the function of recording user behavioral data such as clicks and purchases and using it as learning data for algorithms.

[0487] This invention is a system that analyzes a user's emotions in real time and customizes a virtual shopping experience based on the analysis results. This system is composed of an emotion analysis engine, a server, a database, a camera, a microphone, a terminal, and communication means. Detailed embodiments of each component of this invention will be described below.

[0488] System configuration

[0489] 1. Server

[0490] Emotion analysis engine: Analyzes the user's facial expression and voice data to recognize emotions in real time.

[0491] Database: Stores analysis results and user feedback data and is used to train machine learning algorithms.

[0492] Product recommendation engine: Generates product suggestions and special offers based on sentiment analysis results.

[0493] 2. Terminal

[0494] Camera and microphone: Captures the user's facial expressions and voice data.

[0495] Data transmission module: Sends the captured data to the server.

[0496] Virtual Shopping Environment Interface: Providing users with a virtual shopping experience via a VR headset or smart glasses.

[0497] 3. Means of communication

[0498] Network infrastructure: Carries out data communication between user terminals and servers.

[0499] System processing flow

[0500] 1. Initialization

[0501] The server loads the necessary software, including a sentiment analysis engine and a product recommendation engine, while the device checks the camera and microphone and displays a confirmation message to the user to begin the virtual shopping experience. The user then authorizes the use of the camera and microphone.

[0502] 2. Collection of User Data

[0503] The device captures the user's facial expressions and voice data in real time and sends the data to a server. The camera and microphone use OpenCV and other image and voice processing libraries.

[0504] 3. Emotion analysis

[0505] The server's emotion analysis engine analyzes the received data and recognizes the user's emotions using a machine learning model, and the results are stored in a database.

[0506] 4. Presenting the generated proposals

[0507] The server generates optimal product suggestions and special offers for the user based on the sentiment analysis results and database. The generated suggestions are sent to the terminal and displayed on the user's device. The user can then check the suggested products in the virtual store.

[0508] Example scenario

[0509] For example, if a user is trying on a dress in a virtual fitting room, a camera and microphone capture the user's facial expressions and voice and send them to the server. If the server detects that the user is pleased with the dress, it can suggest other dresses with the same design but in different colors, as well as matching accessories.

[0510] Prompt Sentence Examples

[0511] plain text

[0512] Suggest items related to products that users enjoy

[0513] This invention makes it possible to provide a personalized virtual shopping experience that is tailored to the user's emotions. Users can receive optimal product suggestions based on their emotions in real time, improving their shopping experience.

[0514] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0515] Step 1:

[0516] Initializing the Server

[0517] The server loads the necessary software components, such as the sentiment analysis AI, emotion engine, database, and product recommendation engine. The initialization process checks the connections between each component and ensures that the system works properly. Machine learning models and necessary algorithms are also loaded at this stage. The server is then ready to receive and analyze data from users.

[0518] Step 2:

[0519] Initializing the device

[0520] The device checks the camera and microphone and displays a message to the user saying "Do you want to start shopping online?", at which point the user is prompted to allow the use of the camera and microphone. If permission is granted, the device prepares to capture data through the camera and microphone.

[0521] Step 3:

[0522] User Data Collection

[0523] The device captures the user's facial expressions and voice data in real time and sends the obtained data to the server. During this process, the camera takes a picture of the user's face and the microphone records the user's voice. The captured data is sent to the server via the network. The input data is image data and voice data, and the output is the data sent to the server.

[0524] Step 4:

[0525] Emotion analysis

[0526] The server inputs the received facial expression data and voice data into an emotion analysis engine, which analyzes the user's emotions in real time. This analysis is performed using a machine learning model. The emotion analysis engine uses image processing and voice analysis algorithms to estimate the user's emotions from the data. The input data is facial expression data and voice data, and the output is the emotion analysis result. The analysis results are recorded in a database.

[0527] Step 5:

[0528] Generate product suggestions

[0529] The server generates optimal product suggestions and special offers for the user based on the sentiment analysis results and a database of past users. This step often uses a generative AI model, which allows the server to create products and offers that match the user's current emotions. The input data are the sentiment analysis results and past user data, and the output is the generated product suggestions and special offers.

[0530] Step 6:

[0531] Viewing Proposals

[0532] The server sends the generated product suggestions and special offers to the terminal. The terminal displays the suggestions on the user's screen. The user can then check the suggested products in the virtual store. The input data are the product suggestions and special offers, and the output is the information displayed on the user's screen.

[0533] Step 7:

[0534] Collecting User Feedback

[0535] The device records the user's reactions to the proposed products and offers, such as clicking, trying them on, and purchasing, and sends them to the server. This data will be used to improve future sentiment analysis models and product recommendation engines. The input data is the user's reaction data, and the output is feedback data sent to the server.

[0536] Step 8:

[0537] Update analysis results

[0538] The server analyzes the feedback data and creates new database entries. This data is used to train the sentiment analysis model and product recommendation algorithm to improve the accuracy of future recommendations. The input data is the feedback data, and the output is the updated analysis model and algorithm.

[0539] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0540] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0541] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0542] [Second embodiment]

[0543] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0544] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0545] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0546] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0547] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0548] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0549] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0550] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0551] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0552] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0553] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0554] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0555] This invention relates to a system that analyzes a user's emotions in real time and customizes a virtual shopping experience based on the analysis results. Specifically, the system includes a terminal for capturing the user's facial expression and voice data and a communication means for transmitting the data to a server. The server analyzes the user's emotions based on the received data and generates product suggestions and special offers based on the analysis results. The generated suggestions and offers are displayed on the user's terminal, and the analysis results are updated based on the user's feedback to improve the accuracy of future suggestions.

[0556] System configuration and processing flow

[0557] composition

[0558] The system consists of the following main components:

[0559] 1. Server

[0560] Emotion analysis AI

[0561] Database

[0562] 2. Terminal

[0563] Camera and microphone

[0564] Data Transmission Module

[0565] Virtual shopping environment interface

[0566] 3. Means of communication

[0567] Network Infrastructure

[0568] Program processing

[0569] 1. Initialize the server

[0570] The server loads the sentiment analysis model and the virtual shopping environment, which are generated from historical user data.

[0571] 2. Initialize the device

[0572] The device checks the camera and microphone and displays the message "Do you want to start shopping online?" to the user.

[0573] 3. Collection of User Data

[0574] When a user allows access to the camera and microphone, the device captures the user's facial expressions and voice data and sends it to the server.

[0575] 4. Emotion analysis

[0576] The server analyzes the facial expression and voice data sent from the device and infers the user's emotions in real time. The analysis results are recorded in a database.

[0577] 5. Customize your experience

[0578] The server generates personalized product suggestions and special offers for the user based on the results of sentiment analysis and past shopping history.

[0579] The generated proposals and offers are sent to the terminal and displayed on the user's screen.

[0580] 6. User Feedback

[0581] The user responds to the suggested product or offer (e.g., clicks, tries on, purchases).

[0582] The terminal records this response and sends it to the server as feedback data.

[0583] 7. Update analysis results

[0584] The server updates the sentiment analysis model based on the user's feedback data to improve the accuracy of suggestions for the next shopping session.

[0585] Specific examples

[0586] Scenario: User wants to try on a virtual item

[0587] 1. Initialize the server

[0588] The server loads the avatar and clothing data for virtual try-on.

[0589] 2. Initialize the device

[0590] The device displays a virtual fitting room interface to users wearing a VR headset.

[0591] 3. Collection of User Data

[0592] While the user looks at themselves in the mirror, a camera and microphone capture their facial expressions and voice, sending the data to a server.

[0593] 4. Emotion analysis

[0594] The server analyzes the user's emotions (e.g., surprise, satisfaction) in real time based on facial expression data and voice data.

[0595] 5. Customize your experience

[0596] If the user is satisfied, the server generates data for suggesting clothes of a similar style and transmits it to the terminal.

[0597] The terminal displays these suggestions to the user.

[0598] 6. User Feedback

[0599] Users try on the suggested clothes and respond by rejecting them or purchasing them.

[0600] The device records this response and sends it to the server.

[0601] 7. Update analysis results

[0602] The server integrates user feedback to update the analytical model, improving the accuracy of future proposals.

[0603] In this way, users can enjoy a real-time, personalized online shopping experience, and the system continuously learns and optimizes the user experience.

[0604] The processing flow will be explained below.

[0605] Step 1:

[0606] The server loads the sentiment analysis model and the virtual shopping environment, including the machine learning models and connections to the necessary databases.

[0607] Step 2:

[0608] The device checks the camera and microphone and displays the message "Do you want to start shopping online?" The user allows the use of the camera and microphone.

[0609] Step 3:

[0610] The device periodically captures the user's facial expressions with a camera and collects audio data with a microphone, which is then sent to a server every few seconds.

[0611] Step 4:

[0612] The server analyzes the received facial expression and voice data to infer the user's emotions (e.g., joy, excitement, dissatisfaction) in real time. The analysis results are recorded in a database.

[0613] Step 5:

[0614] The server uses the analysis results and past shopping history to generate personalized product suggestions and special offers for the user, using machine learning algorithms.

[0615] Step 6:

[0616] The server sends the generated proposals and offers to the terminal, which displays this information on the user's screen. The user then confirms the proposed products and offers.

[0617] Step 7:

[0618] The user's reactions to the proposed products and offers (e.g., clicking, trying on, purchasing) are recorded by the device and transmitted to the server.

[0619] Step 8:

[0620] The server analyzes the user's reaction data sent from the terminal and creates a new database entry as feedback data.

[0621] Step 9:

[0622] The server updates the sentiment analysis model and product recommendation algorithm based on the feedback data, improving the accuracy of future recommendations.

[0623] Step 10:

[0624] The device will continue to present new sentiment-based suggestions and offers to the user, and the whole process will be repeated in real time to optimize the user experience.

[0625] Example 1

[0626] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0627] In conventional online shopping, it was difficult to analyze user emotions in real time and personalize the shopping experience based on that. Furthermore, there was a lack of a mechanism for collecting user feedback and updating the analysis results, which meant that product recommendations did not adequately reflect the user's interests and satisfaction. This resulted in a poor user experience and an inability to fully stimulate purchasing motivation.

[0628] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0629] In this invention, the server includes means for loading and initializing an emotion analysis model, means for collecting facial expression and voice data of a user, means for transmitting the collected data to the server, means for analyzing the user's emotions in real time based on the data received by the server, means for recording the analysis results in a database, means for generating personalized product suggestions and special offers based on the emotion analysis results and past shopping history, means for displaying the generated suggestions and offers on the user terminal, means for collecting user feedback and transmitting the feedback data to the server, and means for updating the emotion analysis model based on the feedback data. This makes it possible to analyze the user's emotions in real time and provide a highly personalized shopping experience that reflects the results.

[0630] An "emotion analysis model" is an algorithm or program that analyzes a user's facial expressions and voice data to infer their emotional state.

[0631] A "server" is a computer system that processes and stores data and provides that data to other computers or devices over a network.

[0632] "User's facial expression and voice data" refers to digital data of human facial movements and voice signals captured using a camera and microphone.

[0633] A "database" is a structured collection of data for efficiently storing, retrieving, and updating information.

[0634] "Product suggestion" refers to the act of introducing products that are likely to interest a user based on the results of an emotional analysis of the user and their past shopping history, and the data used to do so.

[0635] "Special Offers" are commercial offers such as special prices or promotions based on the user's interests.

[0636] "Feedback data" refers to data based on user behavior and reactions that indicate interest in and intent to purchase product suggestions and special offers.

[0637] "Personalization" means customizing product proposals and services to suit the characteristics and preferences of individual users.

[0638] "Real-time" is a concept that refers to processing and reactions occurring almost simultaneously with the collection of data.

[0639] This invention relates to a system that analyzes a user's emotions in real time and customizes a virtual shopping experience based on the analysis results. Specifically, the system includes a terminal for capturing the user's facial expression and voice data and a communication means for transmitting the data to a server. The server analyzes the user's emotions based on the received data and generates product suggestions and special offers based on the analysis results. The generated suggestions and offers are displayed on the user's terminal, and the analysis results are updated based on user feedback, thereby improving the accuracy of future suggestions.

[0640] The main components of the system are as follows:

[0641] 1. Server

[0642] Sentiment analysis model (e.g., an AI model using TensorFlow)

[0643] Database (e.g. MySQL)

[0644] 2. Terminal

[0645] Camera and microphone (e.g., a standard webcam or microphone)

[0646] Data transmission module (e.g. Wi-Fi or Bluetooth)

[0647] Virtual shopping environment interface (e.g., built with Unity)

[0648] 3. Means of communication

[0649] Network infrastructure (e.g., 5G, fiber optic connections)

[0650] Specifically, the server first loads the emotion analysis model and virtual shopping environment. This requires initializing an AI model using TensorFlow and establishing a connection to a database of products taken from past user data. The device then diagnoses the camera and microphone, checks their operation, and displays a message to the user asking, "Do you want to start online shopping?" If the user allows access to the camera and microphone, the device captures the user's facial expressions and voice data and sends it to the server.

[0651] The server analyzes the facial expression and voice data sent from the device and infers the user's emotions in real time. The analysis results are recorded in a database. The server then generates personalized product suggestions and special offers for the user based on the emotion analysis results and past shopping history. The generated suggestions and offers are sent to the device and displayed on the user's screen. When the user responds to the suggested products or offers (e.g., clicking, trying them on, purchasing), the device records this response and sends it to the server as feedback data. Finally, the server updates the emotion analysis model based on the user's feedback data to improve the accuracy of suggestions for the next shopping session.

[0652] As a specific example, consider a user using a virtual fitting service. The server loads the virtual fitting avatar and clothing data, and the device displays a virtual fitting room interface to the user wearing a VR headset. While the user looks at themselves in the mirror, a camera and microphone capture their facial expressions and voice, and send the data to the server. The server analyzes the user's emotions (e.g., surprise, satisfaction) in real time based on the facial expression and voice data. The server then suggests more similar styles of clothing based on the emotion analysis results, and the device displays these suggestions to the user. When the user tries on the suggested clothing and responds by rejecting it or purchasing it, the device records this response and sends it to the server. The server integrates the user's feedback to update the analysis model, improving the accuracy of future suggestions.

[0653] Examples of prompt statements

[0654] Prompt: "Analyze the user's facial expressions and voice in a virtual fitting room to suggest outfits that match their emotions."

[0655] Input data example: Facial expression data [image file], voice data ["I like this outfit!"]

[0656] In this way, users can enjoy a real-time, personalized online shopping experience, and the system can continuously learn and optimize the user experience.

[0657] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0658] Step 1:

[0659] Initializing the Server

[0660] The server loads the sentiment analysis model and the virtual shopping environment. Specifically, it first loads the sentiment analysis model parameters into memory using TensorFlow. Next, it establishes a connection to a product database (e.g., MySQL) based on past user data and loads the necessary product information into memory. This allows the server to perform sentiment analysis and product suggestions in real time.

[0661] Input: TensorFlow model parameter file, product database connection information

[0662] Output: Initialized sentiment analysis model, product data loaded into memory.

[0663] Step 2:

[0664] Initializing the device

[0665] The device diagnoses the camera and microphone to ensure they are working properly. It then displays a message to the user asking, "Do you want to start online shopping?" If the user responds, the device prepares to send the input to the server.

[0666] Specific actions include checking the camera video feed and checking the volume level of the microphone input.

[0667] Input: User actions, device camera and microphone diagnostic results

[0668] Output: User camera and microphone permissions, confirmation to start shopping

[0669] Step 3:

[0670] User Data Collection

[0671] Once the user allows the use of the camera and microphone, the device will activate them and continuously capture the user's facial expressions and voice data, which will be transmitted to the server in real time. The camera will capture high-resolution images, and the microphone will record clear audio.

[0672] Input: Video data from camera, audio data from microphone

[0673] Output: Facial expression data and voice data sent to the server

[0674] Step 4:

[0675] Emotion analysis

[0676] The server receives and analyzes the facial expression and voice data sent from the device. It passes the facial expression images and voice data as input to an emotion analysis model, which then infers the user's emotions in real time. The analysis results are then recorded in a database.

[0677] Input: Emotion analysis model loaded from database, received facial expression data and speech data

[0678] Output: User's emotional state, analysis results recorded in a database

[0679] Step 5:

[0680] Customize your experience

[0681] The server generates personalized product suggestions and special offers for the user based on the results of sentiment analysis and past shopping history. The generated suggestions and offers are sent in JSON format to the device, which then displays them on the user's screen.

[0682] Input: Sentiment analysis results, past shopping history, product database

[0683] Output: Generated product suggestions, special offers, and display data sent to the device

[0684] Step 6:

[0685] User Feedback

[0686] When users interact with the proposed products or offers (e.g., click, try on, purchase), the device records these interactions, and the recorded feedback data is sent to the server in real time.

[0687] Input: User interaction data

[0688] Output: Feedback data sent to the server

[0689] Step 7:

[0690] Update analysis results

[0691] The server receives user feedback data and updates the sentiment analysis model based on that data. This update improves the accuracy of suggestions for the next shopping session by optimizing the model parameters and algorithms.

[0692] Input: Feedback data, sentiment analysis model

[0693] Output: Updated sentiment analysis model, improving the accuracy of the next proposal

[0694] Through these steps, the system can analyze user sentiment in real time, generate and present product suggestions and special offers tailored to each user, and continuously learn to optimize the user experience.

[0695] (Application example 1)

[0696] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0697] Conventional virtual shopping experiences have had the challenge of being difficult to flexibly respond to individual users' emotions and preferences. In particular, there has been a lack of systems that can analyze a user's emotional state in real time and make product recommendations based on the results. This means that users are often left feeling dissatisfied because they are rarely presented with products that suit them at the right time. Furthermore, there has been a lack of mechanisms for effectively utilizing user feedback to improve the shopping experience.

[0698] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0699] In this invention, the server includes means for analyzing user emotions in real time, means for collecting user feedback and updating the analysis results, and means for capturing the user's facial expression and voice data using the smart glasses, which enables the generation of personalized product suggestions and special offers based on the user's emotions and feedback and displaying these suggestions on the display of the smart glasses, thereby providing the user with a more comfortable and efficient shopping experience.

[0700] "Means for analyzing a user's emotions in real time" refers to a technology that instantly determines the user's emotional state at any given time based on the facial expressions and voice data they show.

[0701] "Means for customizing virtual shopping experiences" refers to technology that individually adjusts and optimizes the shopping experience based on user sentiment analysis and feedback.

[0702] The "means of collecting user feedback and updating analysis results" refers to a technology that continuously records the reactions and behaviors of users and uses them to improve the accuracy of emotion analysis models.

[0703] "Means for capturing a user's facial expression and voice data using smart glasses" refers to a technology that uses smart glasses with built-in cameras and microphones to obtain a user's facial expression and voice information.

[0704] The "communication means for transmitting the data to the server" refers to a technique for transmitting the captured data to the server via the Internet or other communication network.

[0705] The "means for generating personalized product proposals based on analysis results" refers to a technology that selects and proposes optimal products and offers individually based on the user's emotional state and past shopping history.

[0706] The "means for displaying the generated proposals on the display of the smart glasses" is a technology that uses the display of the smart glasses to visually present the generated product proposals or offers to the user.

[0707] "Means for recording user responses and transmitting them to a server" refers to technology that records the actions and responses that users show to displayed suggestions and offers and transmits them to a server via the Internet or other communications network.

[0708] This invention is a system that analyzes users' emotions in real time and customizes the virtual shopping experience based on the analysis results. Specifically, it uses smart glasses to capture the user's facial expressions and voice data, transmits them to a server for analysis, and generates individually optimized product recommendations.

[0709] System Configuration

[0710] server

[0711] Emotion analysis AI model: A model for analyzing a user's facial expressions and voice data to infer emotions. Models created with TensorFlow and PyTorch are used.

[0712] Database: A database for storing user data, suggestion data, and feedback data. MySQL or PostgreSQL can be used.

[0713] Communication method: Protocol used to send and receive data. WebSocket or gRPC is used.

[0714] Terminal (smart glasses)

[0715] Camera and microphone: Devices required to capture the user's facial expressions and voice data.

[0716] Display: A device used to display suggested products and offers to the user.

[0717] Data transmission module: A module for transmitting captured data to the server.

[0718] Program processing overview

[0719] 1. Initialize the server:

[0720] When the server starts, it loads the sentiment analysis model and resources of the virtual shopping environment, including the model generated based on historical user data.

[0721] 2. Initialize the device:

[0722] When the smart glasses start up, the user is prompted, "Would you like to enter the virtual store?"

[0723] If the user agrees, the camera and microphone will be enabled.

[0724] 3. Collection of User Data:

[0725] The smart glasses capture the user's facial expressions and voice and transmit them to a server in real time.

[0726] 4. Emotion analysis:

[0727] The server analyzes the user's emotions based on the received data, using an emotion analysis AI model, and stores the results in a database.

[0728] 5. Product proposal generation:

[0729] Based on the sentiment analysis results and past shopping history, the server generates personalized product suggestions.

[0730] 6. Displaying Proposals:

[0731] The generated suggestions are displayed on the smart glasses' display, and the user can refer to them while shopping.

[0732] 7. Collecting User Feedback:

[0733] The smart glasses record the user's response to the suggested products (e.g., whether they purchase or try them on) and send it to the server.

[0734] 8. Update analysis results:

[0735] The server updates the emotion analysis model based on user feedback data, which improves the accuracy of future suggestions.

[0736] Specific examples

[0737] When a user is searching for clothes in a virtual store, the smart glasses analyze the user's emotions and, if they are satisfied, suggest, "Would you like to see more similar styles?" Conversely, if the user appears dissatisfied, they suggest, "Would you like to see other styles?"

[0738] Prompt Sentence Examples

[0739] "Analyzing the emotions users show during their virtual store experiences."

[0740] "Analyzes the user's emotions in real time based on facial expressions and voice data, and makes appropriate product recommendations."

[0741] This system allows users to receive personalized product suggestions in real time, and the system continuously learns based on feedback data, further improving the shopping experience in future purchases.

[0742] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0743] Step 1:

[0744] Initializing the Server

[0745] When the server starts up, it loads the emotion analysis model and resources for the virtual shopping environment. At this time, the emotion analysis AI model built based on past user data is loaded. The input is the server startup signal, and the output is that the emotion analysis model and resources are ready for use. Specifically, it retrieves the necessary configuration information from the database and loads the model and resources into memory.

[0746] Step 2:

[0747] Initializing the device

[0748] When the smart glasses start up, they prompt the user, "Would you like to enter the virtual store?" The input is the smart glasses' startup signal, and the output is the display of the prompt. Specifically, the system performs a startup check and then displays the prompt on the display.

[0749] Step 3:

[0750] User Data Collection

[0751] Once the user agrees to the prompt, the smart glasses' camera and microphone are enabled and begin capturing the user's facial expressions and voice data, which are then transmitted to the server in real time. The input is the user's consent and the captured data, and the output is the data transmitted to the server. Specifically, the glasses enable the camera and microphone, continuously capture data, and transmit it using a network protocol.

[0752] Step 4:

[0753] Emotion analysis

[0754] The server analyzes the user's emotions in real time using an emotion analysis AI model based on the received facial expression and voice data. The analysis results are saved in a database. The input is the received data, and the output is the analysis results. Specifically, the facial expression data is passed to an image recognition model, and the voice data is passed to a voice recognition model to determine the emotion.

[0755] Step 5:

[0756] Generate product suggestions

[0757] The server generates individually optimized product recommendations based on the sentiment analysis results and the user's past shopping history. The inputs are the analysis results and shopping history, and the output is the generated product recommendations. Specifically, the server retrieves past history from the database and inputs it, along with the sentiment analysis results, into the product recommendation algorithm.

[0758] Step 6:

[0759] View Suggestions

[0760] The generated product suggestions are displayed on the smart glasses' display. The input is the generated product suggestions, and the output is the visual information displayed on the display. Specifically, the suggestion data is converted into a display format and sent to the display for display.

[0761] Step 7:

[0762] Collecting user feedback

[0763] The smart glasses record the user's reaction to the suggested products (e.g., whether they purchase or try them on), and then send the feedback data to the server. The input is the user's reaction, and the output is the feedback data sent to the server. Specifically, the glasses record the user's interaction with sensors and send the data to the server using a network protocol.

[0764] Step 8:

[0765] Update analysis results

[0766] The server updates the sentiment analysis AI model based on the user's feedback data. This improves the accuracy of future suggestions. The input is the feedback data, and the output is an updated sentiment analysis model. Specifically, the server analyzes the feedback data and partially retrains the existing model.

[0767] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0768] This invention relates to a system that analyzes a user's emotions in real time using an emotion engine and customizes a virtual shopping experience based on the analysis results. Specifically, this system includes a terminal that captures the user's facial expressions and voice data, a communication means for transmitting the data to a server, and a function for recognizing the user's emotions using the emotion engine.

[0769] System configuration and processing flow

[0770] composition

[0771] The system consists of the following main components:

[0772] 1. Server

[0773] Emotion analysis AI

[0774] Database

[0775] Emotion Engine

[0776] 2. Terminal

[0777] Camera and microphone

[0778] Data Transmission Module

[0779] Virtual shopping environment interface

[0780] 3. Means of communication

[0781] Network Infrastructure

[0782] Program processing

[0783] 1. Initialize the server

[0784] The server loads the sentiment analysis model, emotion engine, and virtual shopping environment, including connections to machine learning models and necessary databases.

[0785] 2. Initialize the device

[0786] The device checks the camera and microphone and displays the message "Do you want to start shopping online?" The user allows the use of the camera and microphone.

[0787] 3. Collection of User Data

[0788] When a user allows access to the camera and microphone, the device captures the user's facial expressions and voice data in real time and sends it to the server.

[0789] 4. Emotion analysis

[0790] The server's emotion engine analyzes the received facial expression and voice data to infer the user's emotions (e.g., joy, excitement, frustration) in real time. The analysis results are recorded in a database.

[0791] 5. Customize your experience

[0792] The server uses the sentiment analysis results and past shopping history to generate personalized product suggestions and special offers for the user, using machine learning algorithms.

[0793] 6. Display of proposals

[0794] The server sends the generated proposals and offers to the terminal, which displays this information on the user's screen. The user then confirms the proposed products and offers.

[0795] 7. User Feedback

[0796] The device records the user's reactions to the proposed products and offers (e.g., clicks, try-ons, purchases) and sends them to the server.

[0797] 8. Update analysis results

[0798] The server analyzes the user's reaction data sent from the device and stores it in a database as feedback. It then updates the sentiment analysis model and product recommendation algorithm. This update improves the accuracy of future recommendations.

[0799] Specific examples

[0800] Scenario: User wants to try on a virtual item

[0801] 1. Initialize the server

[0802] The server loads the avatar and clothing data for virtual try-on.

[0803] 2. Initialize the device

[0804] The device displays a virtual fitting room interface to users wearing a VR headset.

[0805] 3. Collection of User Data

[0806] While the user looks at themselves in the mirror, a camera and microphone capture their facial expressions and voice, sending the data to a server.

[0807] 4. Emotion analysis

[0808] The server's emotion engine analyzes the user's facial expression data and voice data to analyze emotions (e.g., surprise, satisfaction) in real time. The analysis results are recorded in a database.

[0809] 5. Customize your experience

[0810] If the user is satisfied, the server generates and sends data to the terminal to suggest clothes of a similar style, which the terminal then displays to the user.

[0811] 6. User Feedback

[0812] The user tries on the suggested clothes and responds by commenting or purchasing them, and the device records these responses and sends them to the server.

[0813] 7. Update analysis results

[0814] The server analyzes the user's feedback data, creates a new database entry as feedback, and updates the sentiment analysis model and suggestion algorithm to improve the accuracy of future suggestions.

[0815] In this way, users can enjoy a real-time, personalized online shopping experience, and the system continuously learns to provide a more optimized user experience.

[0816] The processing flow will be explained below.

[0817] Step 1:

[0818] The server loads the sentiment analysis model, emotion engine, and virtual shopping environment, a process that includes setting the parameters of the machine learning model and connecting to the relevant databases.

[0819] Step 2:

[0820] The device checks the camera and microphone and displays the message "Do you want to start shopping online?" The user allows the use of the camera and microphone.

[0821] Step 3:

[0822] The device captures the user's facial expressions with a camera and collects audio data with a microphone. This capture process is carried out periodically and the data is sent to a server in real time.

[0823] Step 4:

[0824] The server's emotion engine analyzes the received facial expression data and voice data to infer the user's emotions (e.g., joy, excitement, dissatisfaction) in real time. The analysis results are recorded in a database.

[0825] Step 5:

[0826] The server uses the results of the sentiment analysis and past shopping history to generate personalized product suggestions and special offers for the user, using machine learning algorithms.

[0827] Step 6:

[0828] The server sends the generated proposals and offers to the device, which then displays the information on the user's screen. The user can then review the proposed products and offers and take action as necessary.

[0829] Step 7:

[0830] The user's reactions to the proposed products and offers (e.g., clicks, try-ons, purchases) are recorded by the device and transmitted to the server in real time.

[0831] Step 8:

[0832] The server analyzes the user's response data and creates new database entries as feedback. This feedback data is an important factor in future analysis and proposal generation.

[0833] Step 9:

[0834] The server updates the sentiment analysis model and product recommendation algorithm based on the feedback data, improving the accuracy of future recommendations and optimizing the user experience.

[0835] Step 10:

[0836] The terminal will continue to present new suggestions and offers to the user based on their emotions, and the whole process will be repeated in real time, allowing the user to continuously enjoy the virtual shopping experience.

[0837] Example 2

[0838] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0839] Conventional virtual shopping systems have struggled to provide a personalized experience based on users' emotions and preferences. They also lacked the means to collect real-time user feedback and improve the system based on it. This resulted in a uniform shopping experience, making it difficult to improve user satisfaction.

[0840] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for loading an emotion analysis model, means for checking the camera and microphone and confirming with the user whether to start online shopping, means for capturing the user's facial expression and voice data in real time and sending it to the server, means for analyzing the user's emotions in real time, means for generating personalized product suggestions and special offers based on the analysis results and past shopping history, means for displaying the generated suggestions and offers on the user terminal, and means for collecting user feedback and updating the analysis results and the suggestion algorithm. This makes it possible to provide a personalized shopping experience based on the user's emotions and preferences and to continuously improve the system.

[0841] An "emotion analysis model" is a machine learning model that predicts a user's emotions in real time based on their facial expressions and voice data.

[0842] A "camera" is a photographing device for capturing the user's facial expressions.

[0843] A "microphone" is a voice recording device for capturing the user's voice data.

[0844] "Server" refers to a computer system that manages and operates the emotion analysis model, emotion engine, and virtual shopping environment.

[0845] An "emotion engine" is software that analyzes facial expressions and voice data to predict emotions in real time.

[0846] The "database" is a system for storing and managing user sentiment analysis results, shopping history, feedback data, etc.

[0847] A "virtual shopping environment" is a virtual environment in which users can experience shopping online.

[0848] "Communication means" refers to the network infrastructure for transmitting the user's facial expressions and voice data from the terminal to the server.

[0849] "Feedback" refers to the reactions and actions that users take in response to proposed products or offers.

[0850] "Product suggestions" refers to suggesting personalized products to users based on the results of emotional analysis of the user and their past shopping history.

[0851] "Special Offer" refers to a specific discount or promotion that is customized for you.

[0852] A "suggestion algorithm" is a computational means for generating optimal product suggestions and special offers based on the results of a user's sentiment analysis and past history.

[0853] The present invention relates to a system for analyzing a user's emotions in real time and customizing a virtual shopping experience based on the analysis results. A specific embodiment of this system will be described below.

[0854] System configuration

[0855] This system mainly consists of three main components: a server, a terminal, and a communication means.

[0856] 1. Server

[0857] Sentiment analysis model: Refers to a neural network model built using a machine learning library such as TensorFlow.

[0858] Emotion engine: Software that uses OpenAI APIs and other tools to analyze a user's facial expressions and voice data.

[0859] Database: We use a database management system such as MySQL to store and manage user sentiment analysis results, shopping history, and feedback data.

[0860] 2. Terminal

[0861] Camera: A photographic device for capturing the user's facial expressions.

[0862] Microphone: A voice recording device for capturing the user's voice data.

[0863] Data transmission module: A communication module for transmitting facial expression data and voice data to the server.

[0864] Virtual shopping environment interface: The interface through which the user operates, for example, using a VR headset or touchscreen.

[0865] 3. Means of communication

[0866] Network infrastructure: A means of transmitting data from a terminal to a server using communication infrastructure such as the Internet.

[0867] Program processing

[0868] The programs in this system perform the following basic tasks:

[0869] 1. Loading the sentiment analysis model

[0870] The server uses TensorFlow to load a sentiment analysis model, which includes a pre-trained neural network.

[0871] 2. Check your camera and microphone

[0872] The device checks whether the camera and microphone are properly connected, displays a message to the user asking "Do you want to start shopping online?" and asks for permission.

[0873] 3. Data Capture

[0874] The camera captures the user's facial expressions and the microphone collects audio data, which is sent to a server in real time.

[0875] 4. Emotion analysis

[0876] The server's emotion engine analyzes the received facial expression and voice data to infer the user's emotion in real time. For example, it can infer the emotion "satisfied" by analyzing a smiling face and a calm voice.

[0877] 5. Product proposal generation

[0878] The server generates personalized product suggestions and special offers based on the sentiment analysis results and past shopping history, using machine learning algorithms such as Scikit-learn.

[0879] 6. Display of proposals

[0880] The server sends the generated recommendations to the device, which displays them on the user's screen, allowing the user to view the recommendations, click, and purchase.

[0881] 7. Gathering Feedback

[0882] The device records the user's reactions (clicks, trying on, purchases, etc.) and sends them to the server, which analyzes them and updates the sentiment analysis model and proposed algorithm.

[0883] Specific examples

[0884] For example, in the scenario where a user is trying on new clothes while shopping online:

[0885] 1. Initialize the server

[0886] Load the TensorFlow model and emotion engine and connect to the database.

[0887] 2. Initialize the device

[0888] It checks your camera and microphone and displays the message "Do you want to start shopping online?"

[0889] 3. Data Capture

[0890] As the user tries on new clothes, the camera and microphone capture facial expressions and voice data, which are then sent to a server.

[0891] 4. Emotion analysis

[0892] The server receives the captured data and uses an emotion analysis model to infer the emotion "satisfied."

[0893] 5. Product proposal generation

[0894] The server suggests clothes with a similar style to the user based on the emotion of "satisfaction" and past purchase history.

[0895] 6. Display of proposals

[0896] The server sends the proposal to the terminal, which displays it on the user's screen.

[0897] 7. Gathering Feedback

[0898] The user clicks on the suggested clothing and reacts by trying it on or purchasing it, and the device records the reaction and sends it to the server.

[0899] Prompt Sentence Examples

[0900] The following prompt sentences are fed into the generative AI model to perform sentiment analysis:

[0901] A user is trying on new clothes and smiling as he looks at himself in the mirror. His voice is calm and relaxed. How should the server's emotion engine interpret this data and what product recommendations should it make?

[0902] In this way, the system can analyze user sentiment and provide a personalized shopping experience, and it also makes continuous improvements through feedback to provide a more optimized user experience.

[0903] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0904] Step 1: Initialize the server

[0905] explanation

[0906] At system startup, the server does the following:

[0907] Specific actions

[0908] 1. Loading the sentiment analysis model

[0909] Load a sentiment analysis model using TensorFlow. As input, a pre-trained model file is required. As output, the model is loaded into memory and ready for analysis.

[0910] Example: Read the file path of a model and load it using a TensorFlow function.

[0911] 2. Activating the Emotional Engine

[0912] Starts the emotion engine and waits for a connection. It uses the emotion engine's configuration file as input and the engine is started and starts accepting requests as output.

[0913] Example: Initialize the engine by reading the OpenAI API key and necessary configuration information.

[0914] 3. Setting up a virtual shopping environment

[0915] It connects to the required databases and sets up a virtual shopping environment, taking database configuration information and connection credentials as input, and an established database connection ready for operation as output.

[0916] Example: Use MySQL connection information to connect to a database and cache product information and user history.

[0917] Step 2: Reset the device

[0918] explanation

[0919] To provide an interface with the user, the terminal:

[0920] Specific actions

[0921] 1. Hardware check

[0922] Checks whether the camera and microphone are properly connected. It takes hardware status information as input and displays the connection status as output.

[0923] Example: Display a message to the user asking "Do you want to turn on your camera?" and ask for permission.

[0924] 2. Interface display

[0925] Prompts the user with the message "Start shopping online?" Uses user interface configuration information as input and provides the message as output.

[0926] Example: Asking the user for permission using a GUI or VR interface.

[0927] Step 3: Collect user data

[0928] explanation

[0929] The device collects user data in real time and sends it to the server:

[0930] Specific actions

[0931] 1. Data Capture

[0932] The camera continuously captures the user's facial expressions, using real-time video data as input and generating facial expression data as output.

[0933] Example: Activating the camera module to analyze facial expressions frame by frame.

[0934] 2. Data Transmission

[0935] The microphone collects the user's voice data and sends it to the server. It uses real-time voice data as input and sends voice data to the server as output.

[0936] Example: Start an audio capture module and periodically upload audio data to a server.

[0937] Step 4: Sentiment Analysis

[0938] explanation

[0939] The server analyzes the user's emotions based on the received data:

[0940] Specific actions

[0941] 1. Data Reception

[0942] It receives facial expression data and voice data from the terminal. It receives data sent from the terminal and data packets from the relay server as input, and stores the data in an analysis buffer as output.

[0943] Example: Receive data at a server API endpoint and pass it to an analytics engine.

[0944] 2. Emotional inference

[0945] The emotion engine analyzes facial expression and voice data to infer emotions in real time, using the received data as input and generating inferred emotions as output.

[0946] Example: Transferring a smiling face and a calm voice to infer the emotion "satisfied."

[0947] 3. Data Recording

[0948] Record the analysis results in a database. The sentiment analysis results are used as input and a new record is added to the database as output.

[0949] Example: Inserting inferred sentiment into MySQL.

[0950] Step 5: Customize the experience

[0951] explanation

[0952] The server generates personalized suggestions for the user based on the analysis results:

[0953] Specific actions

[0954] 1. Data analysis

[0955] Analyzes the sentiment analysis results and past shopping history. The historical information retrieved from the database and the analysis results are used as input, and the analysis results are generated as output.

[0956] Example: Inferring user preferences by combining past purchase history and current sentiment analysis results.

[0957] 2. Proposal generation

[0958] Generate personalized product recommendations and special offers, using analysis results as input and recommendations as output.

[0959] Example: Using Scikit-learn algorithms to make optimal product recommendations.

[0960] Step 6: View the proposal

[0961] explanation

[0962] Display the suggestions on the device so the user can review them:

[0963] Specific actions

[0964] 1. Submit a proposal

[0965] Send the generated proposal or offer to the device, using the proposal as input and successful transmission as output.

[0966] Example: Send proposal data in JSON format to the device.

[0967] 2. User Interface

[0968] The proposal is displayed on the user's screen. The proposal is used as input, and a visual display is produced as output that the user can confirm.

[0969] Example: Displaying product suggestions using a GUI or VR interface.

[0970] Step 7: User feedback

[0971] explanation

[0972] Collect the user's responses to the suggestions and send them to the server:

[0973] Specific actions

[0974] 1. Feedback Collection

[0975] Record user responses (clicks, try-ons, purchases, etc.) Uses user action data as input and generates feedback data as output.

[0976] Example: Storing a user's click history or purchase history.

[0977] 2. Data Transmission

[0978] The collected feedback data is sent to the server, which uses the feedback data as input and completes sending it to the server as output.

[0979] Example: Uploading feedback data to a server using an API.

[0980] Step 8: Update the analysis results

[0981] explanation

[0982] The server uses the feedback to update the model and algorithms to improve accuracy:

[0983] Specific actions

[0984] 1. Data Analysis

[0985] Analyze the received feedback data, using the feedback data as input and producing the analysis results as output.

[0986] Example: Statistical analysis of user feedback data to extract trends.

[0987] 2. Model Update

[0988] Update sentiment analysis models and product recommendation algorithms: new models and algorithms are generated using the analysis results as input and output.

[0989] Example: Retraining the sentiment analysis model and optimizing the proposed algorithm.

[0990] Through the above processing steps, the system can analyze user sentiment in real time and provide a personalized shopping experience.

[0991] (Application example 2)

[0992] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0993] Conventional online shopping systems have the problem that they do not provide product recommendations that fully understand the user's preferences, resulting in poor user experience. They also have been unable to understand the user's emotions and personalize the shopping experience in real time based on those emotions. Therefore, there is a need to provide a personalized shopping experience that reflects the user's interests.

[0994] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0995] In this invention, the server includes means for analyzing user emotions in real time, means for customizing the virtual shopping experience based on the analysis results, means for collecting user feedback and updating the analysis results, means for recognizing the user's emotions from the captured data and generating emotion-based product suggestions and special offers, and means for displaying the generated suggestions and offers on the user's terminal, thereby enabling personalized product suggestions based on the user's emotions.

[0996] "Means for analyzing user emotions in real time" refers to a function that analyzes the user's facial expressions and voice data obtained using a camera or microphone, etc., and instantly recognizes the user's emotions at that time.

[0997] "Means to customize the virtual shopping experience" refers to the ability to optimize the shopping environment and recommendations for individual users based on the results of sentiment analysis.

[0998] "Means of collecting user feedback and updating analysis results" refers to the function of recording user reactions and behavioral data and using it to improve emotion analysis models and product recommendation algorithms.

[0999] "Means of recognizing a user's emotions from captured data and generating emotion-based product suggestions and special offers" refers to a function that selects and suggests products and services suitable for the user based on emotional data obtained by analyzing collected facial expressions and voice data.

[1000] "Means for displaying generated suggestions and offers on the user's device" refers to the function of displaying product suggestions and special offers generated through sentiment analysis on the user's device, such as a smartphone, smart glasses, or head-mounted display.

[1001] A "terminal for displaying a virtual fitting room interface" is a device that provides an interface for users to try on clothes in a virtual space, and in many cases a head-mounted display is used.

[1002] "Means of recording specific user responses from captured data and using them as feedback" refers to the function of recording user behavioral data such as clicks and purchases and using it as learning data for algorithms.

[1003] This invention is a system that analyzes a user's emotions in real time and customizes a virtual shopping experience based on the analysis results. This system is composed of an emotion analysis engine, a server, a database, a camera, a microphone, a terminal, and communication means. Detailed embodiments of each component of this invention will be described below.

[1004] System configuration

[1005] 1. Server

[1006] Emotion analysis engine: Analyzes the user's facial expression and voice data to recognize emotions in real time.

[1007] Database: Stores analysis results and user feedback data and is used to train machine learning algorithms.

[1008] Product recommendation engine: Generates product suggestions and special offers based on sentiment analysis results.

[1009] 2. Terminal

[1010] Camera and microphone: Captures the user's facial expressions and voice data.

[1011] Data transmission module: Sends the captured data to the server.

[1012] Virtual Shopping Environment Interface: Providing users with a virtual shopping experience via a VR headset or smart glasses.

[1013] 3. Means of communication

[1014] Network infrastructure: Carries out data communication between user terminals and servers.

[1015] System processing flow

[1016] 1. Initialization

[1017] The server loads the necessary software, including a sentiment analysis engine and a product recommendation engine, while the device checks the camera and microphone and displays a confirmation message to the user to begin the virtual shopping experience. The user then authorizes the use of the camera and microphone.

[1018] 2. Collection of User Data

[1019] The device captures the user's facial expressions and voice data in real time and sends the data to a server. The camera and microphone use OpenCV and other image and voice processing libraries.

[1020] 3. Emotion analysis

[1021] The server's emotion analysis engine analyzes the received data and recognizes the user's emotions using a machine learning model, and the results are stored in a database.

[1022] 4. Presenting the generated proposals

[1023] The server generates optimal product suggestions and special offers for the user based on the sentiment analysis results and database. The generated suggestions are sent to the terminal and displayed on the user's device. The user can then check the suggested products in the virtual store.

[1024] Example scenario

[1025] For example, if a user is trying on a dress in a virtual fitting room, a camera and microphone capture the user's facial expressions and voice and send them to the server. If the server detects that the user is pleased with the dress, it can suggest other dresses with the same design but in different colors, as well as matching accessories.

[1026] Prompt Sentence Examples

[1027] plain text

[1028] Suggest items related to products that users enjoy

[1029] This invention makes it possible to provide a personalized virtual shopping experience that is tailored to the user's emotions. Users can receive optimal product suggestions based on their emotions in real time, improving their shopping experience.

[1030] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1031] Step 1:

[1032] Initializing the Server

[1033] The server loads the necessary software components, such as the sentiment analysis AI, emotion engine, database, and product recommendation engine. The initialization process checks the connections between each component and ensures that the system works properly. Machine learning models and necessary algorithms are also loaded at this stage. The server is then ready to receive and analyze data from users.

[1034] Step 2:

[1035] Initializing the device

[1036] The device checks the camera and microphone and displays a message to the user saying "Do you want to start shopping online?", at which point the user is prompted to allow the use of the camera and microphone. If permission is granted, the device prepares to capture data through the camera and microphone.

[1037] Step 3:

[1038] User Data Collection

[1039] The device captures the user's facial expressions and voice data in real time and sends the obtained data to the server. During this process, the camera takes a picture of the user's face and the microphone records the user's voice. The captured data is sent to the server via the network. The input data is image data and voice data, and the output is the data sent to the server.

[1040] Step 4:

[1041] Emotion analysis

[1042] The server inputs the received facial expression data and voice data into an emotion analysis engine, which analyzes the user's emotions in real time. This analysis is performed using a machine learning model. The emotion analysis engine uses image processing and voice analysis algorithms to estimate the user's emotions from the data. The input data is facial expression data and voice data, and the output is the emotion analysis result. The analysis results are recorded in a database.

[1043] Step 5:

[1044] Generate product suggestions

[1045] The server generates optimal product suggestions and special offers for the user based on the sentiment analysis results and a database of past users. This step often uses a generative AI model, which allows the server to create products and offers that match the user's current emotions. The input data are the sentiment analysis results and past user data, and the output is the generated product suggestions and special offers.

[1046] Step 6:

[1047] Viewing Proposals

[1048] The server sends the generated product suggestions and special offers to the terminal. The terminal displays the suggestions on the user's screen. The user can then check the suggested products in the virtual store. The input data are the product suggestions and special offers, and the output is the information displayed on the user's screen.

[1049] Step 7:

[1050] Collecting User Feedback

[1051] The device records the user's reactions to the proposed products and offers, such as clicking, trying them on, and purchasing, and sends them to the server. This data will be used to improve future sentiment analysis models and product recommendation engines. The input data is the user's reaction data, and the output is feedback data sent to the server.

[1052] Step 8:

[1053] Update analysis results

[1054] The server analyzes the feedback data and creates new database entries. This data is used to train the sentiment analysis model and product recommendation algorithm to improve the accuracy of future recommendations. The input data is the feedback data, and the output is the updated analysis model and algorithm.

[1055] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1056] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1057] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1058] [Third embodiment]

[1059] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1060] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1061] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1062] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1063] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1064] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1065] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1066] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1067] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1068] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1069] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1070] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1071] This invention relates to a system that analyzes a user's emotions in real time and customizes a virtual shopping experience based on the analysis results. Specifically, the system includes a terminal for capturing the user's facial expression and voice data and a communication means for transmitting the data to a server. The server analyzes the user's emotions based on the received data and generates product suggestions and special offers based on the analysis results. The generated suggestions and offers are displayed on the user's terminal, and the analysis results are updated based on the user's feedback to improve the accuracy of future suggestions.

[1072] System configuration and processing flow

[1073] composition

[1074] The system consists of the following main components:

[1075] 1. Server

[1076] Emotion analysis AI

[1077] Database

[1078] 2. Terminal

[1079] Camera and microphone

[1080] Data Transmission Module

[1081] Virtual shopping environment interface

[1082] 3. Means of communication

[1083] Network Infrastructure

[1084] Program processing

[1085] 1. Initialize the server

[1086] The server loads the sentiment analysis model and the virtual shopping environment, which are generated from historical user data.

[1087] 2. Initialize the device

[1088] The device checks the camera and microphone and displays the message "Do you want to start shopping online?" to the user.

[1089] 3. Collection of User Data

[1090] When a user allows access to the camera and microphone, the device captures the user's facial expressions and voice data and sends it to the server.

[1091] 4. Emotion analysis

[1092] The server analyzes the facial expression and voice data sent from the device and infers the user's emotions in real time. The analysis results are recorded in a database.

[1093] 5. Customize your experience

[1094] The server generates personalized product suggestions and special offers for the user based on the results of sentiment analysis and past shopping history.

[1095] The generated proposals and offers are sent to the terminal and displayed on the user's screen.

[1096] 6. User Feedback

[1097] The user responds to the suggested product or offer (e.g., clicks, tries on, purchases).

[1098] The terminal records this response and sends it to the server as feedback data.

[1099] 7. Update analysis results

[1100] The server updates the sentiment analysis model based on the user's feedback data to improve the accuracy of suggestions for the next shopping session.

[1101] Specific examples

[1102] Scenario: User wants to try on a virtual item

[1103] 1. Initialize the server

[1104] The server loads the avatar and clothing data for virtual try-on.

[1105] 2. Initialize the device

[1106] The device displays a virtual fitting room interface to users wearing a VR headset.

[1107] 3. Collection of User Data

[1108] While the user looks at themselves in the mirror, a camera and microphone capture their facial expressions and voice, sending the data to a server.

[1109] 4. Emotion analysis

[1110] The server analyzes the user's emotions (e.g., surprise, satisfaction) in real time based on facial expression data and voice data.

[1111] 5. Customize your experience

[1112] If the user is satisfied, the server generates data for suggesting clothes of a similar style and transmits it to the terminal.

[1113] The terminal displays these suggestions to the user.

[1114] 6. User Feedback

[1115] Users try on the suggested clothes and respond by rejecting them or purchasing them.

[1116] The device records this response and sends it to the server.

[1117] 7. Update analysis results

[1118] The server integrates user feedback to update the analytical model, improving the accuracy of future proposals.

[1119] In this way, users can enjoy a real-time, personalized online shopping experience, and the system continuously learns and optimizes the user experience.

[1120] The processing flow will be explained below.

[1121] Step 1:

[1122] The server loads the sentiment analysis model and the virtual shopping environment, including the machine learning models and connections to the necessary databases.

[1123] Step 2:

[1124] The device checks the camera and microphone and displays the message "Do you want to start shopping online?" The user allows the use of the camera and microphone.

[1125] Step 3:

[1126] The device periodically captures the user's facial expressions with a camera and collects audio data with a microphone, which is then sent to a server every few seconds.

[1127] Step 4:

[1128] The server analyzes the received facial expression and voice data to infer the user's emotions (e.g., joy, excitement, dissatisfaction) in real time. The analysis results are recorded in a database.

[1129] Step 5:

[1130] The server uses the analysis results and past shopping history to generate personalized product suggestions and special offers for the user, using machine learning algorithms.

[1131] Step 6:

[1132] The server sends the generated proposals and offers to the terminal, which displays this information on the user's screen. The user then confirms the proposed products and offers.

[1133] Step 7:

[1134] The user's reactions to the proposed products and offers (e.g., clicking, trying on, purchasing) are recorded by the device and transmitted to the server.

[1135] Step 8:

[1136] The server analyzes the user's reaction data sent from the terminal and creates a new database entry as feedback data.

[1137] Step 9:

[1138] The server updates the sentiment analysis model and product recommendation algorithm based on the feedback data, improving the accuracy of future recommendations.

[1139] Step 10:

[1140] The device will continue to present new sentiment-based suggestions and offers to the user, and the whole process will be repeated in real time to optimize the user experience.

[1141] Example 1

[1142] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1143] In conventional online shopping, it was difficult to analyze user emotions in real time and personalize the shopping experience based on that. Furthermore, there was a lack of a mechanism for collecting user feedback and updating the analysis results, which meant that product recommendations did not adequately reflect the user's interests and satisfaction. This resulted in a poor user experience and an inability to fully stimulate purchasing motivation.

[1144] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1145] In this invention, the server includes means for loading and initializing an emotion analysis model, means for collecting facial expression and voice data of a user, means for transmitting the collected data to the server, means for analyzing the user's emotions in real time based on the data received by the server, means for recording the analysis results in a database, means for generating personalized product suggestions and special offers based on the emotion analysis results and past shopping history, means for displaying the generated suggestions and offers on the user terminal, means for collecting user feedback and transmitting the feedback data to the server, and means for updating the emotion analysis model based on the feedback data. This makes it possible to analyze the user's emotions in real time and provide a highly personalized shopping experience that reflects the results.

[1146] An "emotion analysis model" is an algorithm or program that analyzes a user's facial expressions and voice data to infer their emotional state.

[1147] A "server" is a computer system that processes and stores data and provides that data to other computers or devices over a network.

[1148] "User's facial expression and voice data" refers to digital data of human facial movements and voice signals captured using a camera and microphone.

[1149] A "database" is a structured collection of data for efficiently storing, retrieving, and updating information.

[1150] "Product suggestion" refers to the act of introducing products that are likely to interest a user based on the results of an emotional analysis of the user and their past shopping history, and the data used to do so.

[1151] "Special Offers" are commercial offers such as special prices or promotions based on the user's interests.

[1152] "Feedback data" refers to data based on user behavior and reactions that indicate interest in and intent to purchase product suggestions and special offers.

[1153] "Personalization" means customizing product proposals and services to suit the characteristics and preferences of individual users.

[1154] "Real-time" is a concept that refers to processing and reactions occurring almost simultaneously with the collection of data.

[1155] This invention relates to a system that analyzes a user's emotions in real time and customizes a virtual shopping experience based on the analysis results. Specifically, the system includes a terminal for capturing the user's facial expression and voice data and a communication means for transmitting the data to a server. The server analyzes the user's emotions based on the received data and generates product suggestions and special offers based on the analysis results. The generated suggestions and offers are displayed on the user's terminal, and the analysis results are updated based on user feedback, thereby improving the accuracy of future suggestions.

[1156] The main components of the system are as follows:

[1157] 1. Server

[1158] Sentiment analysis model (e.g., an AI model using TensorFlow)

[1159] Database (e.g. MySQL)

[1160] 2. Terminal

[1161] Camera and microphone (e.g., a standard webcam or microphone)

[1162] Data transmission module (e.g. Wi-Fi or Bluetooth)

[1163] Virtual shopping environment interface (e.g., built with Unity)

[1164] 3. Means of communication

[1165] Network infrastructure (e.g., 5G, fiber optic connections)

[1166] Specifically, the server first loads the emotion analysis model and virtual shopping environment. This requires initializing an AI model using TensorFlow and establishing a connection to a database of products taken from past user data. The device then diagnoses the camera and microphone, checks their operation, and displays a message to the user asking, "Do you want to start online shopping?" If the user allows access to the camera and microphone, the device captures the user's facial expressions and voice data and sends it to the server.

[1167] The server analyzes the facial expression and voice data sent from the device and infers the user's emotions in real time. The analysis results are recorded in a database. The server then generates personalized product suggestions and special offers for the user based on the emotion analysis results and past shopping history. The generated suggestions and offers are sent to the device and displayed on the user's screen. When the user responds to the suggested products or offers (e.g., clicking, trying them on, purchasing), the device records this response and sends it to the server as feedback data. Finally, the server updates the emotion analysis model based on the user's feedback data to improve the accuracy of suggestions for the next shopping session.

[1168] As a specific example, consider a user using a virtual fitting service. The server loads the virtual fitting avatar and clothing data, and the device displays a virtual fitting room interface to the user wearing a VR headset. While the user looks at themselves in the mirror, a camera and microphone capture their facial expressions and voice, and send the data to the server. The server analyzes the user's emotions (e.g., surprise, satisfaction) in real time based on the facial expression and voice data. The server then suggests more similar styles of clothing based on the emotion analysis results, and the device displays these suggestions to the user. When the user tries on the suggested clothing and responds by rejecting it or purchasing it, the device records this response and sends it to the server. The server integrates the user's feedback to update the analysis model, improving the accuracy of future suggestions.

[1169] Examples of prompt statements

[1170] Prompt: "Analyze the user's facial expressions and voice in a virtual fitting room to suggest outfits that match their emotions."

[1171] Input data example: Facial expression data [image file], voice data ["I like this outfit!"]

[1172] In this way, users can enjoy a real-time, personalized online shopping experience, and the system can continuously learn and optimize the user experience.

[1173] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1174] Step 1:

[1175] Initializing the Server

[1176] The server loads the sentiment analysis model and the virtual shopping environment. Specifically, it first loads the sentiment analysis model parameters into memory using TensorFlow. Next, it establishes a connection to a product database (e.g., MySQL) based on past user data and loads the necessary product information into memory. This allows the server to perform sentiment analysis and product suggestions in real time.

[1177] Input: TensorFlow model parameter file, product database connection information

[1178] Output: Initialized sentiment analysis model, product data loaded into memory.

[1179] Step 2:

[1180] Initializing the device

[1181] The device diagnoses the camera and microphone to ensure they are working properly. It then displays a message to the user asking, "Do you want to start online shopping?" If the user responds, the device prepares to send the input to the server.

[1182] Specific actions include checking the camera video feed and checking the volume level of the microphone input.

[1183] Input: User actions, device camera and microphone diagnostic results

[1184] Output: User camera and microphone permissions, confirmation to start shopping

[1185] Step 3:

[1186] User Data Collection

[1187] Once the user allows the use of the camera and microphone, the device will activate them and continuously capture the user's facial expressions and voice data, which will be transmitted to the server in real time. The camera will capture high-resolution images, and the microphone will record clear audio.

[1188] Input: Video data from camera, audio data from microphone

[1189] Output: Facial expression data and voice data sent to the server

[1190] Step 4:

[1191] Emotion analysis

[1192] The server receives and analyzes the facial expression and voice data sent from the device. It passes the facial expression images and voice data as input to an emotion analysis model, which then infers the user's emotions in real time. The analysis results are then recorded in a database.

[1193] Input: Emotion analysis model loaded from database, received facial expression data and speech data

[1194] Output: User's emotional state, analysis results recorded in a database

[1195] Step 5:

[1196] Customize your experience

[1197] The server generates personalized product suggestions and special offers for the user based on the results of sentiment analysis and past shopping history. The generated suggestions and offers are sent in JSON format to the device, which then displays them on the user's screen.

[1198] Input: Sentiment analysis results, past shopping history, product database

[1199] Output: Generated product suggestions, special offers, and display data sent to the device

[1200] Step 6:

[1201] User Feedback

[1202] When users interact with the proposed products or offers (e.g., click, try on, purchase), the device records these interactions, and the recorded feedback data is sent to the server in real time.

[1203] Input: User interaction data

[1204] Output: Feedback data sent to the server

[1205] Step 7:

[1206] Update analysis results

[1207] The server receives user feedback data and updates the sentiment analysis model based on that data. This update improves the accuracy of suggestions for the next shopping session by optimizing the model parameters and algorithms.

[1208] Input: Feedback data, sentiment analysis model

[1209] Output: Updated sentiment analysis model, improving the accuracy of the next proposal

[1210] Through these steps, the system can analyze user sentiment in real time, generate and present product suggestions and special offers tailored to each user, and continuously learn to optimize the user experience.

[1211] (Application example 1)

[1212] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1213] Conventional virtual shopping experiences have had the challenge of being difficult to flexibly respond to individual users' emotions and preferences. In particular, there has been a lack of systems that can analyze a user's emotional state in real time and make product recommendations based on the results. This means that users are often left feeling dissatisfied because they are rarely presented with products that suit them at the right time. Furthermore, there has been a lack of mechanisms for effectively utilizing user feedback to improve the shopping experience.

[1214] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1215] In this invention, the server includes means for analyzing user emotions in real time, means for collecting user feedback and updating the analysis results, and means for capturing the user's facial expression and voice data using the smart glasses, which enables the generation of personalized product suggestions and special offers based on the user's emotions and feedback and displaying these suggestions on the display of the smart glasses, thereby providing the user with a more comfortable and efficient shopping experience.

[1216] "Means for analyzing a user's emotions in real time" refers to a technology that instantly determines the user's emotional state at any given time based on the facial expressions and voice data they show.

[1217] "Means for customizing virtual shopping experiences" refers to technology that individually adjusts and optimizes the shopping experience based on user sentiment analysis and feedback.

[1218] The "means of collecting user feedback and updating analysis results" refers to a technology that continuously records the reactions and behaviors of users and uses them to improve the accuracy of emotion analysis models.

[1219] "Means for capturing a user's facial expression and voice data using smart glasses" refers to a technology that uses smart glasses with built-in cameras and microphones to obtain a user's facial expression and voice information.

[1220] The "communication means for transmitting the data to the server" refers to a technique for transmitting the captured data to the server via the Internet or other communication network.

[1221] The "means for generating personalized product proposals based on analysis results" refers to a technology that selects and proposes optimal products and offers individually based on the user's emotional state and past shopping history.

[1222] The "means for displaying the generated proposals on the display of the smart glasses" is a technology that uses the display of the smart glasses to visually present the generated product proposals or offers to the user.

[1223] "Means for recording user responses and transmitting them to a server" refers to technology that records the actions and responses that users show to displayed suggestions and offers and transmits them to a server via the Internet or other communications network.

[1224] This invention is a system that analyzes users' emotions in real time and customizes the virtual shopping experience based on the analysis results. Specifically, it uses smart glasses to capture the user's facial expressions and voice data, transmits them to a server for analysis, and generates individually optimized product recommendations.

[1225] System Configuration

[1226] server

[1227] Emotion analysis AI model: A model for analyzing a user's facial expressions and voice data to infer emotions. Models created with TensorFlow and PyTorch are used.

[1228] Database: A database for storing user data, suggestion data, and feedback data. MySQL or PostgreSQL can be used.

[1229] Communication method: Protocol used to send and receive data. WebSocket or gRPC is used.

[1230] Terminal (smart glasses)

[1231] Camera and microphone: Devices required to capture the user's facial expressions and voice data.

[1232] Display: A device used to display suggested products and offers to the user.

[1233] Data transmission module: A module for transmitting captured data to the server.

[1234] Program processing overview

[1235] 1. Initialize the server:

[1236] When the server starts, it loads the sentiment analysis model and resources of the virtual shopping environment, including the model generated based on historical user data.

[1237] 2. Initialize the device:

[1238] When the smart glasses start up, the user is prompted, "Would you like to enter the virtual store?"

[1239] If the user agrees, the camera and microphone will be enabled.

[1240] 3. Collection of User Data:

[1241] The smart glasses capture the user's facial expressions and voice and transmit them to a server in real time.

[1242] 4. Emotion analysis:

[1243] The server analyzes the user's emotions based on the received data, using an emotion analysis AI model, and stores the results in a database.

[1244] 5. Product proposal generation:

[1245] Based on the sentiment analysis results and past shopping history, the server generates personalized product suggestions.

[1246] 6. Displaying Proposals:

[1247] The generated suggestions are displayed on the smart glasses' display, and the user can refer to them while shopping.

[1248] 7. Collecting User Feedback:

[1249] The smart glasses record the user's response to the suggested products (e.g., whether they purchase or try them on) and send it to the server.

[1250] 8. Update analysis results:

[1251] The server updates the emotion analysis model based on user feedback data, which improves the accuracy of future suggestions.

[1252] Specific examples

[1253] When a user is searching for clothes in a virtual store, the smart glasses analyze the user's emotions and, if they are satisfied, suggest, "Would you like to see more similar styles?" Conversely, if the user appears dissatisfied, they suggest, "Would you like to see other styles?"

[1254] Prompt Sentence Examples

[1255] "Analyzing the emotions users show during their virtual store experiences."

[1256] "Analyzes the user's emotions in real time based on facial expressions and voice data, and makes appropriate product recommendations."

[1257] This system allows users to receive personalized product suggestions in real time, and the system continuously learns based on feedback data, further improving the shopping experience in future purchases.

[1258] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1259] Step 1:

[1260] Initializing the Server

[1261] When the server starts up, it loads the emotion analysis model and resources for the virtual shopping environment. At this time, the emotion analysis AI model built based on past user data is loaded. The input is the server startup signal, and the output is that the emotion analysis model and resources are ready for use. Specifically, it retrieves the necessary configuration information from the database and loads the model and resources into memory.

[1262] Step 2:

[1263] Initializing the device

[1264] When the smart glasses start up, they prompt the user, "Would you like to enter the virtual store?" The input is the smart glasses' startup signal, and the output is the display of the prompt. Specifically, the system performs a startup check and then displays the prompt on the display.

[1265] Step 3:

[1266] User Data Collection

[1267] Once the user agrees to the prompt, the smart glasses' camera and microphone are enabled and begin capturing the user's facial expressions and voice data, which are then transmitted to the server in real time. The input is the user's consent and the captured data, and the output is the data transmitted to the server. Specifically, the glasses enable the camera and microphone, continuously capture data, and transmit it using a network protocol.

[1268] Step 4:

[1269] Emotion analysis

[1270] The server analyzes the user's emotions in real time using an emotion analysis AI model based on the received facial expression and voice data. The analysis results are saved in a database. The input is the received data, and the output is the analysis results. Specifically, the facial expression data is passed to an image recognition model, and the voice data is passed to a voice recognition model to determine the emotion.

[1271] Step 5:

[1272] Generate product suggestions

[1273] The server generates individually optimized product recommendations based on the sentiment analysis results and the user's past shopping history. The inputs are the analysis results and shopping history, and the output is the generated product recommendations. Specifically, the server retrieves past history from the database and inputs it, along with the sentiment analysis results, into the product recommendation algorithm.

[1274] Step 6:

[1275] View Suggestions

[1276] The generated product suggestions are displayed on the display of the smart glasses. The input is the generated product suggestions, and the output is the visual information displayed on the display. Specifically, the suggestion data is converted into a display format and sent to the display for display.

[1277] Step 7:

[1278] Collecting user feedback

[1279] The smart glasses record the user's reaction to the suggested products (e.g., whether they purchase or try them on), and then send the feedback data to the server. The input is the user's reaction, and the output is the feedback data sent to the server. Specifically, the glasses record the user's interaction with sensors and send the data to the server using a network protocol.

[1280] Step 8:

[1281] Update analysis results

[1282] The server updates the sentiment analysis AI model based on the user's feedback data. This improves the accuracy of future suggestions. The input is the feedback data, and the output is an updated sentiment analysis model. Specifically, the server analyzes the feedback data and partially retrains the existing model.

[1283] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1284] This invention relates to a system that analyzes a user's emotions in real time using an emotion engine and customizes a virtual shopping experience based on the analysis results. Specifically, this system includes a terminal that captures the user's facial expressions and voice data, a communication means for transmitting the data to a server, and a function for recognizing the user's emotions using the emotion engine.

[1285] System configuration and processing flow

[1286] composition

[1287] The system consists of the following main components:

[1288] 1. Server

[1289] Emotion analysis AI

[1290] Database

[1291] Emotion Engine

[1292] 2. Terminal

[1293] Camera and microphone

[1294] Data Transmission Module

[1295] Virtual shopping environment interface

[1296] 3. Means of communication

[1297] Network Infrastructure

[1298] Program processing

[1299] 1. Initialize the server

[1300] The server loads the sentiment analysis model, emotion engine, and virtual shopping environment, including connections to machine learning models and necessary databases.

[1301] 2. Initialize the device

[1302] The device checks the camera and microphone and displays the message "Do you want to start shopping online?" The user allows the use of the camera and microphone.

[1303] 3. Collection of User Data

[1304] When a user allows access to the camera and microphone, the device captures the user's facial expressions and voice data in real time and sends it to the server.

[1305] 4. Emotion analysis

[1306] The server's emotion engine analyzes the received facial expression and voice data to infer the user's emotions (e.g., joy, excitement, frustration) in real time. The analysis results are recorded in a database.

[1307] 5. Customize your experience

[1308] The server uses the sentiment analysis results and past shopping history to generate personalized product suggestions and special offers for the user, using machine learning algorithms.

[1309] 6. Display of proposals

[1310] The server sends the generated proposals and offers to the terminal, which displays this information on the user's screen. The user then confirms the proposed products and offers.

[1311] 7. User Feedback

[1312] The device records the user's reactions to the proposed products and offers (e.g., clicks, try-ons, purchases) and sends them to the server.

[1313] 8. Update analysis results

[1314] The server analyzes the user's reaction data sent from the device and stores it in a database as feedback. It then updates the sentiment analysis model and product recommendation algorithm. This update improves the accuracy of future recommendations.

[1315] Specific examples

[1316] Scenario: User wants to try on a virtual item

[1317] 1. Initialize the server

[1318] The server loads the avatar and clothing data for virtual try-on.

[1319] 2. Initialize the device

[1320] The device displays a virtual fitting room interface to users wearing a VR headset.

[1321] 3. Collection of User Data

[1322] While the user looks at themselves in the mirror, a camera and microphone capture their facial expressions and voice, sending the data to a server.

[1323] 4. Emotion analysis

[1324] The server's emotion engine analyzes the user's facial expression data and voice data to analyze emotions (e.g., surprise, satisfaction) in real time. The analysis results are recorded in a database.

[1325] 5. Customize your experience

[1326] If the user is satisfied, the server generates and sends data to the terminal to suggest clothes of a similar style, which the terminal then displays to the user.

[1327] 6. User Feedback

[1328] The user tries on the suggested clothes and responds by commenting or purchasing them, and the device records these responses and sends them to the server.

[1329] 7. Update analysis results

[1330] The server analyzes the user's feedback data, creates a new database entry as feedback, and updates the sentiment analysis model and suggestion algorithm to improve the accuracy of future suggestions.

[1331] In this way, users can enjoy a real-time, personalized online shopping experience, and the system continuously learns to provide a more optimized user experience.

[1332] The processing flow will be explained below.

[1333] Step 1:

[1334] The server loads the sentiment analysis model, emotion engine, and virtual shopping environment, a process that includes setting the parameters of the machine learning model and connecting to the relevant databases.

[1335] Step 2:

[1336] The device checks the camera and microphone and displays the message "Do you want to start shopping online?" The user allows the use of the camera and microphone.

[1337] Step 3:

[1338] The device captures the user's facial expressions with a camera and collects audio data with a microphone. This capture process is carried out periodically and the data is sent to a server in real time.

[1339] Step 4:

[1340] The server's emotion engine analyzes the received facial expression data and voice data to infer the user's emotions (e.g., joy, excitement, dissatisfaction) in real time. The analysis results are recorded in a database.

[1341] Step 5:

[1342] The server uses the results of the sentiment analysis and past shopping history to generate personalized product suggestions and special offers for the user, using machine learning algorithms.

[1343] Step 6:

[1344] The server sends the generated proposals and offers to the device, which then displays the information on the user's screen. The user can then review the proposed products and offers and take action as necessary.

[1345] Step 7:

[1346] The user's reactions to the proposed products and offers (e.g., clicks, try-ons, purchases) are recorded by the device and transmitted to the server in real time.

[1347] Step 8:

[1348] The server analyzes the user's response data and creates new database entries as feedback. This feedback data is an important factor in future analysis and proposal generation.

[1349] Step 9:

[1350] The server updates the sentiment analysis model and product recommendation algorithm based on the feedback data, improving the accuracy of future recommendations and optimizing the user experience.

[1351] Step 10:

[1352] The terminal will continue to present new suggestions and offers to the user based on their emotions, and the whole process will be repeated in real time, allowing the user to continuously enjoy the virtual shopping experience.

[1353] Example 2

[1354] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1355] Conventional virtual shopping systems have struggled to provide a personalized experience based on users' emotions and preferences. They also lacked the means to collect real-time user feedback and improve the system based on it. This resulted in a uniform shopping experience, making it difficult to improve user satisfaction.

[1356] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for loading an emotion analysis model, means for checking the camera and microphone and confirming with the user whether to start online shopping, means for capturing the user's facial expression and voice data in real time and sending it to the server, means for analyzing the user's emotions in real time, means for generating personalized product suggestions and special offers based on the analysis results and past shopping history, means for displaying the generated suggestions and offers on the user terminal, and means for collecting user feedback and updating the analysis results and the suggestion algorithm. This makes it possible to provide a personalized shopping experience based on the user's emotions and preferences and to continuously improve the system.

[1357] An "emotion analysis model" is a machine learning model that predicts a user's emotions in real time based on their facial expressions and voice data.

[1358] A "camera" is a photographing device for capturing the user's facial expressions.

[1359] A "microphone" is a voice recording device for capturing the user's voice data.

[1360] "Server" refers to a computer system that manages and operates the emotion analysis model, emotion engine, and virtual shopping environment.

[1361] An "emotion engine" is software that analyzes facial expressions and voice data to predict emotions in real time.

[1362] The "database" is a system for storing and managing user sentiment analysis results, shopping history, feedback data, etc.

[1363] A "virtual shopping environment" is a virtual environment in which users can experience shopping online.

[1364] "Communication means" refers to the network infrastructure for transmitting the user's facial expressions and voice data from the terminal to the server.

[1365] "Feedback" refers to the reactions and actions that users take in response to proposed products or offers.

[1366] "Product suggestions" refers to suggesting personalized products to users based on the results of emotional analysis of the user and their past shopping history.

[1367] "Special Offer" refers to a specific discount or promotion that is customized for you.

[1368] A "suggestion algorithm" is a computational means for generating optimal product suggestions and special offers based on the results of a user's sentiment analysis and past history.

[1369] The present invention relates to a system for analyzing a user's emotions in real time and customizing a virtual shopping experience based on the analysis results. A specific embodiment of this system will be described below.

[1370] System configuration

[1371] This system mainly consists of three main components: a server, a terminal, and a communication means.

[1372] 1. Server

[1373] Sentiment analysis model: Refers to a neural network model built using a machine learning library such as TensorFlow.

[1374] Emotion engine: Software that uses OpenAI APIs and other tools to analyze a user's facial expressions and voice data.

[1375] Database: We use a database management system such as MySQL to store and manage user sentiment analysis results, shopping history, and feedback data.

[1376] 2. Terminal

[1377] Camera: A photographic device for capturing the user's facial expressions.

[1378] Microphone: A voice recording device for capturing the user's voice data.

[1379] Data transmission module: A communication module for transmitting facial expression data and voice data to the server.

[1380] Virtual shopping environment interface: The interface through which the user operates, for example, using a VR headset or touchscreen.

[1381] 3. Means of communication

[1382] Network infrastructure: A means of transmitting data from a terminal to a server using communication infrastructure such as the Internet.

[1383] Program processing

[1384] The programs in this system perform the following basic tasks:

[1385] 1. Loading the sentiment analysis model

[1386] The server uses TensorFlow to load a sentiment analysis model, which includes a pre-trained neural network.

[1387] 2. Check your camera and microphone

[1388] The device checks whether the camera and microphone are properly connected, displays a message to the user asking "Do you want to start shopping online?" and asks for permission.

[1389] 3. Data Capture

[1390] The camera captures the user's facial expressions and the microphone collects audio data, which is sent to a server in real time.

[1391] 4. Emotion analysis

[1392] The server's emotion engine analyzes the received facial expression and voice data to infer the user's emotion in real time. For example, it can infer the emotion "satisfied" by analyzing a smiling face and a calm voice.

[1393] 5. Product proposal generation

[1394] The server generates personalized product suggestions and special offers based on the sentiment analysis results and past shopping history, using machine learning algorithms such as Scikit-learn.

[1395] 6. Display of proposals

[1396] The server sends the generated recommendations to the device, which displays them on the user's screen, allowing the user to view the recommendations, click, and purchase.

[1397] 7. Gathering Feedback

[1398] The device records the user's reactions (clicks, trying on, purchases, etc.) and sends them to the server, which analyzes them and updates the sentiment analysis model and proposed algorithm.

[1399] Specific examples

[1400] For example, in the scenario where a user is trying on new clothes while shopping online:

[1401] 1. Initialize the server

[1402] Load the TensorFlow model and emotion engine and connect to the database.

[1403] 2. Initialize the device

[1404] It checks your camera and microphone and displays the message "Do you want to start shopping online?"

[1405] 3. Data Capture

[1406] As the user tries on new clothes, the camera and microphone capture facial expressions and voice data, which are then sent to a server.

[1407] 4. Emotion analysis

[1408] The server receives the captured data and uses an emotion analysis model to infer the emotion "satisfied."

[1409] 5. Product proposal generation

[1410] The server suggests clothes with a similar style to the user based on the emotion of "satisfaction" and past purchase history.

[1411] 6. Display of proposals

[1412] The server sends the proposal to the terminal, which displays it on the user's screen.

[1413] 7. Gathering Feedback

[1414] The user clicks on the suggested clothing and reacts by trying it on or purchasing it, and the device records the reaction and sends it to the server.

[1415] Prompt Sentence Examples

[1416] The following prompt sentences are fed into the generative AI model to perform sentiment analysis:

[1417] A user is trying on new clothes and smiling as he looks at himself in the mirror. His voice is calm and relaxed. How should the server's emotion engine interpret this data and what product recommendations should it make?

[1418] In this way, the system can analyze user sentiment and provide a personalized shopping experience, and it also makes continuous improvements through feedback to provide a more optimized user experience.

[1419] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1420] Step 1: Initialize the server

[1421] explanation

[1422] At system startup, the server does the following:

[1423] Specific actions

[1424] 1. Loading the sentiment analysis model

[1425] Load a sentiment analysis model using TensorFlow. As input, a pre-trained model file is required. As output, the model is loaded into memory and ready for analysis.

[1426] Example: Read the file path of a model and load it using a TensorFlow function.

[1427] 2. Activating the Emotional Engine

[1428] Starts the emotion engine and waits for a connection. It uses the emotion engine's configuration file as input and the engine is started and starts accepting requests as output.

[1429] Example: Initialize the engine by reading the OpenAI API key and necessary configuration information.

[1430] 3. Setting up a virtual shopping environment

[1431] It connects to the required databases and sets up a virtual shopping environment, taking database configuration information and connection credentials as input, and an established database connection ready for operation as output.

[1432] Example: Use MySQL connection information to connect to a database and cache product information and user history.

[1433] Step 2: Reset the device

[1434] explanation

[1435] To provide an interface with the user, the terminal:

[1436] Specific actions

[1437] 1. Hardware check

[1438] Checks whether the camera and microphone are properly connected. It takes hardware status information as input and displays the connection status as output.

[1439] Example: Display a message to the user asking "Do you want to turn on your camera?" and ask for permission.

[1440] 2. Interface display

[1441] Prompts the user with the message "Start shopping online?" Uses user interface configuration information as input and provides the message as output.

[1442] Example: Asking the user for permission using a GUI or VR interface.

[1443] Step 3: Collect user data

[1444] explanation

[1445] The device collects user data in real time and sends it to the server:

[1446] Specific actions

[1447] 1. Data Capture

[1448] The camera continuously captures the user's facial expressions, using real-time video data as input and generating facial expression data as output.

[1449] Example: Activating the camera module to analyze facial expressions frame by frame.

[1450] 2. Data Transmission

[1451] The microphone collects the user's voice data and sends it to the server. It uses real-time voice data as input and sends voice data to the server as output.

[1452] Example: Start an audio capture module and periodically upload audio data to a server.

[1453] Step 4: Sentiment Analysis

[1454] explanation

[1455] The server analyzes the user's emotions based on the received data:

[1456] Specific actions

[1457] 1. Data Reception

[1458] It receives facial expression data and voice data from the terminal. It receives data sent from the terminal and data packets from the relay server as input, and stores the data in an analysis buffer as output.

[1459] Example: Receive data at a server API endpoint and pass it to an analytics engine.

[1460] 2. Emotional inference

[1461] The emotion engine analyzes facial expression and voice data to infer emotions in real time, using the received data as input and generating inferred emotions as output.

[1462] Example: Transferring a smiling face and a calm voice to infer the emotion "satisfied."

[1463] 3. Data Recording

[1464] Record the analysis results in a database. The sentiment analysis results are used as input and a new record is added to the database as output.

[1465] Example: Inserting inferred sentiment into MySQL.

[1466] Step 5: Customize the experience

[1467] explanation

[1468] The server generates personalized suggestions for the user based on the analysis results:

[1469] Specific actions

[1470] 1. Data analysis

[1471] Analyzes the sentiment analysis results and past shopping history. The historical information retrieved from the database and the analysis results are used as input, and the analysis results are generated as output.

[1472] Example: Inferring user preferences by combining past purchase history and current sentiment analysis results.

[1473] 2. Proposal generation

[1474] Generate personalized product recommendations and special offers, using analysis results as input and recommendations as output.

[1475] Example: Using Scikit-learn algorithms to make optimal product recommendations.

[1476] Step 6: View the proposal

[1477] explanation

[1478] Display the suggestions on the device so the user can review them:

[1479] Specific actions

[1480] 1. Submit a proposal

[1481] Send the generated proposal or offer to the device, using the proposal as input and successful transmission as output.

[1482] Example: Send proposal data in JSON format to the device.

[1483] 2. User Interface

[1484] The proposal is displayed on the user's screen. The proposal is used as input, and a visual display is produced as output that the user can confirm.

[1485] Example: Displaying product suggestions using a GUI or VR interface.

[1486] Step 7: User feedback

[1487] explanation

[1488] Collect the user's responses to the suggestions and send them to the server:

[1489] Specific actions

[1490] 1. Feedback Collection

[1491] Record user responses (clicks, try-ons, purchases, etc.) Uses user action data as input and generates feedback data as output.

[1492] Example: Storing a user's click history or purchase history.

[1493] 2. Data Transmission

[1494] The collected feedback data is sent to the server, which uses the feedback data as input and completes sending it to the server as output.

[1495] Example: Uploading feedback data to a server using an API.

[1496] Step 8: Update the analysis results

[1497] explanation

[1498] The server uses the feedback to update the model and algorithms to improve accuracy:

[1499] Specific actions

[1500] 1. Data Analysis

[1501] Analyze the received feedback data, using the feedback data as input and producing the analysis results as output.

[1502] Example: Statistical analysis of user feedback data to extract trends.

[1503] 2. Model Update

[1504] Update sentiment analysis models and product recommendation algorithms: new models and algorithms are generated using the analysis results as input and output.

[1505] Example: Retraining the sentiment analysis model and optimizing the proposed algorithm.

[1506] Through the above processing steps, the system can analyze user sentiment in real time and provide a personalized shopping experience.

[1507] (Application example 2)

[1508] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1509] Conventional online shopping systems have the problem that they do not provide product recommendations that fully understand the user's preferences, resulting in poor user experience. They also have been unable to understand the user's emotions and personalize the shopping experience in real time based on those emotions. Therefore, there is a need to provide a personalized shopping experience that reflects the user's interests.

[1510] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1511] In this invention, the server includes means for analyzing user emotions in real time, means for customizing the virtual shopping experience based on the analysis results, means for collecting user feedback and updating the analysis results, means for recognizing the user's emotions from the captured data and generating emotion-based product suggestions and special offers, and means for displaying the generated suggestions and offers on the user's terminal, thereby enabling personalized product suggestions based on the user's emotions.

[1512] "Means for analyzing user emotions in real time" refers to a function that analyzes the user's facial expressions and voice data obtained using a camera or microphone, etc., and instantly recognizes the user's emotions at that time.

[1513] "Means to customize the virtual shopping experience" refers to the ability to optimize the shopping environment and recommendations for individual users based on the results of sentiment analysis.

[1514] "Means of collecting user feedback and updating analysis results" refers to the function of recording user reactions and behavioral data and using it to improve emotion analysis models and product recommendation algorithms.

[1515] "Means of recognizing a user's emotions from captured data and generating emotion-based product suggestions and special offers" refers to a function that selects and suggests products and services suitable for the user based on emotional data obtained by analyzing collected facial expressions and voice data.

[1516] "Means for displaying generated suggestions and offers on the user's device" refers to the function of displaying product suggestions and special offers generated through sentiment analysis on the user's device, such as a smartphone, smart glasses, or head-mounted display.

[1517] A "terminal for displaying a virtual fitting room interface" is a device that provides an interface for users to try on clothes in a virtual space, and in many cases a head-mounted display is used.

[1518] "Means of recording specific user responses from captured data and using them as feedback" refers to the function of recording user behavioral data such as clicks and purchases and using it as learning data for algorithms.

[1519] This invention is a system that analyzes a user's emotions in real time and customizes a virtual shopping experience based on the analysis results. This system is composed of an emotion analysis engine, a server, a database, a camera, a microphone, a terminal, and communication means. Detailed embodiments of each component of this invention will be described below.

[1520] System configuration

[1521] 1. Server

[1522] Emotion analysis engine: Analyzes the user's facial expression and voice data to recognize emotions in real time.

[1523] Database: Stores analysis results and user feedback data and is used to train machine learning algorithms.

[1524] Product recommendation engine: Generates product suggestions and special offers based on sentiment analysis results.

[1525] 2. Terminal

[1526] Camera and microphone: Captures the user's facial expressions and voice data.

[1527] Data transmission module: Sends the captured data to the server.

[1528] Virtual Shopping Environment Interface: Providing users with a virtual shopping experience via a VR headset or smart glasses.

[1529] 3. Means of communication

[1530] Network infrastructure: Carries out data communication between user terminals and servers.

[1531] System processing flow

[1532] 1. Initialization

[1533] The server loads the necessary software, including a sentiment analysis engine and a product recommendation engine, while the device checks the camera and microphone and displays a confirmation message to the user to begin the virtual shopping experience. The user then authorizes the use of the camera and microphone.

[1534] 2. Collection of User Data

[1535] The device captures the user's facial expressions and voice data in real time and sends the data to a server. The camera and microphone use OpenCV and other image and voice processing libraries.

[1536] 3. Emotion analysis

[1537] The server's emotion analysis engine analyzes the received data and recognizes the user's emotions using a machine learning model, and the results are stored in a database.

[1538] 4. Presenting the generated proposals

[1539] The server generates optimal product suggestions and special offers for the user based on the sentiment analysis results and database. The generated suggestions are sent to the terminal and displayed on the user's device. The user can then check the suggested products in the virtual store.

[1540] Example scenario

[1541] For example, if a user is trying on a dress in a virtual fitting room, a camera and microphone capture the user's facial expressions and voice and send them to the server. If the server detects that the user is pleased with the dress, it can suggest other dresses with the same design but in different colors, as well as matching accessories.

[1542] Prompt Sentence Examples

[1543] plain text

[1544] Suggest items related to products that users enjoy

[1545] This invention makes it possible to provide a personalized virtual shopping experience that is tailored to the user's emotions. Users can receive optimal product suggestions based on their emotions in real time, improving their shopping experience.

[1546] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1547] Step 1:

[1548] Initializing the Server

[1549] The server loads the necessary software components, such as the sentiment analysis AI, emotion engine, database, and product recommendation engine. The initialization process checks the connections between each component and ensures that the system works properly. Machine learning models and necessary algorithms are also loaded at this stage. The server is then ready to receive and analyze data from users.

[1550] Step 2:

[1551] Initializing the device

[1552] The device checks the camera and microphone and displays a message to the user saying "Do you want to start shopping online?", at which point the user is prompted to allow the use of the camera and microphone. If permission is granted, the device prepares to capture data through the camera and microphone.

[1553] Step 3:

[1554] User Data Collection

[1555] The device captures the user's facial expressions and voice data in real time and sends the obtained data to the server. During this process, the camera takes a picture of the user's face and the microphone records the user's voice. The captured data is sent to the server via the network. The input data is image data and voice data, and the output is the data sent to the server.

[1556] Step 4:

[1557] Emotion analysis

[1558] The server inputs the received facial expression data and voice data into an emotion analysis engine, which analyzes the user's emotions in real time. This analysis is performed using a machine learning model. The emotion analysis engine uses image processing and voice analysis algorithms to estimate the user's emotions from the data. The input data is facial expression data and voice data, and the output is the emotion analysis result. The analysis results are recorded in a database.

[1559] Step 5:

[1560] Generate product suggestions

[1561] The server generates optimal product suggestions and special offers for the user based on the sentiment analysis results and a database of past users. This step often uses a generative AI model, which allows the server to create products and offers that match the user's current emotions. The input data are the sentiment analysis results and past user data, and the output is the generated product suggestions and special offers.

[1562] Step 6:

[1563] Viewing Proposals

[1564] The server sends the generated product suggestions and special offers to the terminal. The terminal displays the suggestions on the user's screen. The user can then check the suggested products in the virtual store. The input data are the product suggestions and special offers, and the output is the information displayed on the user's screen.

[1565] Step 7:

[1566] Collecting User Feedback

[1567] The device records the user's reactions to the proposed products and offers, such as clicking, trying them on, and purchasing, and sends them to the server. This data will be used to improve future sentiment analysis models and product recommendation engines. The input data is the user's reaction data, and the output is feedback data sent to the server.

[1568] Step 8:

[1569] Update analysis results

[1570] The server analyzes the feedback data and creates new database entries. This data is used to train the sentiment analysis model and product recommendation algorithm to improve the accuracy of future recommendations. The input data is the feedback data, and the output is the updated analysis model and algorithm.

[1571] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1572] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1573] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1574] [Fourth embodiment]

[1575] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1576] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1577] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1578] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1579] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1580] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1581] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1582] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1583] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1584] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1585] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1586] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1587] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1588] This invention relates to a system that analyzes a user's emotions in real time and customizes a virtual shopping experience based on the analysis results. Specifically, the system includes a terminal for capturing the user's facial expression and voice data and a communication means for transmitting the data to a server. The server analyzes the user's emotions based on the received data and generates product suggestions and special offers based on the analysis results. The generated suggestions and offers are displayed on the user's terminal, and the analysis results are updated based on the user's feedback to improve the accuracy of future suggestions.

[1589] System configuration and processing flow

[1590] composition

[1591] The system consists of the following main components:

[1592] 1. Server

[1593] Emotion analysis AI

[1594] Database

[1595] 2. Terminal

[1596] Camera and microphone

[1597] Data Transmission Module

[1598] Virtual shopping environment interface

[1599] 3. Means of communication

[1600] Network Infrastructure

[1601] Program processing

[1602] 1. Initialize the server

[1603] The server loads the sentiment analysis model and the virtual shopping environment, which are generated from historical user data.

[1604] 2. Initialize the device

[1605] The device checks the camera and microphone and displays the message "Do you want to start shopping online?" to the user.

[1606] 3. Collection of User Data

[1607] When a user allows access to the camera and microphone, the device captures the user's facial expressions and voice data and sends it to the server.

[1608] 4. Emotion analysis

[1609] The server analyzes the facial expression and voice data sent from the device and infers the user's emotions in real time. The analysis results are recorded in a database.

[1610] 5. Customize your experience

[1611] The server generates personalized product suggestions and special offers for the user based on the results of sentiment analysis and past shopping history.

[1612] The generated proposals and offers are sent to the terminal and displayed on the user's screen.

[1613] 6. User Feedback

[1614] The user responds to the suggested product or offer (e.g., clicks, tries on, purchases).

[1615] The terminal records this response and sends it to the server as feedback data.

[1616] 7. Update analysis results

[1617] The server updates the sentiment analysis model based on the user's feedback data to improve the accuracy of suggestions for the next shopping session.

[1618] Specific examples

[1619] Scenario: User wants to try on a virtual item

[1620] 1. Initialize the server

[1621] The server loads the avatar and clothing data for virtual try-on.

[1622] 2. Initialize the device

[1623] The device displays a virtual fitting room interface to users wearing a VR headset.

[1624] 3. Collection of User Data

[1625] While the user looks at themselves in the mirror, a camera and microphone capture their facial expressions and voice, sending the data to a server.

[1626] 4. Emotion analysis

[1627] The server analyzes the user's emotions (e.g., surprise, satisfaction) in real time based on facial expression data and voice data.

[1628] 5. Customize your experience

[1629] If the user is satisfied, the server generates data for suggesting clothes of a similar style and transmits it to the terminal.

[1630] The terminal displays these suggestions to the user.

[1631] 6. User Feedback

[1632] Users try on the suggested clothes and respond by rejecting them or purchasing them.

[1633] The device records this response and sends it to the server.

[1634] 7. Update analysis results

[1635] The server integrates user feedback to update the analytical model, improving the accuracy of future proposals.

[1636] In this way, users can enjoy a real-time, personalized online shopping experience, and the system continuously learns and optimizes the user experience.

[1637] The processing flow will be explained below.

[1638] Step 1:

[1639] The server loads the sentiment analysis model and the virtual shopping environment, including the machine learning models and connections to the necessary databases.

[1640] Step 2:

[1641] The device checks the camera and microphone and displays the message "Do you want to start shopping online?" The user allows the use of the camera and microphone.

[1642] Step 3:

[1643] The device periodically captures the user's facial expressions with a camera and collects audio data with a microphone, which is then sent to a server every few seconds.

[1644] Step 4:

[1645] The server analyzes the received facial expression and voice data to infer the user's emotions (e.g., joy, excitement, dissatisfaction) in real time. The analysis results are recorded in a database.

[1646] Step 5:

[1647] The server uses the analysis results and past shopping history to generate personalized product suggestions and special offers for the user, using machine learning algorithms.

[1648] Step 6:

[1649] The server sends the generated proposals and offers to the terminal, which displays this information on the user's screen. The user then confirms the proposed products and offers.

[1650] Step 7:

[1651] The user's reactions to the proposed products and offers (e.g., clicking, trying on, purchasing) are recorded by the device and transmitted to the server.

[1652] Step 8:

[1653] The server analyzes the user's reaction data sent from the terminal and creates a new database entry as feedback data.

[1654] Step 9:

[1655] The server updates the sentiment analysis model and product recommendation algorithm based on the feedback data, improving the accuracy of future recommendations.

[1656] Step 10:

[1657] The device will continue to present new sentiment-based suggestions and offers to the user, and the whole process will be repeated in real time to optimize the user experience.

[1658] Example 1

[1659] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1660] In conventional online shopping, it was difficult to analyze user emotions in real time and personalize the shopping experience based on that. Furthermore, there was a lack of a mechanism for collecting user feedback and updating the analysis results, which meant that product recommendations did not adequately reflect the user's interests and satisfaction. This resulted in a poor user experience and an inability to fully stimulate purchasing motivation.

[1661] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1662] In this invention, the server includes means for loading and initializing an emotion analysis model, means for collecting facial expression and voice data of a user, means for transmitting the collected data to the server, means for analyzing the user's emotions in real time based on the data received by the server, means for recording the analysis results in a database, means for generating personalized product suggestions and special offers based on the emotion analysis results and past shopping history, means for displaying the generated suggestions and offers on the user terminal, means for collecting user feedback and transmitting the feedback data to the server, and means for updating the emotion analysis model based on the feedback data. This makes it possible to analyze the user's emotions in real time and provide a highly personalized shopping experience that reflects the results.

[1663] An "emotion analysis model" is an algorithm or program that analyzes a user's facial expressions and voice data to infer their emotional state.

[1664] A "server" is a computer system that processes and stores data and provides that data to other computers or devices over a network.

[1665] "User's facial expression and voice data" refers to digital data of human facial movements and voice signals captured using a camera and microphone.

[1666] A "database" is a structured collection of data for efficiently storing, retrieving, and updating information.

[1667] "Product suggestion" refers to the act of introducing products that are likely to interest a user based on the results of an emotional analysis of the user and their past shopping history, and the data used to do so.

[1668] "Special Offers" are commercial offers such as special prices or promotions based on the user's interests.

[1669] "Feedback data" refers to data based on user behavior and reactions that indicate interest in and intent to purchase product suggestions and special offers.

[1670] "Personalization" means customizing product proposals and services to suit the characteristics and preferences of individual users.

[1671] "Real-time" is a concept that refers to processing and reactions occurring almost simultaneously with the collection of data.

[1672] This invention relates to a system that analyzes a user's emotions in real time and customizes a virtual shopping experience based on the analysis results. Specifically, the system includes a terminal for capturing the user's facial expression and voice data and a communication means for transmitting the data to a server. The server analyzes the user's emotions based on the received data and generates product suggestions and special offers based on the analysis results. The generated suggestions and offers are displayed on the user's terminal, and the analysis results are updated based on user feedback, thereby improving the accuracy of future suggestions.

[1673] The main components of the system are as follows:

[1674] 1. Server

[1675] Sentiment analysis model (e.g., an AI model using TensorFlow)

[1676] Database (e.g. MySQL)

[1677] 2. Terminal

[1678] Camera and microphone (e.g., a standard webcam or microphone)

[1679] Data transmission module (e.g. Wi-Fi or Bluetooth)

[1680] Virtual shopping environment interface (e.g., built with Unity)

[1681] 3. Means of communication

[1682] Network infrastructure (e.g., 5G, fiber optic connections)

[1683] Specifically, the server first loads the emotion analysis model and virtual shopping environment. This requires initializing an AI model using TensorFlow and establishing a connection to a database of products taken from past user data. The device then diagnoses the camera and microphone, checks their operation, and displays a message to the user asking, "Do you want to start online shopping?" If the user allows access to the camera and microphone, the device captures the user's facial expressions and voice data and sends it to the server.

[1684] The server analyzes the facial expression and voice data sent from the device and infers the user's emotions in real time. The analysis results are recorded in a database. The server then generates personalized product suggestions and special offers for the user based on the emotion analysis results and past shopping history. The generated suggestions and offers are sent to the device and displayed on the user's screen. When the user responds to the suggested products or offers (e.g., clicking, trying them on, purchasing), the device records this response and sends it to the server as feedback data. Finally, the server updates the emotion analysis model based on the user's feedback data to improve the accuracy of suggestions for the next shopping session.

[1685] As a specific example, consider a user using a virtual fitting service. The server loads the virtual fitting avatar and clothing data, and the device displays a virtual fitting room interface to the user wearing a VR headset. While the user looks at themselves in the mirror, a camera and microphone capture their facial expressions and voice, and send the data to the server. The server analyzes the user's emotions (e.g., surprise, satisfaction) in real time based on the facial expression and voice data. The server then suggests more similar styles of clothing based on the emotion analysis results, and the device displays these suggestions to the user. When the user tries on the suggested clothing and responds by rejecting it or purchasing it, the device records this response and sends it to the server. The server integrates the user's feedback to update the analysis model, improving the accuracy of future suggestions.

[1686] Examples of prompt statements

[1687] Prompt: "Analyze the user's facial expressions and voice in a virtual fitting room to suggest outfits that match their emotions."

[1688] Input data example: Facial expression data [image file], voice data ["I like this outfit!"]

[1689] In this way, users can enjoy a real-time, personalized online shopping experience, and the system can continuously learn and optimize the user experience.

[1690] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1691] Step 1:

[1692] Initializing the Server

[1693] The server loads the sentiment analysis model and the virtual shopping environment. Specifically, it first loads the sentiment analysis model parameters into memory using TensorFlow. Next, it establishes a connection to a product database (e.g., MySQL) based on past user data and loads the necessary product information into memory. This allows the server to perform sentiment analysis and product suggestions in real time.

[1694] Input: TensorFlow model parameter file, product database connection information

[1695] Output: Initialized sentiment analysis model, product data loaded into memory.

[1696] Step 2:

[1697] Initializing the device

[1698] The device diagnoses the camera and microphone to ensure they are working properly. It then displays a message to the user asking, "Do you want to start online shopping?" If the user responds, the device prepares to send the input to the server.

[1699] Specific actions include checking the camera video feed and checking the volume level of the microphone input.

[1700] Input: User actions, device camera and microphone diagnostic results

[1701] Output: User camera and microphone permissions, confirmation to start shopping

[1702] Step 3:

[1703] User Data Collection

[1704] Once the user allows the use of the camera and microphone, the device will activate them and continuously capture the user's facial expressions and voice data, which will be transmitted to the server in real time. The camera will capture high-resolution images, and the microphone will record clear audio.

[1705] Input: Video data from camera, audio data from microphone

[1706] Output: Facial expression data and voice data sent to the server

[1707] Step 4:

[1708] Emotion analysis

[1709] The server receives and analyzes the facial expression and voice data sent from the device. It passes the facial expression images and voice data as input to an emotion analysis model, which then infers the user's emotions in real time. The analysis results are then recorded in a database.

[1710] Input: Emotion analysis model loaded from database, received facial expression data and speech data

[1711] Output: User's emotional state, analysis results recorded in a database

[1712] Step 5:

[1713] Customize your experience

[1714] The server generates personalized product suggestions and special offers for the user based on the results of sentiment analysis and past shopping history. The generated suggestions and offers are sent in JSON format to the device, which then displays them on the user's screen.

[1715] Input: Sentiment analysis results, past shopping history, product database

[1716] Output: Generated product suggestions, special offers, and display data sent to the device

[1717] Step 6:

[1718] User Feedback

[1719] When users interact with the proposed products or offers (e.g., click, try on, purchase), the device records these interactions, and the recorded feedback data is sent to the server in real time.

[1720] Input: User interaction data

[1721] Output: Feedback data sent to the server

[1722] Step 7:

[1723] Update analysis results

[1724] The server receives user feedback data and updates the sentiment analysis model based on that data. This update improves the accuracy of suggestions for the next shopping session by optimizing the model parameters and algorithms.

[1725] Input: Feedback data, sentiment analysis model

[1726] Output: Updated sentiment analysis model, improving the accuracy of the next proposal

[1727] Through these steps, the system can analyze user sentiment in real time, generate and present product suggestions and special offers tailored to each user, and continuously learn to optimize the user experience.

[1728] (Application example 1)

[1729] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1730] Conventional virtual shopping experiences have had the challenge of being difficult to flexibly respond to individual users' emotions and preferences. In particular, there has been a lack of systems that can analyze a user's emotional state in real time and make product recommendations based on the results. This means that users are often left feeling dissatisfied because they are rarely presented with products that suit them at the right time. Furthermore, there has been a lack of mechanisms for effectively utilizing user feedback to improve the shopping experience.

[1731] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1732] In this invention, the server includes means for analyzing user emotions in real time, means for collecting user feedback and updating the analysis results, and means for capturing the user's facial expression and voice data using the smart glasses, which enables the generation of personalized product suggestions and special offers based on the user's emotions and feedback and displaying these suggestions on the display of the smart glasses, thereby providing the user with a more comfortable and efficient shopping experience.

[1733] "Means for analyzing a user's emotions in real time" refers to a technology that instantly determines the user's emotional state at any given time based on the facial expressions and voice data they show.

[1734] "Means for customizing virtual shopping experiences" refers to technology that individually adjusts and optimizes the shopping experience based on user sentiment analysis and feedback.

[1735] The "means of collecting user feedback and updating analysis results" refers to a technology that continuously records the reactions and behaviors of users and uses them to improve the accuracy of emotion analysis models.

[1736] "Means for capturing a user's facial expression and voice data using smart glasses" refers to a technology that uses smart glasses with built-in cameras and microphones to obtain a user's facial expression and voice information.

[1737] The "communication means for transmitting the data to the server" refers to a technique for transmitting the captured data to the server via the Internet or other communication network.

[1738] The "means for generating personalized product proposals based on analysis results" refers to a technology that selects and proposes optimal products and offers individually based on the user's emotional state and past shopping history.

[1739] The "means for displaying the generated proposals on the display of the smart glasses" is a technology that uses the display of the smart glasses to visually present the generated product proposals or offers to the user.

[1740] "Means for recording user responses and transmitting them to a server" refers to technology that records the actions and responses that users show to displayed suggestions and offers and transmits them to a server via the Internet or other communications network.

[1741] This invention is a system that analyzes users' emotions in real time and customizes the virtual shopping experience based on the analysis results. Specifically, it uses smart glasses to capture the user's facial expressions and voice data, transmits them to a server for analysis, and generates individually optimized product recommendations.

[1742] System Configuration

[1743] server

[1744] Emotion analysis AI model: A model for analyzing a user's facial expressions and voice data to infer emotions. Models created with TensorFlow and PyTorch are used.

[1745] Database: A database for storing user data, suggestion data, and feedback data. MySQL or PostgreSQL can be used.

[1746] Communication method: Protocol used to send and receive data. WebSocket or gRPC is used.

[1747] Terminal (smart glasses)

[1748] Camera and microphone: Devices required to capture the user's facial expressions and voice data.

[1749] Display: A device used to display suggested products and offers to the user.

[1750] Data transmission module: A module for transmitting captured data to the server.

[1751] Program processing overview

[1752] 1. Initialize the server:

[1753] When the server starts, it loads the sentiment analysis model and resources of the virtual shopping environment, including the model generated based on historical user data.

[1754] 2. Initialize the device:

[1755] When the smart glasses start up, the user is prompted, "Would you like to enter the virtual store?"

[1756] If the user agrees, the camera and microphone will be enabled.

[1757] 3. Collection of User Data:

[1758] The smart glasses capture the user's facial expressions and voice and transmit them to a server in real time.

[1759] 4. Emotion analysis:

[1760] The server analyzes the user's emotions based on the received data, using an emotion analysis AI model, and stores the results in a database.

[1761] 5. Product proposal generation:

[1762] Based on the sentiment analysis results and past shopping history, the server generates personalized product suggestions.

[1763] 6. Displaying Proposals:

[1764] The generated suggestions are displayed on the smart glasses' display, and the user can refer to them while shopping.

[1765] 7. Collecting User Feedback:

[1766] The smart glasses record the user's response to the suggested products (e.g., whether they purchase or try them on) and send it to the server.

[1767] 8. Update analysis results:

[1768] The server updates the emotion analysis model based on user feedback data, which improves the accuracy of future suggestions.

[1769] Specific examples

[1770] When a user is searching for clothes in a virtual store, the smart glasses analyze the user's emotions and, if they are satisfied, suggest, "Would you like to see more similar styles?" Conversely, if the user appears dissatisfied, they suggest, "Would you like to see other styles?"

[1771] Prompt Sentence Examples

[1772] "Analyzing the emotions users show during their virtual store experiences."

[1773] "Analyzes the user's emotions in real time based on facial expressions and voice data, and makes appropriate product recommendations."

[1774] This system allows users to receive personalized product suggestions in real time, and the system continuously learns based on feedback data, further improving the shopping experience in future purchases.

[1775] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1776] Step 1:

[1777] Initializing the Server

[1778] When the server starts up, it loads the emotion analysis model and resources for the virtual shopping environment. At this time, the emotion analysis AI model built based on past user data is loaded. The input is the server startup signal, and the output is that the emotion analysis model and resources are ready for use. Specifically, it retrieves the necessary configuration information from the database and loads the model and resources into memory.

[1779] Step 2:

[1780] Initializing the device

[1781] When the smart glasses start up, they prompt the user, "Would you like to enter the virtual store?" The input is the smart glasses' startup signal, and the output is the display of the prompt. Specifically, the system performs a startup check and then displays the prompt on the display.

[1782] Step 3:

[1783] User Data Collection

[1784] Once the user agrees to the prompt, the smart glasses' camera and microphone are enabled and begin capturing the user's facial expressions and voice data, which are then transmitted to the server in real time. The input is the user's consent and the captured data, and the output is the data transmitted to the server. Specifically, the glasses enable the camera and microphone, continuously capture data, and transmit it using a network protocol.

[1785] Step 4:

[1786] Emotion analysis

[1787] The server analyzes the user's emotions in real time using an emotion analysis AI model based on the received facial expression and voice data. The analysis results are saved in a database. The input is the received data, and the output is the analysis results. Specifically, the facial expression data is passed to an image recognition model, and the voice data is passed to a voice recognition model to determine the emotion.

[1788] Step 5:

[1789] Generate product suggestions

[1790] The server generates individually optimized product recommendations based on the sentiment analysis results and the user's past shopping history. The inputs are the analysis results and shopping history, and the output is the generated product recommendations. Specifically, the server retrieves past history from the database and inputs it, along with the sentiment analysis results, into the product recommendation algorithm.

[1791] Step 6:

[1792] View Suggestions

[1793] The generated product suggestions are displayed on the smart glasses' display. The input is the generated product suggestions, and the output is the visual information displayed on the display. Specifically, the suggestion data is converted into a display format and sent to the display for display.

[1794] Step 7:

[1795] Collecting user feedback

[1796] The smart glasses record the user's reaction to the suggested products (e.g., whether they purchase or try them on), and then send the feedback data to the server. The input is the user's reaction, and the output is the feedback data sent to the server. Specifically, the glasses record the user's interaction with sensors and send the data to the server using a network protocol.

[1797] Step 8:

[1798] Update analysis results

[1799] The server updates the sentiment analysis AI model based on the user's feedback data. This improves the accuracy of future suggestions. The input is the feedback data, and the output is an updated sentiment analysis model. Specifically, the server analyzes the feedback data and partially retrains the existing model.

[1800] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1801] This invention relates to a system that analyzes a user's emotions in real time using an emotion engine and customizes a virtual shopping experience based on the analysis results. Specifically, this system includes a terminal that captures the user's facial expressions and voice data, a communication means for transmitting the data to a server, and a function for recognizing the user's emotions using the emotion engine.

[1802] System configuration and processing flow

[1803] composition

[1804] The system consists of the following main components:

[1805] 1. Server

[1806] Emotion analysis AI

[1807] Database

[1808] Emotion Engine

[1809] 2. Terminal

[1810] Camera and microphone

[1811] Data Transmission Module

[1812] Virtual shopping environment interface

[1813] 3. Means of communication

[1814] Network Infrastructure

[1815] Program processing

[1816] 1. Initialize the server

[1817] The server loads the sentiment analysis model, emotion engine, and virtual shopping environment, including connections to machine learning models and necessary databases.

[1818] 2. Initialize the device

[1819] The device checks the camera and microphone and displays the message "Do you want to start shopping online?" The user allows the use of the camera and microphone.

[1820] 3. Collection of User Data

[1821] When a user allows access to the camera and microphone, the device captures the user's facial expressions and voice data in real time and sends it to the server.

[1822] 4. Emotion analysis

[1823] The server's emotion engine analyzes the received facial expression and voice data to infer the user's emotions (e.g., joy, excitement, frustration) in real time. The analysis results are recorded in a database.

[1824] 5. Customize your experience

[1825] The server uses the sentiment analysis results and past shopping history to generate personalized product suggestions and special offers for the user, using machine learning algorithms.

[1826] 6. Display of proposals

[1827] The server sends the generated proposals and offers to the terminal, which displays this information on the user's screen. The user then confirms the proposed products and offers.

[1828] 7. User Feedback

[1829] The device records the user's reactions to the proposed products and offers (e.g., clicks, try-ons, purchases) and sends them to the server.

[1830] 8. Update analysis results

[1831] The server analyzes the user's reaction data sent from the device and stores it in a database as feedback. It then updates the sentiment analysis model and product recommendation algorithm. This update improves the accuracy of future recommendations.

[1832] Specific examples

[1833] Scenario: User wants to try on a virtual item

[1834] 1. Initialize the server

[1835] The server loads the avatar and clothing data for virtual try-on.

[1836] 2. Initialize the device

[1837] The device displays a virtual fitting room interface to users wearing a VR headset.

[1838] 3. Collection of User Data

[1839] While the user looks at themselves in the mirror, a camera and microphone capture their facial expressions and voice, sending the data to a server.

[1840] 4. Emotion analysis

[1841] The server's emotion engine analyzes the user's facial expression data and voice data to analyze emotions (e.g., surprise, satisfaction) in real time. The analysis results are recorded in a database.

[1842] 5. Customize your experience

[1843] If the user is satisfied, the server generates and sends data to the terminal to suggest clothes of a similar style, which the terminal then displays to the user.

[1844] 6. User Feedback

[1845] The user tries on the suggested clothes and responds by commenting or purchasing them, and the device records these responses and sends them to the server.

[1846] 7. Update analysis results

[1847] The server analyzes the user's feedback data, creates a new database entry as feedback, and updates the sentiment analysis model and suggestion algorithm to improve the accuracy of future suggestions.

[1848] In this way, users can enjoy a real-time, personalized online shopping experience, and the system continuously learns to provide a more optimized user experience.

[1849] The processing flow will be explained below.

[1850] Step 1:

[1851] The server loads the sentiment analysis model, emotion engine, and virtual shopping environment, a process that includes setting the parameters of the machine learning model and connecting to the relevant databases.

[1852] Step 2:

[1853] The device checks the camera and microphone and displays the message "Do you want to start shopping online?" The user allows the use of the camera and microphone.

[1854] Step 3:

[1855] The device captures the user's facial expressions with a camera and collects audio data with a microphone. This capture process is carried out periodically and the data is sent to a server in real time.

[1856] Step 4:

[1857] The server's emotion engine analyzes the received facial expression data and voice data to infer the user's emotions (e.g., joy, excitement, dissatisfaction) in real time. The analysis results are recorded in a database.

[1858] Step 5:

[1859] The server uses the results of the sentiment analysis and past shopping history to generate personalized product suggestions and special offers for the user, using machine learning algorithms.

[1860] Step 6:

[1861] The server sends the generated proposals and offers to the device, which then displays the information on the user's screen. The user can then review the proposed products and offers and take action as necessary.

[1862] Step 7:

[1863] The user's reactions to the proposed products and offers (e.g., clicks, try-ons, purchases) are recorded by the device and transmitted to the server in real time.

[1864] Step 8:

[1865] The server analyzes the user's response data and creates new database entries as feedback. This feedback data is an important factor in future analysis and proposal generation.

[1866] Step 9:

[1867] The server updates the sentiment analysis model and product recommendation algorithm based on the feedback data, improving the accuracy of future recommendations and optimizing the user experience.

[1868] Step 10:

[1869] The terminal will continue to present new suggestions and offers to the user based on their emotions, and the whole process will be repeated in real time, allowing the user to continuously enjoy the virtual shopping experience.

[1870] Example 2

[1871] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1872] Conventional virtual shopping systems have struggled to provide a personalized experience based on users' emotions and preferences. They also lacked the means to collect real-time user feedback and improve the system based on it. This resulted in a uniform shopping experience, making it difficult to improve user satisfaction.

[1873] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for loading an emotion analysis model, means for checking the camera and microphone and confirming with the user whether to start online shopping, means for capturing the user's facial expression and voice data in real time and sending it to the server, means for analyzing the user's emotions in real time, means for generating personalized product suggestions and special offers based on the analysis results and past shopping history, means for displaying the generated suggestions and offers on the user terminal, and means for collecting user feedback and updating the analysis results and the suggestion algorithm. This makes it possible to provide a personalized shopping experience based on the user's emotions and preferences and to continuously improve the system.

[1874] An "emotion analysis model" is a machine learning model that predicts a user's emotions in real time based on their facial expressions and voice data.

[1875] A "camera" is a photographing device for capturing the user's facial expressions.

[1876] A "microphone" is a voice recording device for capturing the user's voice data.

[1877] "Server" refers to a computer system that manages and operates the emotion analysis model, emotion engine, and virtual shopping environment.

[1878] An "emotion engine" is software that analyzes facial expressions and voice data to predict emotions in real time.

[1879] The "database" is a system for storing and managing user sentiment analysis results, shopping history, feedback data, etc.

[1880] A "virtual shopping environment" is a virtual environment in which users can experience shopping online.

[1881] "Communication means" refers to the network infrastructure for transmitting the user's facial expressions and voice data from the terminal to the server.

[1882] "Feedback" refers to the reactions and actions that users take in response to proposed products or offers.

[1883] "Product suggestions" refers to suggesting personalized products to users based on the results of emotional analysis of the user and their past shopping history.

[1884] "Special Offer" refers to a specific discount or promotion that is customized for you.

[1885] A "suggestion algorithm" is a computational means for generating optimal product suggestions and special offers based on the results of a user's sentiment analysis and past history.

[1886] The present invention relates to a system for analyzing a user's emotions in real time and customizing a virtual shopping experience based on the analysis results. A specific embodiment of this system will be described below.

[1887] System configuration

[1888] This system mainly consists of three main components: a server, a terminal, and a communication means.

[1889] 1. Server

[1890] Sentiment analysis model: Refers to a neural network model built using a machine learning library such as TensorFlow.

[1891] Emotion engine: Software that uses OpenAI APIs and other tools to analyze a user's facial expressions and voice data.

[1892] Database: We use a database management system such as MySQL to store and manage user sentiment analysis results, shopping history, and feedback data.

[1893] 2. Terminal

[1894] Camera: A photographic device for capturing the user's facial expressions.

[1895] Microphone: A voice recording device for capturing the user's voice data.

[1896] Data transmission module: A communication module for transmitting facial expression data and voice data to the server.

[1897] Virtual shopping environment interface: The interface through which the user operates, for example, using a VR headset or touchscreen.

[1898] 3. Means of communication

[1899] Network infrastructure: A means of transmitting data from a terminal to a server using communication infrastructure such as the Internet.

[1900] Program processing

[1901] The programs in this system perform the following basic tasks:

[1902] 1. Loading the sentiment analysis model

[1903] The server uses TensorFlow to load a sentiment analysis model, which includes a pre-trained neural network.

[1904] 2. Check your camera and microphone

[1905] The device checks whether the camera and microphone are properly connected, displays a message to the user asking "Do you want to start shopping online?" and asks for permission.

[1906] 3. Data Capture

[1907] The camera captures the user's facial expressions and the microphone collects audio data, which is sent to a server in real time.

[1908] 4. Emotion analysis

[1909] The server's emotion engine analyzes the received facial expression and voice data to infer the user's emotion in real time. For example, it can infer the emotion "satisfied" by analyzing a smiling face and a calm voice.

[1910] 5. Product proposal generation

[1911] The server generates personalized product suggestions and special offers based on the sentiment analysis results and past shopping history, using machine learning algorithms such as Scikit-learn.

[1912] 6. Display of proposals

[1913] The server sends the generated recommendations to the device, which displays them on the user's screen, allowing the user to view the recommendations, click, and purchase.

[1914] 7. Gathering Feedback

[1915] The device records the user's reactions (clicks, trying on, purchases, etc.) and sends them to the server, which analyzes them and updates the sentiment analysis model and proposed algorithm.

[1916] Specific examples

[1917] For example, in the scenario where a user is trying on new clothes while shopping online:

[1918] 1. Initialize the server

[1919] Load the TensorFlow model and emotion engine and connect to the database.

[1920] 2. Initialize the device

[1921] It checks your camera and microphone and displays the message "Do you want to start shopping online?"

[1922] 3. Data Capture

[1923] As the user tries on new clothes, the camera and microphone capture facial expressions and voice data, which are then sent to a server.

[1924] 4. Emotion analysis

[1925] The server receives the captured data and uses an emotion analysis model to infer the emotion "satisfied."

[1926] 5. Product proposal generation

[1927] The server suggests clothes with a similar style to the user based on the emotion of "satisfaction" and past purchase history.

[1928] 6. Display of proposals

[1929] The server sends the proposal to the terminal, which displays it on the user's screen.

[1930] 7. Gathering Feedback

[1931] The user clicks on the suggested clothing and reacts by trying it on or purchasing it, and the device records the reaction and sends it to the server.

[1932] Prompt Sentence Examples

[1933] The following prompt sentences are fed into the generative AI model to perform sentiment analysis:

[1934] A user is trying on new clothes and smiling as he looks at himself in the mirror. His voice is calm and relaxed. How should the server's emotion engine interpret this data and what product recommendations should it make?

[1935] In this way, the system can analyze user sentiment and provide a personalized shopping experience, and it also makes continuous improvements through feedback to provide a more optimized user experience.

[1936] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1937] Step 1: Initialize the server

[1938] explanation

[1939] At system startup, the server does the following:

[1940] Specific actions

[1941] 1. Loading the sentiment analysis model

[1942] Load a sentiment analysis model using TensorFlow. As input, a pre-trained model file is required. As output, the model is loaded into memory and ready for analysis.

[1943] Example: Read the file path of a model and load it using a TensorFlow function.

[1944] 2. Activating the Emotional Engine

[1945] Starts the emotion engine and waits for a connection. It uses the emotion engine's configuration file as input and the engine is started and starts accepting requests as output.

[1946] Example: Initialize the engine by reading the OpenAI API key and necessary configuration information.

[1947] 3. Setting up a virtual shopping environment

[1948] It connects to the required databases and sets up a virtual shopping environment, taking database configuration information and connection credentials as input, and an established database connection ready for operation as output.

[1949] Example: Use MySQL connection information to connect to a database and cache product information and user history.

[1950] Step 2: Reset the device

[1951] explanation

[1952] To provide an interface with the user, the terminal:

[1953] Specific actions

[1954] 1. Hardware check

[1955] Checks whether the camera and microphone are properly connected. It takes hardware status information as input and displays the connection status as output.

[1956] Example: Display a message to the user asking "Do you want to turn on your camera?" and ask for permission.

[1957] 2. Interface display

[1958] Prompts the user with the message "Start shopping online?" Uses user interface configuration information as input and provides the message as output.

[1959] Example: Asking the user for permission using a GUI or VR interface.

[1960] Step 3: Collect user data

[1961] explanation

[1962] The device collects user data in real time and sends it to the server:

[1963] Specific actions

[1964] 1. Data Capture

[1965] The camera continuously captures the user's facial expressions, using real-time video data as input and generating facial expression data as output.

[1966] Example: Activating the camera module to analyze facial expressions frame by frame.

[1967] 2. Data Transmission

[1968] The microphone collects the user's voice data and sends it to the server. It uses real-time voice data as input and sends voice data to the server as output.

[1969] Example: Start an audio capture module and periodically upload audio data to a server.

[1970] Step 4: Sentiment Analysis

[1971] explanation

[1972] The server analyzes the user's emotions based on the received data:

[1973] Specific actions

[1974] 1. Data Reception

[1975] It receives facial expression data and voice data from the terminal. It receives data sent from the terminal and data packets from the relay server as input, and stores the data in an analysis buffer as output.

[1976] Example: Receive data at a server API endpoint and pass it to an analytics engine.

[1977] 2. Emotional inference

[1978] The emotion engine analyzes facial expression and voice data to infer emotions in real time, using the received data as input and generating inferred emotions as output.

[1979] Example: Transferring a smiling face and a calm voice to infer the emotion "satisfied."

[1980] 3. Data Recording

[1981] Record the analysis results in a database. The sentiment analysis results are used as input and a new record is added to the database as output.

[1982] Example: Inserting inferred sentiment into MySQL.

[1983] Step 5: Customize the experience

[1984] explanation

[1985] The server generates personalized suggestions for the user based on the analysis results:

[1986] Specific actions

[1987] 1. Data analysis

[1988] Analyzes the sentiment analysis results and past shopping history. The historical information retrieved from the database and the analysis results are used as input, and the analysis results are generated as output.

[1989] Example: Inferring user preferences by combining past purchase history and current sentiment analysis results.

[1990] 2. Proposal generation

[1991] Generate personalized product recommendations and special offers, using analysis results as input and recommendations as output.

[1992] Example: Using Scikit-learn algorithms to make optimal product recommendations.

[1993] Step 6: View the proposal

[1994] explanation

[1995] Display the suggestions on the device so the user can review them:

[1996] Specific actions

[1997] 1. Submit a proposal

[1998] Send the generated proposal or offer to the device, using the proposal as input and successful transmission as output.

[1999] Example: Send proposal data in JSON format to the device.

[2000] 2. User Interface

[2001] The proposal is displayed on the user's screen. The proposal is used as input, and a visual display is produced as output that the user can confirm.

[2002] Example: Displaying product suggestions using a GUI or VR interface.

[2003] Step 7: User feedback

[2004] explanation

[2005] Collect the user's responses to the suggestions and send them to the server:

[2006] Specific actions

[2007] 1. Feedback Collection

[2008] Record user responses (clicks, try-ons, purchases, etc.) Uses user action data as input and generates feedback data as output.

[2009] Example: Storing a user's click history or purchase history.

[2010] 2. Data Transmission

[2011] The collected feedback data is sent to the server, which uses the feedback data as input and completes sending it to the server as output.

[2012] Example: Uploading feedback data to a server using an API.

[2013] Step 8: Update the analysis results

[2014] explanation

[2015] The server uses the feedback to update the model and algorithms to improve accuracy:

[2016] Specific actions

[2017] 1. Data Analysis

[2018] Analyze the received feedback data, using the feedback data as input and producing the analysis results as output.

[2019] Example: Statistical analysis of user feedback data to extract trends.

[2020] 2. Model Update

[2021] Update sentiment analysis models and product recommendation algorithms: new models and algorithms are generated using the analysis results as input and output.

[2022] Example: Retraining the sentiment analysis model and optimizing the proposed algorithm.

[2023] Through the above processing steps, the system can analyze user sentiment in real time and provide a personalized shopping experience.

[2024] (Application example 2)

[2025] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2026] Conventional online shopping systems have the problem that they do not provide product recommendations that fully understand the user's preferences, resulting in poor user experience. They also have been unable to understand the user's emotions and personalize the shopping experience in real time based on those emotions. Therefore, there is a need to provide a personalized shopping experience that reflects the user's interests.

[2027] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[2028] In this invention, the server includes means for analyzing user emotions in real time, means for customizing the virtual shopping experience based on the analysis results, means for collecting user feedback and updating the analysis results, means for recognizing the user's emotions from the captured data and generating emotion-based product suggestions and special offers, and means for displaying the generated suggestions and offers on the user's terminal, thereby enabling personalized product suggestions based on the user's emotions.

[2029] "Means for analyzing user emotions in real time" refers to a function that analyzes the user's facial expressions and voice data obtained using a camera or microphone, etc., and instantly recognizes the user's emotions at that time.

[2030] "Means to customize the virtual shopping experience" refers to the ability to optimize the shopping environment and recommendations for individual users based on the results of sentiment analysis.

[2031] "Means of collecting user feedback and updating analysis results" refers to the function of recording user reactions and behavioral data and using it to improve emotion analysis models and product recommendation algorithms.

[2032] "Means of recognizing a user's emotions from captured data and generating emotion-based product suggestions and special offers" refers to a function that selects and suggests products and services suitable for the user based on emotional data obtained by analyzing collected facial expressions and voice data.

[2033] "Means for displaying generated suggestions and offers on the user's device" refers to the function of displaying product suggestions and special offers generated through sentiment analysis on the user's device, such as a smartphone, smart glasses, or head-mounted display.

[2034] A "terminal for displaying a virtual fitting room interface" is a device that provides an interface for users to try on clothes in a virtual space, and in many cases a head-mounted display is used.

[2035] "Means of recording specific user responses from captured data and using them as feedback" refers to the function of recording user behavioral data such as clicks and purchases and using it as learning data for algorithms.

[2036] This invention is a system that analyzes a user's emotions in real time and customizes a virtual shopping experience based on the analysis results. This system is composed of an emotion analysis engine, a server, a database, a camera, a microphone, a terminal, and communication means. Detailed embodiments of each component of this invention will be described below.

[2037] System configuration

[2038] 1. Server

[2039] Emotion analysis engine: Analyzes the user's facial expression and voice data to recognize emotions in real time.

[2040] Database: Stores analysis results and user feedback data and is used to train machine learning algorithms.

[2041] Product recommendation engine: Generates product suggestions and special offers based on sentiment analysis results.

[2042] 2. Terminal

[2043] Camera and microphone: Captures the user's facial expressions and voice data.

[2044] Data transmission module: Sends the captured data to the server.

[2045] Virtual Shopping Environment Interface: Providing users with a virtual shopping experience via a VR headset or smart glasses.

[2046] 3. Means of communication

[2047] Network infrastructure: Carries out data communication between user terminals and servers.

[2048] System processing flow

[2049] 1. Initialization

[2050] The server loads the necessary software, including a sentiment analysis engine and a product recommendation engine, while the device checks the camera and microphone and displays a confirmation message to the user to begin the virtual shopping experience. The user then authorizes the use of the camera and microphone.

[2051] 2. Collection of User Data

[2052] The device captures the user's facial expressions and voice data in real time and sends the data to a server. The camera and microphone use OpenCV and other image and voice processing libraries.

[2053] 3. Emotion analysis

[2054] The server's emotion analysis engine analyzes the received data and recognizes the user's emotions using a machine learning model, and the results are stored in a database.

[2055] 4. Presenting the generated proposals

[2056] The server generates optimal product suggestions and special offers for the user based on the sentiment analysis results and database. The generated suggestions are sent to the terminal and displayed on the user's device. The user can then check the suggested products in the virtual store.

[2057] Example scenario

[2058] For example, if a user is trying on a dress in a virtual fitting room, a camera and microphone capture the user's facial expressions and voice and send them to the server. If the server detects that the user is pleased with the dress, it can suggest other dresses with the same design but in different colors, as well as matching accessories.

[2059] Prompt Sentence Examples

[2060] plain text

[2061] Suggest items related to products that users enjoy

[2062] This invention makes it possible to provide a personalized virtual shopping experience that is tailored to the user's emotions. Users can receive optimal product suggestions based on their emotions in real time, improving their shopping experience.

[2063] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2064] Step 1:

[2065] Initializing the Server

[2066] The server loads the necessary software components, such as the sentiment analysis AI, emotion engine, database, and product recommendation engine. The initialization process checks the connections between each component and ensures that the system works properly. Machine learning models and necessary algorithms are also loaded at this stage. The server is then ready to receive and analyze data from users.

[2067] Step 2:

[2068] Initializing the device

[2069] The device checks the camera and microphone and displays a message to the user saying "Do you want to start shopping online?", at which point the user is prompted to allow the use of the camera and microphone. If permission is granted, the device prepares to capture data through the camera and microphone.

[2070] Step 3:

[2071] User Data Collection

[2072] The device captures the user's facial expressions and voice data in real time and sends the obtained data to the server. During this process, the camera takes a picture of the user's face and the microphone records the user's voice. The captured data is sent to the server via the network. The input data is image data and voice data, and the output is the data sent to the server.

[2073] Step 4:

[2074] Emotion analysis

[2075] The server inputs the received facial expression data and voice data into an emotion analysis engine, which analyzes the user's emotions in real time. This analysis is performed using a machine learning model. The emotion analysis engine uses image processing and voice analysis algorithms to estimate the user's emotions from the data. The input data is facial expression data and voice data, and the output is the emotion analysis result. The analysis results are recorded in a database.

[2076] Step 5:

[2077] Generate product suggestions

[2078] The server generates optimal product suggestions and special offers for the user based on the sentiment analysis results and a database of past users. This step often uses a generative AI model, which allows the server to create products and offers that match the user's current emotions. The input data are the sentiment analysis results and past user data, and the output is the generated product suggestions and special offers.

[2079] Step 6:

[2080] Viewing Proposals

[2081] The server sends the generated product suggestions and special offers to the terminal. The terminal displays the suggestions on the user's screen. The user can then check the suggested products in the virtual store. The input data are the product suggestions and special offers, and the output is the information displayed on the user's screen.

[2082] Step 7:

[2083] Collecting user feedback

[2084] The device records the user's reactions to the proposed products and offers, such as clicking, trying them on, and purchasing, and sends them to the server. This data will be used to improve future sentiment analysis models and product recommendation engines. The input data is the user's reaction data, and the output is feedback data sent to the server.

[2085] Step 8:

[2086] Update analysis results

[2087] The server analyzes the feedback data and creates new database entries. This data is used to train the sentiment analysis model and product recommendation algorithm to improve the accuracy of future recommendations. The input data is the feedback data, and the output is the updated analysis model and algorithm.

[2088] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2089] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2090] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2091] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2092] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2093] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2094] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2095] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2096] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2097] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2098] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2099] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2100] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2101] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2102] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[2103] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[2104] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[2105] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[2106] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[2107] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[2108] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[2109] The following is further disclosed regarding the above embodiment.

[2110] (Claim 1)

[2111] A means for analyzing user emotions in real time;

[2112] a means for customizing the virtual shopping experience based on the results of that analysis; and

[2113] a means of collecting user feedback and updating the analysis results;

[2114] A system including:

[2115] (Claim 2)

[2116] a terminal for capturing facial expression and voice data of a user;

[2117] a communication means for transmitting the data to a server;

[2118] The system of claim 1 further comprising:

[2119] (Claim 3)

[2120] means for generating product suggestions and special offers based on the results of the user sentiment analysis;

[2121] means for displaying the generated proposals or offers on a user terminal;

[2122] The system of claim 1 further comprising:

[2123] "Example 1"

[2124] (Claim 1)

[2125] a means for loading and initializing the sentiment analysis model;

[2126] means for collecting facial expression and voice data of a user;

[2127] means for transmitting the collected data to a server;

[2128] A means for analyzing user emotions in real time based on the data received on the server side;

[2129] A means for recording the analysis results in a database;

[2130] A means for generating personalized product suggestions and special offers from the results of sentiment analysis and past shopping history;

[2131] means for displaying the generated proposals or offers on a user terminal;

[2132] means for collecting user feedback and transmitting the feedback data to a server;

[2133] means for updating the sentiment analysis model based on the feedback data;

[2134] A system including:

[2135] (Claim 2)

[2136] a terminal for capturing facial expression and voice data of a user;

[2137] a communication means for transmitting the data to a server;

[2138] The system of claim 1 further comprising:

[2139] (Claim 3)

[2140] means for generating product suggestions and special offers based on the results of the user sentiment analysis;

[2141] means for displaying the generated proposals or offers on a user terminal;

[2142] The system of claim 1 further comprising:

[2143] "Application Example 1"

[2144] (Claim 1)

[2145] A means for analyzing user emotions in real time;

[2146] a means for customizing the virtual shopping experience based on the results of that analysis; and

[2147] a means of collecting user feedback and updating the analysis results;

[2148] means for capturing facial expression and voice data of a user using the smart glasses;

[2149] a communication means for transmitting the data to a server;

[2150] means for generating personalized product suggestions based on the analysis results and displaying the generated suggestions on a display of the smart glasses;

[2151] means for recording and transmitting user responses to a server;

[2152] A system including:

[2153] (Claim 2)

[2154] a terminal for capturing facial expression and voice data of a user using smart glasses;

[2155] a communication means for transmitting the data to a server;

[2156] The system of claim 1 further comprising:

[2157] (Claim 3)

[2158] A means for generating personalized product suggestions and special offers based on the results of the user sentiment analysis;

[2159] means for displaying the generated suggestions or offers on a display of the smart glasses;

[2160] The system of claim 1 further comprising:

[2161] "Example 2: Combining Emotion Engines"

[2162] (Claim 1)

[2163] means for loading a sentiment analysis model;

[2164] A means to check the camera and microphone and ask the user if they want to start shopping online;

[2165] means for capturing the user's facial expression and voice data in real time and transmitting the captured data to a server;

[2166] A means for analyzing user emotions in real time;

[2167] means for generating personalized product suggestions and special offers based on the analysis and past shopping history; and

[2168] means for displaying the generated proposals or offers on a user terminal;

[2169] A means of collecting user feedback and updating the analysis results and proposed algorithms;

[2170] A system including:

[2171] (Claim 2)

[2172] a terminal for capturing facial expression and voice data of a user;

[2173] a communication means for transmitting the data to a server;

[2174] The system of claim 1 further comprising:

[2175] (Claim 3)

[2176] A means for analyzing user feedback data and updating a sentiment analysis model and a product recommendation algorithm;

[2177] The system of claim 1 further comprising:

[2178] "Application example 2 when combining emotion engines"

[2179] (Claim 1)

[2180] A means for analyzing user emotions in real time;

[2181] a means for customizing the virtual shopping experience based on the results of that analysis; and

[2182] a means of collecting user feedback and updating the analysis results;

[2183] means for recognizing user emotions from the captured data and generating emotion-based product suggestions and special offers;

[2184] means for displaying the generated proposals or offers on the user's terminal;

[2185] A system including:

[2186] (Claim 2)

[2187] a terminal for capturing facial expression and voice data of a user;

[2188] a communication means for transmitting the data to a server;

[2189] a terminal for displaying an interface of the virtual fitting room;

[2190] The system of claim 1 further comprising:

[2191] (Claim 3)

[2192] means for generating product suggestions and special offers based on the results of the user sentiment analysis;

[2193] a means for recording specific user responses from the captured data and using them as feedback;

[2194] means for displaying the generated suggestions or offers on a user's device and recording the user's responses;

[2195] The system of claim 1 further comprising: [Explanation of symbols]

[2196] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for analyzing user emotions in real time; a means for customizing the virtual shopping experience based on the results of that analysis; and a means of collecting user feedback and updating the analysis results; A system including:

2. a terminal for capturing facial expression and voice data of a user; a communication means for transmitting the data to a server; The system of claim 1 further comprising:

3. means for generating product suggestions and special offers based on the results of the user sentiment analysis; means for displaying the generated proposals or offers on a user terminal; The system of claim 1 further comprising:

Images