System

Abstract

A system is provided.SOLUTION: A system, comprising: means for detecting a user's approach; means for initiating a conversation with the user and collecting the user's needs; means for sending the collected user's needs to a server; means for the server selecting an optimal product based on the user's needs and generating a suggestion list; means for presenting the generated suggestion list to the user; and means for collecting the user's reaction again and sending it to the server to optimize the next suggestion.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Current sales methods in shopping malls have limitations, particularly a lack of mechanisms to proactively address the individual needs of each customer. Furthermore, manual sales methods make it difficult to fully grasp each customer's individual needs in a short amount of time, resulting in a tendency for customer experiences to become uniform. This can lead to lower customer satisfaction and lower contract acquisition rates. This invention aims to improve the customer purchasing experience by utilizing the latest AI technology to generate personalized proposals in real time based on the customer's interests and needs. [Means for solving the problem]

[0005] This invention is a system including means for detecting the approach of a user, means for starting a conversation with the user and collecting the user's needs, means for transmitting the collected user needs to a server, means for the server to select optimal products based on the user's needs and generate a proposal list, means for presenting the generated proposal list to the user, and means for collecting the user's reactions again and transmitting them to the server to optimize the next proposal.

[0006] Furthermore, it is equipped with a means for converting conversations with users into text data using voice recognition technology and sending it to a server, as well as a means for recognizing the user's face and identifying them by linking them to past request data. In this way, it becomes possible to respond precisely to user needs and provide efficient, highly accurate personalized suggestions.

[0007] "User" refers to a customer visiting a shopping mall who receives product suggestions through dialogue with a sales assistant robot.

[0008] The "terminal" refers to a sales assistant robot, a device that collects needs through dialogue with the user and sends them to a server.

[0009] "Server" refers to the central processing unit of the system that analyzes collected user needs data and generates optimal product proposals.

[0010] "Means of detection" refers to technologies such as sensors and cameras that recognize when a user approaches.

[0011] "Means of collection" refers to technology that collects needs and requests through dialogue with users, and includes functions such as voice recognition and text conversion.

[0012] "Means for transmitting" refers to the communication technology used to transmit the collected data to the server.

[0013] "Means for generating a proposal list" refers to an algorithm or program that the server uses to select optimal products based on the user's needs and create that list.

[0014] "Presentation means" refers to a display or voice synthesis technology that displays the generated proposal list to the user in an easy-to-understand manner.

[0015] The "means for collecting responses" refers to a technology for collecting responses to the proposed product from users and sending them back to the server.

[0016] "Speech recognition technology" refers to technology for converting a user's speech into text data.

[0017] "Facial recognition technology" refers to technology that identifies a user's face and links it to past request data. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0019] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0022] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0023] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0024] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0026] [First embodiment]

[0027] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0028] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0031] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0034] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0038] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0039] This invention is an AI sales assistant robot system that guides users visiting shopping malls about products such as smartphones and water dispensers. This system operates mainly through interactions with terminals, servers, and users. Below, we provide a detailed explanation of the operation of each element and provide specific examples.

[0040] System configuration

[0041] 1. Terminal (Sales Assistant Robot):

[0042] Dialogue directly with users to gather their needs.

[0043] Built-in sensors and cameras may be used to detect the user's approach and facial recognition technology may be used to identify the user.

[0044] Using voice recognition technology, the user's questions and requests are converted into text data and sent to the server.

[0045] 2. Server:

[0046] The user's needs data transmitted from the terminal is received and analyzed.

[0047] Using AI algorithms, we generate product lists that best suit the user's needs.

[0048] The generated proposal list is sent to the terminal.

[0049] It accumulates user response data and performs self-learning to improve the accuracy of its next suggestions.

[0050] Program processing explanation

[0051] Device behavior

[0052] 1. User approach detection:

[0053] The device uses its built-in camera and sensors to detect when a user approaches, along with facial recognition technology to determine whether the user is a known or new user.

[0054] 2. Initiating the initial dialogue:

[0055] When the device detects a user approaching, it greets them with "Hello! Is there anything you're looking for?"

[0056] 3. Gathering needs:

[0057] If the user answers, "I'm looking for a new smartphone," the device will then ask, "What features are you looking for in a smartphone?" If the user answers, "I want a smartphone with a good camera," the device will send this information to the server.

[0058] Server Operation

[0059] 1. Data Receipt and Analysis:

[0060] The server receives the user's needs ("a smartphone with a good camera") sent from the device, searches the database using an AI algorithm, and generates the optimal product list.

[0061] 2. Generate and send the proposal list:

[0062] The server transmits the generated product list to the terminal.

[0063] Handling suggestions and feedback

[0064] 1. Presentation of the proposal:

[0065] The device advises the user, "This smartphone has excellent camera performance. Its high resolution makes it particularly good for night photography."

[0066] 2. Collecting user responses:

[0067] If the user responds with, "That sounds good. Are there any other options?" the device sends that information back to the server and generates additional suggestions.

[0068] 3. Generate and present re-proposals:

[0069] The server searches the database again based on the new needs (price is also important), generates an optimal product list, and sends it to the device. The device then re-suggests to the user, "This smartphone also has a high-performance camera, but it's more reasonably priced."

[0070] Specific examples

[0071] 1. The user approaches and is detected:

[0072] Terminal: "Hello! Is there anything you're looking for?"

[0073] User: "Hi. I'm looking for a new smartphone."

[0074] 2. Make suggestions based on needs:

[0075] Device: "What features of a smartphone are you looking for?"

[0076] User: "I want a smartphone with a good camera."

[0077] The terminal sends this request to the server.

[0078] The server generates a list of optimal smartphones and sends it to the device.

[0079] Device: "The camera on this smartphone is excellent. It's particularly high-resolution and great for night photography."

[0080] 3. The user makes an additional request:

[0081] User: "That sounds good. Are there any other options?"

[0082] The device sends a new request to the server, gets another product list, and makes another suggestion.

[0083] Device: "This phone also has a great camera, but it's more affordable."

[0084] Continuous self-learning

[0085] The server accumulates all interaction data and continuously updates the model to optimize the next suggestion, thus providing each user with the best possible suggestions in real time.

[0086] In this way, the present invention efficiently delivers personalized offers tailored to each customer's individual needs, enhancing the customer experience.

[0087] The processing flow will be explained below.

[0088] Step 1:

[0089] The device uses its built-in camera and sensors to detect the user's approach, and may then use facial recognition technology to identify the user.

[0090] Step 2:

[0091] When the device approaches a user, it greets them with "Hello! Is there anything you're looking for?"

[0092] Step 3:

[0093] The user replies, "I'm looking for a new phone."

[0094] Step 4:

[0095] The device converts the user's speech into text data using voice recognition technology and continues to ask questions such as, "What features of a smartphone are you looking for?" At this time, it presents options such as camera performance and battery life.

[0096] Step 5:

[0097] The user responds, "I want a smartphone with a good camera." The device converts this response back into text data and sends it to the server.

[0098] Step 6:

[0099] The server analyzes the user's needs ("I want a smartphone with a good camera") and searches the database to generate the optimal product list.

[0100] Step 7:

[0101] The server transmits the generated product list to the terminal.

[0102] Step 8:

[0103] The device will advise the user, "This smartphone has excellent camera performance. Its high resolution makes it particularly good for taking night shots."

[0104] Step 9:

[0105] The user asks, "That sounds good. What other options do I have?"

[0106] Step 10:

[0107] The device converts the new need ("more reasonable options") into text data and sends it to the server.

[0108] Step 11:

[0109] The server searches the database again based on the new needs and generates a further product list.

[0110] Step 12:

[0111] The server transmits the regenerated product list to the terminal.

[0112] Step 13:

[0113] The device will then make a new suggestion to the user, saying, "This smartphone also has a high-performance camera, but it's more reasonably priced."

[0114] Step 14:

[0115] The user responds, "That's great. Can you give me more detailed specs?"

[0116] Step 15:

[0117] The device sends the user's response to the server, and the server generates detailed specification information and sends it to the device.

[0118] Step 16:

[0119] The device provides detailed information to the user, and the server stores all interaction data and updates the model to further optimize the next suggestion.

[0120] Example 1

[0121] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0122] Traditional product guidance in shopping malls relies on face-to-face human resources and is inefficient. It is also difficult to provide personalized suggestions tailored to user needs, making it difficult to quickly and accurately guide users to the products they are looking for. Furthermore, there is a lack of a mechanism for efficiently incorporating user feedback, making it difficult to improve the quality of next suggestions.

[0123] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0124] In this invention, the server includes means for detecting the approach of a user, means for starting a conversation with the user and collecting the user's needs, means for transmitting the collected user needs to the server, means for the server to select optimal products based on the user's needs and generate a proposal list, means for presenting the generated proposal list to the user, means for recollecting the user's reactions and transmitting them to the server to optimize the next proposal, and means for the server to accumulate dialogue data and perform self-learning to optimize the next proposal, thereby enabling efficient and personalized product guidance according to the user's needs.

[0125] "Means for detecting the approach of a user" refers to a device or technology that uses a built-in camera or sensor to detect when a user approaches within a certain range.

[0126] "Means for starting a conversation with the user and gathering user needs" refers to the process of using speech synthesis technology and natural language processing technology to converse with the user and gather information and product requests that the user is looking for.

[0127] The "means for transmitting the collected user needs to the server" refers to a communication means or protocol for transmitting the information collected from the user to the server via a network.

[0128] "Means by which the server selects optimal products based on the user's needs and generates a proposal list" refers to algorithms or technologies by which the server analyzes collected user information, selects relevant products from a database, and generates a list.

[0129] The "means for presenting the generated proposal list to the user" refers to a device such as a display or a speaker that visually or audibly presents the generated product proposal list to the user.

[0130] "Means for collecting user responses again and sending them to the server to optimize future suggestions" refers to the process of collecting the responses that users give to suggestions and sending that information back to the server to improve the quality of future suggestions.

[0131] "Means for the server to accumulate dialogue data and self-learn to optimize the next proposal" refers to a technology in which the server saves past dialogue data with the user, uses that data to apply machine learning technology, and improves the accuracy of the next proposal.

[0132] This invention is an AI sales assistant robot system that guides users visiting shopping malls about products such as smartphones and water dispensers. This system operates mainly through interactions between terminals, servers, and users.

[0133] System configuration

[0134] 1. Terminal (Sales Assistant Robot):

[0135] Approach Detection:

[0136] The device uses a built-in camera (e.g., Logitech HD Pro Webcam C920) and sensors (e.g., HC-SR501 PIR motion sensor) to detect the approach of a user, and uses facial recognition technology (e.g., OpenCV library) to determine whether the user is a known or new user.

[0137] Example: When a user approaches within 2 meters of a robot, the device's sensors detect this and the camera captures the user's face.

[0138] Beginning the initial interaction:

[0139] When the device detects the user's approach, it uses speech synthesis software (e.g., Google® Text-to-Speech API) to greet the user, saying, "Hello! Is there anything I can help you with?"

[0140] Example: When the device says, "Hello! Is there anything I can help you with?" the user turns towards the device in surprise.

[0141] Gathering needs:

[0142] When the user responds by saying, "I'm looking for a new smartphone," the device uses voice recognition technology (e.g., Google Cloud Speech-to-Text API) to convert this request into text data and send it to the server.

[0143] Example: When a user says, "I'm looking for a new smartphone," the device converts it into text and sends it to the server as "I'm looking for a new smartphone."

[0144] 2. Server:

[0145] Data reception and analysis:

[0146] The server receives user needs (e.g., "a smartphone with a good camera") sent from the device, searches the database using AI algorithms (e.g., Python and the Scikit-learn library), and generates the optimal product list. This uses a generative AI model such as the XGBoost model.

[0147] Example: The server receives the need "Looking for a new smartphone" and selects a suitable smartphone from the database.

[0148] Generate and send a list of suggestions:

[0149] The server transmits the generated product list to the terminal.

[0150] Example: The server generates a list of "smartphones with good camera performance" and sends it to the device.

[0151] 3. Handling Suggestions and Feedback:

[0152] Presentation of proposal:

[0153] The device advises the user, "This smartphone has excellent camera performance. Its high resolution makes it particularly good for night photography."

[0154] Example: The device suggests to the user, "This smartphone has excellent camera performance."

[0155] User response collection and re-proposal:

[0156] If the user responds with, "That sounds good. Are there any other options?" the device sends that information back to the server and generates additional suggestions.

[0157] Example: When a user asks, "Are there any other options available?", the device sends the request to the server.

[0158] Generate and present re-proposals:

[0159] The server receives the new needs, generates a new appropriate product list, and sends it to the terminal, which then makes new suggestions to the user based on this list.

[0160] Example: The device re-proposes, "This smartphone also has a high-performance camera, but at a more reasonable price."

[0161] 4. Continuous self-learning:

[0162] The server stores all interaction data and continuously trains the AI ​​model to optimize the next suggestion, resulting in more personalized suggestions for the next user.

[0163] Example: The server uses user interaction data to update the model to improve the accuracy of the next suggestion.

[0164] The overall process of this system makes it possible to efficiently gather user needs and make optimal product recommendations in real time. By using the above-mentioned means, shopping mall visitors can receive prompt and appropriate product information, providing a more satisfying shopping experience.

[0165] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0166] System program processing flow

[0167] Step 1:

[0168] User approach detection

[0169] The device uses its built-in camera and sensors to constantly monitor whether a user is approaching. When the user approaches within a certain range, the device uses facial recognition technology (e.g., OpenCV library) to recognize the user. At this time, the device's camera captures video and the sensor receives distance data as input. Based on this data, the device runs a facial recognition algorithm to output facial information.

[0170] Specific operations: camera image capture, sensor data acquisition, face recognition

[0171] Input: Camera image data, sensor data

[0172] Output: Face recognition result (user is new or known)

[0173] Step 2:

[0174] Starting the initial dialogue

[0175] When the device detects the user's proximity, it uses speech synthesis software (e.g., Google Text-to-Speech API) to greet the user with "Hello! Is there anything I can help you with?" This speech is transmitted directly to the user through the speaker.

[0176] Specific operations: speech synthesis, speech output from speaker

[0177] Input: Approach detection result

[0178] Output: Synthesized speech greeting

[0179] Step 3:

[0180] Gathering needs

[0181] When the user responds by saying "I'm looking for a new smartphone," the device uses voice recognition technology (e.g., Google Cloud Speech-to-Text API) to convert this request into text data, which is then sent to the server for further processing.

[0182] Specific operations: speech recognition, text data generation

[0183] Input: User voice input

[0184] Output: Text data (e.g. "Looking for a new smartphone")

[0185] Step 4:

[0186] Sending data

[0187] The terminal sends the converted text data to the server, often using HTTP or WebSocket as the communication protocol.

[0188] Specific operations: sending data, using communication protocols

[0189] Input: Text data

[0190] Output: Send text data to the server

[0191] Step 5:

[0192] Data reception and analysis

[0193] The server receives the user needs data sent from the device and analyzes it using an AI algorithm (e.g., Python and the Scikit-learn library). The server then accesses a database and generates a list of products that best fit the user's needs. This uses a generative AI model such as the XGBoost model.

[0194] Specific operations: receiving data, analyzing it with AI algorithms, and executing database queries

[0195] Input: User needs data

[0196] Output: Optimal product list

[0197] Step 6:

[0198] Generate and send a list of suggestions

[0199] The server generates an optimal product list based on the analysis results and sends it to the terminal. This list includes detailed product information (e.g., product name, specifications, price, etc.).

[0200] Specific operations: Generate product list, send data

[0201] Input: Analysis results of the AI ​​algorithm

[0202] Output: Send product list to terminal

[0203] Step 7:

[0204] Proposal presentation

[0205] Based on the product list received from the server, the device makes suggestions to the user via voice, providing specific explanations such as, "This smartphone has excellent camera performance."

[0206] Specific operations: speech synthesis, speech output from speaker

[0207] Input: Product list

[0208] Output: Spoken suggestions

[0209] Step 8:

[0210] Collecting user responses and re-proposing

[0211] When the user makes additional requests, the device converts their responses into text using voice recognition technology again and sends it to the server.

[0212] Specific operations: speech recognition, text data generation, data transmission

[0213] Input: User's voice response

[0214] Output: Send text data to the server

[0215] Step 9:

[0216] Generate and present re-proposals

[0217] The server receives the new request, generates a new appropriate product list, and sends it to the terminal, which then makes new suggestions to the user based on this list.

[0218] Specific operations: receiving data, analyzing with AI algorithms, generating product list, sending data, and synthesizing voice

[0219] Input: New user request

[0220] Output: Re-proposed product list

[0221] Step 10:

[0222] Data accumulation and learning

[0223] The server stores all interaction data and continuously trains the AI ​​model to optimize the next suggestion, resulting in more personalized suggestions for the next user.

[0224] Specific actions: Data accumulation, AI model training, performance improvement

[0225] Input: User interaction data

[0226] Output: Optimized AI model

[0227] (Application example 1)

[0228] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0229] In today's brick-and-mortar stores, it is difficult to quickly propose products that accurately meet customer needs. Furthermore, there is a lack of systems to effectively utilize the information obtained through dialogue with customers, and the accuracy of proposals and the optimization of future proposals are not sufficiently implemented. As a result, improvements to the customer experience are hindered, leading to a decline in store sales and customer satisfaction.

[0230] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0231] In this invention, the server includes means for detecting the approach of a user, means for initiating a conversation with the user and collecting the user's needs, means for transmitting the collected user needs to the server, means for the server to select optimal information based on the user's needs and generate a proposal list, means for presenting the generated proposal list to the user, means for recollecting the user's responses and transmitting them to the server to optimize the next proposal, means for analyzing the user's needs using an AI algorithm and searching a database, and means for self-learning based on the collected feedback to improve the accuracy of proposals from the next time onwards. This enables product proposals that quickly and accurately meet customer needs, improving the customer experience and increasing store sales.

[0232] "Means for detecting the approach of a user" refers to technology that uses a camera or sensor to detect the presence of a user when they arrive within a pre-set distance.

[0233] "Means for starting a conversation with the user and gathering information about the user's needs" refers to a function that uses voice recognition technology and natural language processing to converse with the user and understand their requests and wishes.

[0234] The "means for transmitting collected user needs to a server" is a communication means for transferring collected data to a central database via the Internet.

[0235] "Means for the server to select the most appropriate information based on the user's needs and generate a recommendation list" refers to a function that uses AI algorithms and data analysis technology to find the products and services that best meet the user's requirements and create a recommendation list.

[0236] The "means for presenting the generated proposal list to the user" is a function that uses a display or voice synthesis technology to present information that is determined to be most suitable to the user in an easy-to-understand manner.

[0237] "Means for recollecting user responses and sending them to a server to optimize the next proposal" refers to a technology that collects the interest and feedback that users have shown in the proposal and sends that data to a server to improve the quality of future proposals.

[0238] "Means of analyzing user needs using AI algorithms and searching a database" refers to a function that analyzes needs data collected using machine learning and deep learning, and searches a database for appropriate information.

[0239] "Means for self-learning based on collected feedback to improve the accuracy of future proposals" refers to a system that uses accumulated user feedback data as learning material to improve the AI ​​model and fine-tune the algorithm, thereby improving the accuracy of proposals.

[0240] This invention is an AI sales assistant system for suggesting products according to customer needs in shopping malls and brick-and-mortar stores, and it mainly operates using a terminal (e.g., a smartphone or robot) and a server. The main function of this system is to detect when a user approaches, collect and analyze the user's needs through dialogue, and generate a suggestion list.

[0241] System configuration

[0242] Device configuration

[0243] 1. User approach detection:

[0244] The device uses its built-in camera and sensors to detect approaching users in real time, and uses facial recognition technology (e.g., TENSORFLOW®) to distinguish between existing and new users.

[0245] 2. Initiating conversations and gathering needs:

[0246] When the user approaches, the device greets the user through voice output (e.g., Google Cloud Text-to-Speech) and initiates a dialogue. It then uses voice recognition technology (e.g., Google Cloud Speech-to-Text) to convert the user's request into text data and transmits that data to the server.

[0247] 3. Present the list of proposals and gather feedback:

[0248] The device that receives the proposal list presents it to the user via voice output or a display (e.g., a smartphone or robot screen). The user's reactions and feedback are collected again and sent to the server.

[0249] Server Configuration

[0250] 1. Data Receipt and Analysis:

[0251] The server receives the user's needs data sent from the device and searches the database using an AI algorithm. Specifically, it analyzes the data using a machine learning model (e.g., a generative AI model) to generate an optimal product list.

[0252] 2. Generate the proposal list:

[0253] Based on the analysis results, a product list that best suits the user's needs is generated and sent to the terminal.

[0254] 3. Feedback accumulation and self-learning:

[0255] The server accumulates user feedback data sent from the device and performs self-learning to improve the accuracy of suggestions from the next time onwards, thereby enabling continuous improvement of the accuracy of suggestions.

[0256] Hardware and software used

[0257] Cameras and sensors: Used to detect the user's approach.

[0258] Microphone and speaker: Used for audio input and output.

[0259] Display: Used to visually display the suggestion list.

[0260] TensorFlow: Used to provide facial recognition technology.

[0261] Google Cloud Speech-to-Text: Used to provide speech recognition technology.

[0262] Google Cloud Text-to-Speech: Used to provide audio output.

[0263] Firebase or AWS® DynamoDB: Used to manage data and manage AI models.

[0264] Specific examples

[0265] 1. User A approaches:

[0266] Terminal: "Hello! Is there anything you're looking for?"

[0267] User: "Hi. I'm looking for a new smartphone."

[0268] 2. Make suggestions based on needs:

[0269] Device: "What features of a smartphone are you looking for?"

[0270] User: "I want a smartphone with a good camera."

[0271] The terminal sends this request to the server.

[0272] The server generates a list of optimal smartphones and sends it to the device.

[0273] Device: "The camera on this smartphone is excellent. It's particularly high-resolution and great for night photography."

[0274] 3. The user makes an additional request:

[0275] User: "That sounds good. Are there any other options?"

[0276] The device sends a new request to the server, gets another product list, and makes another suggestion.

[0277] Device: "This phone also has a great camera, but it's more affordable."

[0278] Prompt Sentence Examples

[0279] "Design an AI assistant for a physical store that detects the user's proximity and suggests the best products based on their needs. Write a Python program that recognizes the customer's voice input and generates and presents a list of products."

[0280] As a result, the present invention makes it possible to quickly grasp the needs of customers in physical stores and provide the most suitable products to meet those needs, thereby improving the customer experience and increasing store sales.

[0281] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0282] Step 1:

[0283] The device uses its built-in camera and sensors to detect when a user approaches. Specifically, it analyzes the frames captured by the camera and uses facial recognition technology (e.g., TensorFlow) to identify whether the user is an existing or new user. Here, the input is the camera image frame, and the output is the user approach detection information.

[0284] Step 2:

[0285] When the device detects the user's approach, it uses voice output (e.g., Google Cloud Text-to-Speech) to address the user, saying, "Hello! Is there anything I can help you with?", and begins a dialogue. Here, the input is user detection information, and the output is the start of a conversation with the user.

[0286] Step 3:

[0287] When the user responds, "I'm looking for a new smartphone," the device uses voice recognition technology (e.g., Google Cloud Speech-to-Text) to convert this speech into text data. It then asks, "What features of a smartphone are you looking for?" Here, the input is voice data, and the output is text data.

[0288] Step 4:

[0289] If the user answers, "I want a smartphone with a good camera," the device sends this need data to the server. Here, the input is text data, and the output is the transmission of the need data to the server.

[0290] Step 5:

[0291] The server receives the received needs data and analyzes it using an AI algorithm (e.g., a generative AI model). Specifically, it uses a machine learning model to generate an optimal smartphone list. Here, the input is the needs data and the output is a product list.

[0292] Step 6:

[0293] The server sends the generated product list to the terminal, which then uses voice output and a display to suggest to the user, "This smartphone has excellent camera performance. Its high resolution makes it particularly good for nighttime photography." Here, the input is the product list, and the output is the suggestion to the user.

[0294] Step 7:

[0295] The user responds with "That sounds good. Are there any other options?" and the device sends this feedback back to the server. Here, the input is the user's feedback and the output is sending the feedback to the server.

[0296] Step 8:

[0297] The server analyzes the feedback data and generates an additional proposal list. The server then sends the generated proposal list to the device, which then makes a re-proposal, saying, "This smartphone also has a high-performance camera, but it's more reasonably priced." Here, the input is the feedback data, and the output is the new product list and the re-proposal.

[0298] Step 9:

[0299] The server accumulates all the interaction data and performs self-learning to optimize the next suggestion, thereby improving the accuracy of suggestions for the user's future requests. Here, the input is all the interaction data, and the output is the updated AI model.

[0300] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0301] This invention is an AI sales assistant robot system that guides users visiting shopping malls about products such as smartphones and water dispensers. This system operates primarily through dialogue between terminals, servers, and users, and by combining it with an emotion engine, it recognizes the user's emotions and optimizes the content of its recommendations. Below, we provide a detailed explanation of the operation of each element and provide specific examples.

[0302] System configuration

[0303] 1. Terminal (Sales Assistant Robot):

[0304] Dialogue directly with users to gather their needs.

[0305] Built-in sensors and cameras may be used to detect the user's approach and facial recognition technology may be used to identify the user.

[0306] Using voice recognition technology, the user's questions and requests are converted into text data and sent to the server.

[0307] Equipped with an emotion engine, it recognizes emotions in real time from the user's facial expressions and tone of voice.

[0308] 2. Server:

[0309] The system receives and analyzes the user's needs data and emotion data transmitted from the terminal.

[0310] Using AI algorithms, we generate product lists that best suit the user's needs.

[0311] Based on emotional data, suggestions are optimized to suit the user's emotional state.

[0312] The generated proposal list is sent to the terminal.

[0313] It accumulates user response data and performs self-learning to improve the accuracy of its next suggestions.

[0314] Program processing explanation

[0315] Device behavior

[0316] 1. User approach detection:

[0317] The device uses its built-in camera and sensors to detect when a user approaches, along with facial recognition technology to determine whether the user is a known or new user.

[0318] 2. Initiating the initial dialogue:

[0319] When the device detects a user approaching, it greets them with "Hello! Is there anything you're looking for?"

[0320] 3. Gathering needs:

[0321] The user responds, "I'm looking for a new smartphone." The device then asks, "What features are you looking for in a smartphone?" and offers options such as camera performance and battery life.

[0322] 4. Emotion recognition:

[0323] The user responds, "I want a smartphone with a good camera." During this utterance, the device's emotion engine analyzes the user's emotions in real time from their facial expressions and tone of voice, and sends that data to the server.

[0324] Server Operation

[0325] 1. Data Receipt and Analysis:

[0326] The server receives the user's needs ("a smartphone with a good camera") and emotional data sent from the device, searches the database using an AI algorithm, and generates the optimal product list.

[0327] 2. Generate and send the proposal list:

[0328] The server then sends the generated product list to the terminal, taking into consideration the emotional data and prioritizing expressions and products that are likely to interest the user.

[0329] Handling suggestions and feedback

[0330] 1. Presentation of the proposal:

[0331] The device advises the user, "This smartphone has excellent camera performance. Its high resolution makes it particularly good for night photography."

[0332] 2. Collecting user responses:

[0333] The user asks, "That sounds good. Are there any other options?" The device's emotion engine again analyzes the user's emotions and sends the results to the server.

[0334] 3. Generate and present re-proposals:

[0335] The server searches the database again based on the new need ("more reasonable price") and emotion data, generates a more optimal product list, and sends it to the device. The device then makes a new suggestion to the user, saying, "This smartphone also has a high-performance camera, but at a more reasonable price."

[0336] Specific examples

[0337] 1. The user approaches and is detected:

[0338] Terminal: "Hello! Is there anything you're looking for?"

[0339] User: "Hi. I'm looking for a new smartphone."

[0340] 2. Make suggestions based on needs:

[0341] Device: "What features of a smartphone are you looking for?"

[0342] User: "I want a smartphone with a good camera."

[0343] The terminal transmits this request and emotion data to the server.

[0344] The server generates a list of optimal smartphones and sends it to the device.

[0345] Device: "The camera on this smartphone is excellent. It's particularly high-resolution and great for night photography."

[0346] 3. The user makes an additional request:

[0347] User: "That sounds good. Are there any other options?"

[0348] The device sends a new request and emotion data to the server, retrieves another product list, and makes another suggestion.

[0349] Device: "This phone also has a great camera, but it's more affordable."

[0350] Continuous self-learning

[0351] The server accumulates all conversation and emotion data and continuously updates the model to optimize the next recommendation, thereby providing each user with the best possible recommendation in real time, providing a better purchasing experience.

[0352] In this way, the present invention efficiently delivers personalized offers tailored to each customer's individual needs and emotions, enhancing the customer experience.

[0353] The processing flow will be explained below.

[0354] Step 1:

[0355] The device uses its built-in camera and sensors to detect the user's approach, and may then use facial recognition technology to identify the user.

[0356] Step 2:

[0357] When the device approaches a user, it greets them with "Hello! Is there anything you're looking for?"

[0358] Step 3:

[0359] The user replies, "I'm looking for a new phone."

[0360] Step 4:

[0361] The device converts the user's speech into text data using voice recognition technology and continues to ask questions such as, "What features of a smartphone are you looking for?" At this time, it presents options such as camera performance and battery life.

[0362] Step 5:

[0363] The user responds, "I want a smartphone with a good camera." The device converts this response back into text data and sends it to the server. The emotion engine analyzes the user's facial expressions and tone of voice, and sends the user's emotional data to the server.

[0364] Step 6:

[0365] The server analyzes the user's needs ("I want a smartphone with a good camera") and emotional data received. The server searches the database and uses an AI algorithm to generate an optimal product list. At this time, the server takes into account the emotional data and prioritizes products that are likely to interest the user.

[0366] Step 7:

[0367] The server transmits the generated product list to the terminal.

[0368] Step 8:

[0369] The device will advise the user, "This smartphone has excellent camera performance. Its high resolution makes it particularly good for taking night shots."

[0370] Step 9:

[0371] When a user asks, "That sounds good. Are there any other options?", the emotion engine analyzes the user's facial expressions and tone of voice to obtain emotional data.

[0372] Step 10:

[0373] The device again sends the new need ("more reasonable options") and emotion data to the server.

[0374] Step 11:

[0375] The server searches the database again based on the new needs and emotional data, and uses an AI algorithm to generate an even more optimal product list, taking into account the emotional data and including products that are likely to interest the user.

[0376] Step 12:

[0377] The server transmits the regenerated product list to the terminal.

[0378] Step 13:

[0379] The device will then make a new suggestion to the user, saying, "This smartphone also has a high-performance camera, but it's more reasonably priced."

[0380] Step 14:

[0381] The user responds, "That's great. Please tell me more about the specs." The emotion engine analyzes the user's facial expression and tone of voice at this time to obtain emotional data.

[0382] Step 15:

[0383] The device sends the user's reaction and emotional data to the server, which then generates detailed specification information and sends it to the device using the most appropriate expressions based on the emotional data.

[0384] Step 16:

[0385] The device provides detailed specifications to the user, and sends the user's reaction and emotional data to the server, which then accumulates all the data and updates the model to optimize the next proposal.

[0386] Example 2

[0387] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0388] Conventional customer service systems in shopping malls have had difficulty fully reflecting individual emotions and detailed requests when proposing products that meet users' needs. As a result, they have been unable to make appropriate proposals that stimulate purchasing motivation, and their effectiveness in improving customer satisfaction and promoting sales has been limited. The object of the present invention is to solve these problems by providing a system that recognizes users' emotions in real time and makes appropriate product proposals.

[0389] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0390] In this invention, the server includes means for detecting the approach of a user, means for starting a conversation with the user and collecting the user's needs, means for transmitting the collected user needs and emotion data to the server, means for the server to select optimal products based on the user's needs and emotion data and generate a proposal list, means for presenting the generated proposal list to the user, means for recollecting the user's reactions and transmitting them to the server to optimize the next proposal, and means for grasping the user's emotions in real time using an emotion recognition engine. This makes it possible to flexibly respond to the individual emotions and needs of the user and make optimal product proposals.

[0391] "User" refers to an individual who receives product information from the system.

[0392] "Server" refers to a central management unit that analyzes data sent by users and generates optimal product suggestions.

[0393] "Terminal" refers to a device that directly interacts with users, gathers their needs, and connects to a server.

[0394] "Emotional data" refers to information that indicates the emotional state of a user, as determined by facial expressions and vocal tones.

[0395] "Proposal list" refers to a list of optimal products generated by the server and presented to the user.

[0396] An "emotion recognition engine" refers to software that analyzes a user's facial expressions and tone of voice in real time to determine their emotions.

[0397] "Speech recognition technology" refers to technology that converts a user's speech into text data.

[0398] "Facial recognition technology" refers to technology that uses a camera to identify a user's face and determine whether it is familiar or new.

[0399] MODE FOR CARRYING OUT THE INVENTION

[0400] (System Overview)

[0401] This invention is an AI sales assistant robot system that guides users visiting a shopping mall to the most suitable products. This system operates through dialogue with the user and, by combining it with an emotion recognition engine, recognizes the user's emotions and optimizes the content of the recommendations. Specific embodiments for implementing this system are described below.

[0402] (Device configuration and operation)

[0403] The terminal will be installed in shopping malls as a sales assistant robot and is equipped with a built-in camera, infrared sensors, voice recognition technology, and an emotion recognition engine.

[0404] User approach detection

[0405] The device uses a built-in camera and infrared sensor to detect the user's movement when they approach within a certain distance. Facial recognition technology is used to determine whether the user is an existing or new user. For example, the built-in camera can be a standard webcam, and the infrared sensor can be a commercially available motion sensor. Facial recognition uses technologies such as Face++.

[0406] Starting the initial dialogue

[0407] When a user approaches, the device will speak a pre-programmed initial greeting (e.g., "Hello! Is there anything I can help you with?") using voice synthesis technology, which can use technologies such as Google Text-to-Speech.

[0408] Gathering needs

[0409] If the user responds, "I'm looking for a new smartphone," the device uses voice recognition software (e.g., Google Cloud Speech-to-Text) to convert the utterance into text data, then asks, "What features are you looking for in a smartphone?" and presents options such as "camera performance," "battery life," and "price range."

[0410] emotion recognition

[0411] When a user answers, "I want a smartphone with a good camera," the device's emotion recognition engine (e.g., Affectiva) is used to determine the user's emotion in real time from their facial expressions and tone of voice. This emotion data is then sent to the server along with the user's response.

[0412] (Server configuration and operation)

[0413] The server acts as a central control unit that analyzes the data sent from the terminals and generates product suggestions. The following processes are performed:

[0414] Data reception and analysis

[0415] The server receives the user's needs (e.g., "a smartphone with a good camera") and emotional data sent from the device. It uses an AI algorithm to search the database and generate the optimal product list. The AI ​​algorithm uses a natural language processing model using TensorFlow.

[0416] Generate and send a list of suggestions

[0417] The server then sends the generated product list to the device. It considers the user's emotional data and prioritizes expressions and products that are likely to interest the user. If the user is excited, the latest high-performance camera smartphones are placed at the top of the list.

[0418] (Suggestion and feedback handling)

[0419] The terminal makes suggestions to the user based on the product list sent from the server.

[0420] Proposal presentation

[0421] The device will suggest to the user, "This smartphone has excellent camera performance, especially with high resolution and excellent night photography." By emphasizing specific features (e.g., the night photography function "Night Sight"), it will attract the user's interest.

[0422] Collecting user responses

[0423] The user asks a follow-up question, "That sounds good. Are there any other options?" The device's emotion recognition engine again analyzes the user's facial expressions and tone of voice and sends the results to the server.

[0424] Generate and present re-proposals

[0425] The server searches the database again based on the new need (e.g., "reasonable price") and emotion data, generates a more suitable product list, and sends it to the device. The device then makes a new suggestion, saying, "This smartphone also has a high-performance camera, but at a more reasonable price."

[0426] (Continuous self-learning)

[0427] The server accumulates all conversation and emotion data and continually updates the AI ​​model to improve the accuracy of the next recommendation. For example, if a user frequently requests "reasonable prices," the server will prioritize presenting options with lowered prices to that user.

[0428] Example prompt

[0429] Example prompt: "User is looking for a new smartphone. Suggest a smartphone that focuses on camera performance, and if they inquire about price, suggest a more affordable option."

[0430] In this way, the present invention can flexibly respond to the individual needs and feelings of the user and make optimal product suggestions.

[0431] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0432] Program processing flow

[0433] Step 1: Detecting user approach

[0434] Input: The device receives data from the built-in camera and infrared sensor.

[0435] Data processing: Analyzes camera footage and reads motion sensor signals.

[0436] Output: Detects that the user is approaching within a certain distance.

[0437] Specific operation: The device analyzes camera footage in real time, identifies the user's face using facial recognition technology, and detects the user's approach from the infrared sensor signal to determine whether the user is a new or existing user.

[0438] Step 2: Initiating initial interactions

[0439] Input: Signal that detects the user's approach.

[0440] Data processing: Generation of greetings and speech synthesis.

[0441] Output: The device says "Hello! Is there anything I can help you with?"

[0442] Specific behavior: Uses the device's speech synthesis technology (e.g., Google Text-to-Speech) to audibly transmit a programmed greeting to the user.

[0443] Step 3: Gathering needs

[0444] Input: User utterance (e.g., "I'm looking for a new smartphone").

[0445] Data processing: Convert speech into text data using speech recognition technology.

[0446] Output: The text data is "Looking for a new smartphone."

[0447] What it does: The device uses voice recognition software (e.g., Google Cloud Speech-to-Text) to convert what the user says into text in real time, analyzes that text, and generates the next question (e.g., "What features of a smartphone are you looking for?").

[0448] Step 4: Emotion Recognition

[0449] Input: User's speech and facial expression data.

[0450] Data processing: Analyze user emotions using an emotion recognition engine.

[0451] Output: Sentiment data as the analysis result.

[0452] Specific operation: The device uses an emotion recognition engine (e.g., Affectiva) to determine the user's emotions (joy, anger, sadness, and happiness) from their facial expressions and vocal tones. The obtained emotional data is then sent to the server along with the user's needs.

[0453] Step 5: Server receives and analyzes data

[0454] Input: User needs and emotion data sent from the device.

[0455] Data processing: Using AI algorithms, we generate product lists that best suit the user's needs.

[0456] Output: A list of optimal products.

[0457] How it works: The server analyzes user needs using a natural language processing model powered by TensorFlow, searches the database to select the most suitable products, and adjusts the order of the list taking into account emotional data.

[0458] Step 6: Generate and submit a proposal list

[0459] Input: Product list generated by an AI algorithm.

[0460] Data processing: Adjust product priorities based on user sentiment.

[0461] Output: Sending the product list to the terminal.

[0462] Specific operation: The server sends the generated product list to the terminal. At this time, expressions that are likely to interest the user are given priority in the list.

[0463] Step 7: Proposal Presentation

[0464] Input: Product list sent from the server.

[0465] Data processing: Audio conversion of product list.

[0466] Output: Spoken product recommendations to the user.

[0467] Specific operation: The device will suggest to the user using a synthesized voice, "This smartphone has excellent camera performance. Its high resolution makes it particularly good for night photography."

[0468] Step 8: Collect user responses

[0469] Input: User's follow-up question (e.g., "That sounds good. Are there any other options I can choose?").

[0470] Data processing: The follow-up questions are converted into text data using voice recognition technology. The emotion data is then analyzed again using an emotion recognition engine.

[0471] Output: A new request to the server and sending emotion data.

[0472] Specific operation: The device performs voice recognition of the follow-up question, re-determines the user's current emotional state using the emotion recognition engine, and sends the analysis results to the server.

[0473] Step 9: Generate and present re-proposals

[0474] Input: A new product list from the server.

[0475] Data processing: Audio generation of new product listings.

[0476] Output: A spoken re-suggestion to the user.

[0477] Specific operation: The device makes a re-suggestion by synthesizing voice, saying, "This smartphone also has a high-performance camera, but it's more reasonably priced."

[0478] Prompt Sentence Examples

[0479] Example prompt: "User is looking for a new smartphone. Suggest a smartphone that focuses on camera performance, and if they inquire about price, suggest a more affordable option."

[0480] (Application example 2)

[0481] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0482] Conventional shopping support systems have difficulty accurately gathering user needs and suggesting optimal products. Furthermore, because they do not take the user's emotions into consideration, the content of the suggestions is unlikely to capture the user's interest and fail to stimulate purchasing motivation. Furthermore, there is a lack of personalized suggestions linked to past request data, making it impossible to achieve optimization for each user. There is a need to solve these problems and provide a more effective customer experience.

[0483] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0484] In this invention, the server includes means for detecting the approach of a user, means for initiating a conversation with the user and collecting the user's needs, means for transmitting the collected user needs to the server, means for the server to select optimal products based on the user's needs and generate a proposal list, means for presenting the generated proposal list to the user, means for recollecting the user's reactions and transmitting the data to the server to optimize the next proposal, means for recognizing the user's emotions in real time using emotion recognition technology and transmitting the data to the server, and means for optimizing the proposal content based on the user's emotion data. This makes it possible to make personalized proposals according to the user's needs and improve the customer experience.

[0485] "Means for detecting a user's approach" refers to technology or devices that use cameras or sensors to detect a user's approach to the system.

[0486] "Means for starting a conversation with a user and gathering information about the user's needs" refers to technology or devices that use voice recognition or natural language processing to engage in a dialogue with a user and gather information about the user's requests and wishes.

[0487] The "means for transmitting collected user needs to a server" refers to a communication technology or device for transferring the user needs collected by a terminal to a server as data.

[0488] "Means for the server to select optimal products based on the user's needs and generate a proposal list" refers to technology or software that allows the server to analyze the user's needs, select optimal products based on those needs, and create a list.

[0489] The "means for presenting the generated proposal list to the user" refers to a technique or device for visually or audibly presenting the generated proposal list to the user.

[0490] "Means for re-collecting user responses and sending them to a server to optimize the next proposal" refers to technology or devices for re-collecting user responses to a proposal, sending them to a server, and optimizing the content of the next proposal based on that data.

[0491] "Means for recognizing a user's emotions in real time using emotion recognition technology and transmitting that data to a server" refers to technology or devices that analyze a user's facial expressions and tone of voice to recognize emotions and transmit the results to a server in real time.

[0492] "Means for optimizing proposal content based on user emotion data" refers to technology or devices for optimizing the proposed products or presentation content by taking into account the recognized user emotion data.

[0493] This invention is a smart shopping assistant system that interacts with customers in physical stores and makes product recommendations. This system consists of the following main components:

[0494] Components

[0495] Hardware

[0496] 1. Sales assistant robot: A device that directly interacts with users and gathers their needs. It may use built-in cameras, microphones, and sensors to detect the user's approach and identify them using facial recognition technology.

[0497] 2. Built-in camera and microphone: Captures the user's facial expressions and voice, providing data for emotion recognition technology.

[0498] 3. Approach detection sensor: Sensor technology that detects the approach of a user.

[0499] software

[0500] 1. Emotion recognition engine: Technology that analyzes the user's facial expressions and vocal tone in real time to recognize their emotional state. It uses technologies such as "Affectiva" and "Microsoft® Azure® Emotion API."

[0501] 2. Speech recognition system: A technology that converts user speech into text data. Services such as "Google Cloud Speech-to-Text" and "Amazon Transcribe" are used.

[0502] 3. Database management system: Used to store data on user needs and emotions and make optimal product recommendations. Examples include MySQL (registered trademark) and MongoDB.

[0503] 4. Natural language processing engine: A technology used to analyze collected data and make product suggestions based on user needs. Engines used include "GPT-4 (registered trademark) by OpenAI (registered trademark)" and "BERT."

[0504] Operation explanation

[0505] Device behavior

[0506] 1. User approach detection: The sales assistant robot's built-in camera and sensors detect the user's approach and identify the user using facial recognition technology.

[0507] 2. Start of initial interaction: When the robot detects the user's approach, it greets them with "Hello! Is there anything I can help you with?"

[0508] 3. Gathering needs: If the user responds, "I'm looking for a new smartphone," the robot will ask for more specific needs, such as, "Are you looking for a smartphone with a good camera?"

[0509] 4. Emotion recognition: During this speech, the emotion recognition engine processes the user's facial expressions and tone of voice in real time and sends the emotional data to the server.

[0510] Server Operation

[0511] 1. Data reception and analysis: The server receives the user's needs and emotion data sent from the device and analyzes the data using a natural language processing engine.

[0512] 2. Generate and send a list of proposals: Search the database, generate an optimal product list based on the user's needs and emotional data, and send it to the terminal.

[0513] Handling suggestions and feedback

[0514] 1. Proposal presentation: The sales assistant robot makes a proposal to the user, saying, "This smartphone has excellent camera performance. It has high resolution and is particularly good at taking night shots."

[0515] 2. Collecting user responses: If the user asks, "That sounds good. Are there any other options?", the robot sends a new request and emotional data to the server.

[0516] 3. Generation and presentation of a new recommendation: The server generates a more optimal product list based on the new needs and emotional data and sends it to the device. The robot makes a new recommendation, saying, "This smartphone also has a high-performance camera, but it's more reasonably priced."

[0517] Examples and prompts

[0518] Examples:

[0519] The user approaches and the approach is detected.

[0520] Robot: "Hello! Is there anything you're looking for?"

[0521] User: "Hi. I'm looking for a new smartphone."

[0522] Example prompt sentence:

[0523] The user approaches and the approach is detected.

[0524] Robot: "Hello! Is there anything you're looking for?"

[0525] User: "Hi. I'm looking for a new smartphone."

[0526] Robot: "What features are you looking for in a smartphone?"

[0527] User: "I want a smartphone with a good camera."

[0528] The robot sends this request and emotional data to the server.

[0529] The server generates a list of optimal smartphones and sends it to the robot.

[0530] Robot: "The camera on this smartphone is very good. It's especially good at taking high-resolution photos at night."

[0531] User: "That sounds good. Are there any other options?"

[0532] The robot sends a new request and emotion data to the server and gets the suggestion list again.

[0533] Robot: "This smartphone also has a high-performance camera, but at a reasonable price."

[0534] In this way, the smart shopping assistant system can highly personalize the customer experience in physical stores and efficiently make optimal product recommendations.

[0535] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0536] Step 1:

[0537] User approach detection

[0538] Input: Real-time video data from cameras and sensors.

[0539] How it works: The sales assistant robot (terminal) monitors its surroundings using a built-in camera and proximity detection sensors, and detects when a user approaches. It uses facial recognition technology to determine whether the user is a known or new user.

[0540] Output: Information on the detection of the user's approach and the user ID identified as the face recognition result.

[0541] Step 2:

[0542] Starting the initial dialogue

[0543] Input: Information on the user's approach detection and user ID.

[0544] How it works: The device greets the user with a voice message saying, "Hello! Is there anything I can help you with?" and begins the conversation.

[0545] Output: A spoken response from the user.

[0546] Step 3:

[0547] Gathering needs

[0548] Input: A spoken response from the user.

[0549] How it works: The device uses a voice recognition system (Google Cloud Speech-to-Text) to convert the user's voice into text data, then asks, "What kind of smartphone are you looking for?"

[0550] Output: User needs information as converted text data.

[0551] Step 4:

[0552] emotion recognition

[0553] Input: User's voice response and video data.

[0554] How it works: The device uses an emotion recognition engine (Affectiva or Microsoft Azure Emotion API) to analyze the user's facial expressions and voice tone in real time and generate emotion data.

[0555] Output: User emotion data.

[0556] Step 5:

[0557] Sending data

[0558] Input: User needs information and emotion data.

[0559] Operation: The terminal sends the collected data to the server.

[0560] Output: User needs information and emotion data sent to the server.

[0561] Step 6:

[0562] Generate a list of suggestions

[0563] Input: User needs information and emotion data sent to the server.

[0564] How it works: The server analyzes the data using a natural language processing engine (GPT-4 by OpenAI, BERT, etc.), searches for relevant product information in a database management system (MySQL or MongoDB), generates an optimal product list, and prioritizes it taking into account sentiment data.

[0565] Output: The generated optimal product list.

[0566] Step 7:

[0567] Presenting the proposal

[0568] Input: Generated optimal product list.

[0569] What it does: The device recommends the product to the user, saying, "This smartphone has excellent camera performance. Its high resolution makes it particularly good for nighttime photography."

[0570] Output: Product information presented to the user.

[0571] Step 8:

[0572] Collecting user responses

[0573] Input: User response (voice, facial expression).

[0574] How it works: The device uses its voice recognition system and emotion recognition engine to again collect the user's additional requests and reactions and sends this data to the server.

[0575] Output: Newly collected user needs information and emotion data.

[0576] Step 9:

[0577] Generate and present re-proposals

[0578] Input: Newly collected user needs information and emotion data.

[0579] How it works: The server again analyzes the data using a natural language processing engine, searches the database, and generates a new list of suggestions. The list is then sent to the device, which then makes a new suggestion, saying, "This smartphone also has a high-performance camera, but it's more reasonably priced."

[0580] Output: Product information re-presented to the user.

[0581] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0582] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0583] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0584] [Second embodiment]

[0585] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0586] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0587] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0588] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0589] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0590] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0591] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0592] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0593] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0594] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0595] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0596] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0597] This invention is an AI sales assistant robot system that guides users visiting shopping malls about products such as smartphones and water dispensers. This system operates mainly through interactions with terminals, servers, and users. Below, we provide a detailed explanation of the operation of each element and provide specific examples.

[0598] System configuration

[0599] 1. Terminal (Sales Assistant Robot):

[0600] Dialogue directly with users to gather their needs.

[0601] Built-in sensors and cameras may be used to detect the user's approach and facial recognition technology may be used to identify the user.

[0602] Using voice recognition technology, the user's questions and requests are converted into text data and sent to the server.

[0603] 2. Server:

[0604] The user's needs data transmitted from the terminal is received and analyzed.

[0605] Using AI algorithms, we generate product lists that best suit the user's needs.

[0606] The generated proposal list is sent to the terminal.

[0607] It accumulates user response data and performs self-learning to improve the accuracy of its next suggestions.

[0608] Program processing explanation

[0609] Device behavior

[0610] 1. User approach detection:

[0611] The device uses its built-in camera and sensors to detect when a user approaches, along with facial recognition technology to determine whether the user is a known or new user.

[0612] 2. Initiating the initial dialogue:

[0613] When the device detects a user approaching, it greets them with "Hello! Is there anything you're looking for?"

[0614] 3. Gathering needs:

[0615] If the user answers, "I'm looking for a new smartphone," the device will then ask, "What features are you looking for in a smartphone?" If the user answers, "I want a smartphone with a good camera," the device will send this information to the server.

[0616] Server Operation

[0617] 1. Data Receipt and Analysis:

[0618] The server receives the user's needs ("a smartphone with a good camera") sent from the device, searches the database using an AI algorithm, and generates the optimal product list.

[0619] 2. Generate and send the proposal list:

[0620] The server transmits the generated product list to the terminal.

[0621] Handling suggestions and feedback

[0622] 1. Presentation of the proposal:

[0623] The device advises the user, "This smartphone has excellent camera performance. Its high resolution makes it particularly good for night photography."

[0624] 2. Collecting user responses:

[0625] If the user responds with, "That sounds good. Are there any other options?" the device sends that information back to the server and generates additional suggestions.

[0626] 3. Generate and present re-proposals:

[0627] The server searches the database again based on the new needs (price is also important), generates an optimal product list, and sends it to the device. The device then re-suggests to the user, "This smartphone also has a high-performance camera, but it's more reasonably priced."

[0628] Specific examples

[0629] 1. The user approaches and is detected:

[0630] Terminal: "Hello! Is there anything you're looking for?"

[0631] User: "Hi. I'm looking for a new smartphone."

[0632] 2. Make suggestions based on needs:

[0633] Device: "What features of a smartphone are you looking for?"

[0634] User: "I want a smartphone with a good camera."

[0635] The terminal sends this request to the server.

[0636] The server generates a list of optimal smartphones and sends it to the device.

[0637] Device: "The camera on this smartphone is excellent. It's particularly high-resolution and great for night photography."

[0638] 3. The user makes an additional request:

[0639] User: "That sounds good. Are there any other options?"

[0640] The device sends a new request to the server, gets another product list, and makes another suggestion.

[0641] Device: "This phone also has a great camera, but it's more affordable."

[0642] Continuous self-learning

[0643] The server accumulates all interaction data and continuously updates the model to optimize the next suggestion, thus providing each user with the best possible suggestions in real time.

[0644] In this way, the present invention efficiently delivers personalized offers tailored to each customer's individual needs, enhancing the customer experience.

[0645] The processing flow will be explained below.

[0646] Step 1:

[0647] The device uses its built-in camera and sensors to detect the user's approach, and may then use facial recognition technology to identify the user.

[0648] Step 2:

[0649] When the device approaches a user, it greets them with "Hello! Is there anything you're looking for?"

[0650] Step 3:

[0651] The user replies, "I'm looking for a new phone."

[0652] Step 4:

[0653] The device converts the user's speech into text data using voice recognition technology and continues to ask questions such as, "What features of a smartphone are you looking for?" At this time, it presents options such as camera performance and battery life.

[0654] Step 5:

[0655] The user responds, "I want a smartphone with a good camera." The device converts this response back into text data and sends it to the server.

[0656] Step 6:

[0657] The server analyzes the user's needs ("I want a smartphone with a good camera") and searches the database to generate the optimal product list.

[0658] Step 7:

[0659] The server transmits the generated product list to the terminal.

[0660] Step 8:

[0661] The device will advise the user, "This smartphone has excellent camera performance. Its high resolution makes it particularly good for taking night shots."

[0662] Step 9:

[0663] The user asks, "That sounds good. What other options do I have?"

[0664] Step 10:

[0665] The device converts the new need ("more reasonable options") into text data and sends it to the server.

[0666] Step 11:

[0667] The server searches the database again based on the new needs and generates a further product list.

[0668] Step 12:

[0669] The server transmits the regenerated product list to the terminal.

[0670] Step 13:

[0671] The device will then make a new suggestion to the user, saying, "This smartphone also has a high-performance camera, but it's more reasonably priced."

[0672] Step 14:

[0673] The user responds, "That's great. Can you give me more detailed specs?"

[0674] Step 15:

[0675] The device sends the user's response to the server, and the server generates detailed specification information and sends it to the device.

[0676] Step 16:

[0677] The device provides detailed information to the user, and the server stores all interaction data and updates the model to further optimize the next suggestion.

[0678] Example 1

[0679] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0680] Traditional product guidance in shopping malls relies on face-to-face human resources and is inefficient. It is also difficult to provide personalized suggestions tailored to user needs, making it difficult to quickly and accurately guide users to the products they are looking for. Furthermore, there is a lack of a mechanism for efficiently incorporating user feedback, making it difficult to improve the quality of next suggestions.

[0681] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0682] In this invention, the server includes means for detecting the approach of a user, means for starting a conversation with the user and collecting the user's needs, means for transmitting the collected user needs to the server, means for the server to select optimal products based on the user's needs and generate a proposal list, means for presenting the generated proposal list to the user, means for recollecting the user's reactions and transmitting them to the server to optimize the next proposal, and means for the server to accumulate dialogue data and perform self-learning to optimize the next proposal, thereby enabling efficient and personalized product guidance according to the user's needs.

[0683] "Means for detecting the approach of a user" refers to a device or technology that uses a built-in camera or sensor to detect when a user approaches within a certain range.

[0684] "Means for starting a conversation with the user and gathering user needs" refers to the process of using speech synthesis technology and natural language processing technology to converse with the user and gather information and product requests that the user is looking for.

[0685] The "means for transmitting the collected user needs to the server" refers to a communication means or protocol for transmitting the information collected from the user to the server via a network.

[0686] "Means by which the server selects optimal products based on the user's needs and generates a proposal list" refers to algorithms or technologies by which the server analyzes collected user information, selects relevant products from a database, and generates a list.

[0687] The "means for presenting the generated proposal list to the user" refers to a device such as a display or a speaker that visually or audibly presents the generated product proposal list to the user.

[0688] "Means for collecting user responses again and sending them to the server to optimize future suggestions" refers to the process of collecting the responses that users give to suggestions and sending that information back to the server to improve the quality of future suggestions.

[0689] "Means for the server to accumulate dialogue data and self-learn to optimize the next proposal" refers to a technology in which the server saves past dialogue data with the user, uses that data to apply machine learning technology, and improves the accuracy of the next proposal.

[0690] This invention is an AI sales assistant robot system that guides users visiting shopping malls about products such as smartphones and water dispensers. This system operates mainly through interactions between terminals, servers, and users.

[0691] System configuration

[0692] 1. Terminal (Sales Assistant Robot):

[0693] Approach Detection:

[0694] The device uses a built-in camera (e.g., Logitech HD Pro Webcam C920) and sensors (e.g., HC-SR501 PIR motion sensor) to detect the approach of a user, and uses facial recognition technology (e.g., OpenCV library) to determine whether the user is a known or new user.

[0695] Example: When a user approaches within 2 meters of a robot, the device's sensors detect this and the camera captures the user's face.

[0696] Beginning the initial interaction:

[0697] When the device detects the user's approach, it uses speech synthesis software (e.g., Google Text-to-Speech API) to greet the user, saying, "Hello! Is there anything I can help you with?"

[0698] Example: When the device says, "Hello! Is there anything I can help you with?" the user turns towards the device in surprise.

[0699] Gathering needs:

[0700] When the user responds by saying, "I'm looking for a new smartphone," the device uses voice recognition technology (e.g., Google Cloud Speech-to-Text API) to convert this request into text data and send it to the server.

[0701] Example: When a user says, "I'm looking for a new smartphone," the device converts it into text and sends it to the server as "I'm looking for a new smartphone."

[0702] 2. Server:

[0703] Data reception and analysis:

[0704] The server receives user needs (e.g., "a smartphone with a good camera") sent from the device, searches the database using AI algorithms (e.g., Python and the Scikit-learn library), and generates the optimal product list. This uses a generative AI model such as the XGBoost model.

[0705] Example: The server receives the need "Looking for a new smartphone" and selects a suitable smartphone from the database.

[0706] Generate and send a list of suggestions:

[0707] The server transmits the generated product list to the terminal.

[0708] Example: The server generates a list of "smartphones with good camera performance" and sends it to the device.

[0709] 3. Handling Suggestions and Feedback:

[0710] Presentation of proposal:

[0711] The device advises the user, "This smartphone has excellent camera performance. Its high resolution makes it particularly good for night photography."

[0712] Example: The device suggests to the user, "This smartphone has excellent camera performance."

[0713] User response collection and re-proposal:

[0714] If the user responds with, "That sounds good. Are there any other options?" the device sends that information back to the server and generates additional suggestions.

[0715] Example: When a user asks, "Are there any other options available?", the device sends the request to the server.

[0716] Generate and present re-proposals:

[0717] The server receives the new needs, generates a new appropriate product list, and sends it to the terminal, which then makes new suggestions to the user based on this list.

[0718] Example: The device re-proposes, "This smartphone also has a high-performance camera, but at a more reasonable price."

[0719] 4. Continuous self-learning:

[0720] The server stores all interaction data and continuously trains the AI ​​model to optimize the next suggestion, resulting in more personalized suggestions for the next user.

[0721] Example: The server uses user interaction data to update the model to improve the accuracy of the next suggestion.

[0722] The overall process of this system makes it possible to efficiently gather user needs and make optimal product recommendations in real time. By using the above-mentioned means, shopping mall visitors can receive prompt and appropriate product information, providing a more satisfying shopping experience.

[0723] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0724] System program processing flow

[0725] Step 1:

[0726] User approach detection

[0727] The device uses its built-in camera and sensors to constantly monitor whether a user is approaching. When the user approaches within a certain range, the device uses facial recognition technology (e.g., OpenCV library) to recognize the user. At this time, the device's camera captures video and the sensor receives distance data as input. Based on this data, the device runs a facial recognition algorithm to output facial information.

[0728] Specific operations: camera image capture, sensor data acquisition, face recognition

[0729] Input: Camera image data, sensor data

[0730] Output: Face recognition result (user is new or known)

[0731] Step 2:

[0732] Starting the initial dialogue

[0733] When the device detects the user's proximity, it uses speech synthesis software (e.g., Google Text-to-Speech API) to greet the user with "Hello! Is there anything I can help you with?" This speech is transmitted directly to the user through the speaker.

[0734] Specific operations: speech synthesis, speech output from speaker

[0735] Input: Approach detection result

[0736] Output: Synthesized speech greeting

[0737] Step 3:

[0738] Gathering needs

[0739] When the user responds by saying "I'm looking for a new smartphone," the device uses voice recognition technology (e.g., Google Cloud Speech-to-Text API) to convert this request into text data, which is then sent to the server for further processing.

[0740] Specific operations: speech recognition, text data generation

[0741] Input: User voice input

[0742] Output: Text data (e.g. "Looking for a new smartphone")

[0743] Step 4:

[0744] Sending data

[0745] The terminal sends the converted text data to the server, often using HTTP or WebSocket as the communication protocol.

[0746] Specific operations: sending data, using communication protocols

[0747] Input: Text data

[0748] Output: Send text data to the server

[0749] Step 5:

[0750] Data reception and analysis

[0751] The server receives the user needs data sent from the device and analyzes it using an AI algorithm (e.g., Python and the Scikit-learn library). The server then accesses a database and generates a list of products that best fit the user's needs. This uses a generative AI model such as the XGBoost model.

[0752] Specific operations: receiving data, analyzing it with AI algorithms, and executing database queries

[0753] Input: User needs data

[0754] Output: Optimal product list

[0755] Step 6:

[0756] Generate and send a list of suggestions

[0757] The server generates an optimal product list based on the analysis results and sends it to the terminal. This list includes detailed product information (e.g., product name, specifications, price, etc.).

[0758] Specific operations: Generate product list, send data

[0759] Input: Analysis results of the AI ​​algorithm

[0760] Output: Send product list to terminal

[0761] Step 7:

[0762] Proposal presentation

[0763] Based on the product list received from the server, the device makes suggestions to the user via voice, providing specific explanations such as, "This smartphone has excellent camera performance."

[0764] Specific operations: speech synthesis, speech output from speaker

[0765] Input: Product list

[0766] Output: Spoken suggestions

[0767] Step 8:

[0768] Collecting user responses and re-proposing

[0769] When the user makes additional requests, the device converts their responses into text using voice recognition technology again and sends it to the server.

[0770] Specific operations: speech recognition, text data generation, data transmission

[0771] Input: User's voice response

[0772] Output: Send text data to the server

[0773] Step 9:

[0774] Generate and present re-proposals

[0775] The server receives the new request, generates a new appropriate product list, and sends it to the terminal, which then makes new suggestions to the user based on this list.

[0776] Specific operations: receiving data, analyzing with AI algorithms, generating product list, sending data, and synthesizing voice

[0777] Input: New user request

[0778] Output: Re-proposed product list

[0779] Step 10:

[0780] Data accumulation and learning

[0781] The server stores all interaction data and continuously trains the AI ​​model to optimize the next suggestion, resulting in more personalized suggestions for the next user.

[0782] Specific actions: Data accumulation, AI model training, performance improvement

[0783] Input: User interaction data

[0784] Output: Optimized AI model

[0785] (Application example 1)

[0786] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0787] In today's brick-and-mortar stores, it is difficult to quickly propose products that accurately meet customer needs. Furthermore, there is a lack of systems to effectively utilize the information obtained through dialogue with customers, and the accuracy of proposals and the optimization of future proposals are not sufficiently implemented. As a result, improvements to the customer experience are hindered, leading to a decline in store sales and customer satisfaction.

[0788] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0789] In this invention, the server includes means for detecting the approach of a user, means for initiating a conversation with the user and collecting the user's needs, means for transmitting the collected user needs to the server, means for the server to select optimal information based on the user's needs and generate a proposal list, means for presenting the generated proposal list to the user, means for recollecting the user's responses and transmitting them to the server to optimize the next proposal, means for analyzing the user's needs using an AI algorithm and searching a database, and means for self-learning based on the collected feedback to improve the accuracy of proposals from the next time onwards. This enables product proposals that quickly and accurately meet customer needs, improving the customer experience and increasing store sales.

[0790] "Means for detecting the approach of a user" refers to technology that uses a camera or sensor to detect the presence of a user when they arrive within a pre-set distance.

[0791] "Means for starting a conversation with the user and gathering information about the user's needs" refers to a function that uses voice recognition technology and natural language processing to converse with the user and understand their requests and wishes.

[0792] The "means for transmitting collected user needs to a server" is a communication means for transferring collected data to a central database via the Internet.

[0793] "Means for the server to select the most appropriate information based on the user's needs and generate a recommendation list" refers to a function that uses AI algorithms and data analysis technology to find the products and services that best meet the user's requirements and create a recommendation list.

[0794] The "means for presenting the generated proposal list to the user" is a function that uses a display or voice synthesis technology to present information that is determined to be most suitable to the user in an easy-to-understand manner.

[0795] "Means for recollecting user responses and sending them to a server to optimize the next proposal" refers to a technology that collects the interest and feedback that users have shown in the proposal and sends that data to a server to improve the quality of future proposals.

[0796] "Means of analyzing user needs using AI algorithms and searching a database" refers to a function that analyzes needs data collected using machine learning and deep learning, and searches a database for appropriate information.

[0797] "Means for self-learning based on collected feedback to improve the accuracy of future proposals" refers to a system that uses accumulated user feedback data as learning material to improve the AI ​​model and fine-tune the algorithm, thereby improving the accuracy of proposals.

[0798] This invention is an AI sales assistant system for suggesting products according to customer needs in shopping malls and brick-and-mortar stores, and it mainly operates using a terminal (e.g., a smartphone or robot) and a server. The main function of this system is to detect when a user approaches, collect and analyze the user's needs through dialogue, and generate a suggestion list.

[0799] System configuration

[0800] Device configuration

[0801] 1. User approach detection:

[0802] The device uses its built-in camera and sensors to detect approaching users in real time, and uses facial recognition technology (e.g., TensorFlow) to distinguish between existing and new users.

[0803] 2. Initiating conversations and gathering needs:

[0804] When the user approaches, the device greets the user through voice output (e.g., Google Cloud Text-to-Speech) and initiates a dialogue. It then uses voice recognition technology (e.g., Google Cloud Speech-to-Text) to convert the user's request into text data and transmits that data to the server.

[0805] 3. Present the list of proposals and gather feedback:

[0806] The device that receives the proposal list presents it to the user via voice output or a display (e.g., a smartphone or robot screen). The user's reactions and feedback are collected again and sent to the server.

[0807] Server Configuration

[0808] 1. Data Receipt and Analysis:

[0809] The server receives the user's needs data sent from the device and searches the database using an AI algorithm. Specifically, it analyzes the data using a machine learning model (e.g., a generative AI model) to generate an optimal product list.

[0810] 2. Generate the proposal list:

[0811] Based on the analysis results, a product list that best suits the user's needs is generated and sent to the terminal.

[0812] 3. Feedback accumulation and self-learning:

[0813] The server accumulates user feedback data sent from the device and performs self-learning to improve the accuracy of suggestions from the next time onwards, thereby enabling continuous improvement of the accuracy of suggestions.

[0814] Hardware and software used

[0815] Cameras and sensors: Used to detect the user's approach.

[0816] Microphone and speaker: Used for audio input and output.

[0817] Display: Used to visually display the suggestion list.

[0818] TensorFlow: Used to provide facial recognition technology.

[0819] Google Cloud Speech-to-Text: Used to provide speech recognition technology.

[0820] Google Cloud Text-to-Speech: Used to provide audio output.

[0821] Firebase or AWS DynamoDB: Used to manage data and manage AI models.

[0822] Specific examples

[0823] 1. User A approaches:

[0824] Terminal: "Hello! Is there anything you're looking for?"

[0825] User: "Hi. I'm looking for a new smartphone."

[0826] 2. Make suggestions based on needs:

[0827] Device: "What features of a smartphone are you looking for?"

[0828] User: "I want a smartphone with a good camera."

[0829] The terminal sends this request to the server.

[0830] The server generates a list of optimal smartphones and sends it to the device.

[0831] Device: "The camera on this smartphone is excellent. It's particularly high-resolution and great for night photography."

[0832] 3. The user makes an additional request:

[0833] User: "That sounds good. Are there any other options?"

[0834] The device sends a new request to the server, gets another product list, and makes another suggestion.

[0835] Device: "This phone also has a great camera, but it's more affordable."

[0836] Prompt Sentence Examples

[0837] "Design an AI assistant for a physical store that detects the user's proximity and suggests the best products based on their needs. Write a Python program that recognizes the customer's voice input and generates and presents a list of products."

[0838] As a result, the present invention makes it possible to quickly grasp the needs of customers in physical stores and provide the most suitable products to meet those needs, thereby improving the customer experience and increasing store sales.

[0839] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0840] Step 1:

[0841] The device uses its built-in camera and sensors to detect when a user approaches. Specifically, it analyzes the frames captured by the camera and uses facial recognition technology (e.g., TensorFlow) to identify whether the user is an existing or new user. Here, the input is the camera image frame, and the output is the user approach detection information.

[0842] Step 2:

[0843] When the device detects the user's approach, it uses voice output (e.g., Google Cloud Text-to-Speech) to address the user, saying, "Hello! Is there anything I can help you with?", and begins a dialogue. Here, the input is user detection information, and the output is the start of a conversation with the user.

[0844] Step 3:

[0845] When the user responds, "I'm looking for a new smartphone," the device uses voice recognition technology (e.g., Google Cloud Speech-to-Text) to convert this speech into text data. It then asks, "What features of a smartphone are you looking for?" Here, the input is voice data, and the output is text data.

[0846] Step 4:

[0847] If the user answers, "I want a smartphone with a good camera," the device sends this need data to the server. Here, the input is text data, and the output is the transmission of the need data to the server.

[0848] Step 5:

[0849] The server receives the received needs data and analyzes it using an AI algorithm (e.g., a generative AI model). Specifically, it uses a machine learning model to generate an optimal smartphone list. Here, the input is the needs data and the output is a product list.

[0850] Step 6:

[0851] The server sends the generated product list to the terminal, which then uses voice output and a display to suggest to the user, "This smartphone has excellent camera performance. Its high resolution makes it particularly good for nighttime photography." Here, the input is the product list, and the output is the suggestion to the user.

[0852] Step 7:

[0853] The user responds with "That sounds good. Are there any other options?" and the device sends this feedback back to the server. Here, the input is the user's feedback and the output is sending the feedback to the server.

[0854] Step 8:

[0855] The server analyzes the feedback data and generates an additional proposal list. The server then sends the generated proposal list to the device, which then makes a re-proposal, saying, "This smartphone also has a high-performance camera, but it's more reasonably priced." Here, the input is the feedback data, and the output is the new product list and the re-proposal.

[0856] Step 9:

[0857] The server accumulates all the interaction data and performs self-learning to optimize the next suggestion, thereby improving the accuracy of suggestions for the user's future requests. Here, the input is all the interaction data, and the output is the updated AI model.

[0858] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0859] This invention is an AI sales assistant robot system that guides users visiting shopping malls about products such as smartphones and water dispensers. This system operates primarily through dialogue between terminals, servers, and users, and by combining it with an emotion engine, it recognizes the user's emotions and optimizes the content of its recommendations. Below, we provide a detailed explanation of the operation of each element and provide specific examples.

[0860] System configuration

[0861] 1. Terminal (Sales Assistant Robot):

[0862] Dialogue directly with users to gather their needs.

[0863] Built-in sensors and cameras may be used to detect the user's approach and facial recognition technology may be used to identify the user.

[0864] Using voice recognition technology, the user's questions and requests are converted into text data and sent to the server.

[0865] Equipped with an emotion engine, it recognizes emotions in real time from the user's facial expressions and tone of voice.

[0866] 2. Server:

[0867] The system receives and analyzes the user's needs data and emotion data transmitted from the terminal.

[0868] Using AI algorithms, we generate product lists that best suit the user's needs.

[0869] Based on emotional data, suggestions are optimized to suit the user's emotional state.

[0870] The generated proposal list is sent to the terminal.

[0871] It accumulates user response data and performs self-learning to improve the accuracy of its next suggestions.

[0872] Program processing explanation

[0873] Device behavior

[0874] 1. User approach detection:

[0875] The device uses its built-in camera and sensors to detect when a user approaches, along with facial recognition technology to determine whether the user is a known or new user.

[0876] 2. Initiating the initial dialogue:

[0877] When the device detects a user approaching, it greets them with "Hello! Is there anything you're looking for?"

[0878] 3. Gathering needs:

[0879] The user responds, "I'm looking for a new smartphone." The device then asks, "What features are you looking for in a smartphone?" and offers options such as camera performance and battery life.

[0880] 4. Emotion recognition:

[0881] The user responds, "I want a smartphone with a good camera." During this utterance, the device's emotion engine analyzes the user's emotions in real time from their facial expressions and tone of voice, and sends that data to the server.

[0882] Server Operation

[0883] 1. Data Receipt and Analysis:

[0884] The server receives the user's needs ("a smartphone with a good camera") and emotional data sent from the device, searches the database using an AI algorithm, and generates the optimal product list.

[0885] 2. Generate and send the proposal list:

[0886] The server then sends the generated product list to the terminal, taking into consideration the emotional data and prioritizing expressions and products that are likely to interest the user.

[0887] Handling suggestions and feedback

[0888] 1. Presentation of the proposal:

[0889] The device advises the user, "This smartphone has excellent camera performance. Its high resolution makes it particularly good for night photography."

[0890] 2. Collecting user responses:

[0891] The user asks, "That sounds good. Are there any other options?" The device's emotion engine again analyzes the user's emotions and sends the results to the server.

[0892] 3. Generate and present re-proposals:

[0893] The server searches the database again based on the new need ("more reasonable price") and emotion data, generates a more optimal product list, and sends it to the device. The device then makes a new suggestion to the user, saying, "This smartphone also has a high-performance camera, but at a more reasonable price."

[0894] Specific examples

[0895] 1. The user approaches and is detected:

[0896] Terminal: "Hello! Is there anything you're looking for?"

[0897] User: "Hi. I'm looking for a new smartphone."

[0898] 2. Make suggestions based on needs:

[0899] Device: "What features of a smartphone are you looking for?"

[0900] User: "I want a smartphone with a good camera."

[0901] The terminal transmits this request and emotion data to the server.

[0902] The server generates a list of optimal smartphones and sends it to the device.

[0903] Device: "The camera on this smartphone is excellent. It's particularly high-resolution and great for night photography."

[0904] 3. The user makes an additional request:

[0905] User: "That sounds good. Are there any other options?"

[0906] The device sends a new request and emotion data to the server, retrieves another product list, and makes another suggestion.

[0907] Device: "This phone also has a great camera, but it's more affordable."

[0908] Continuous self-learning

[0909] The server accumulates all conversation and emotion data and continuously updates the model to optimize the next recommendation, thereby providing each user with the best possible recommendation in real time, providing a better purchasing experience.

[0910] In this way, the present invention efficiently delivers personalized offers tailored to each customer's individual needs and emotions, enhancing the customer experience.

[0911] The processing flow will be explained below.

[0912] Step 1:

[0913] The device uses its built-in camera and sensors to detect the user's approach, and may then use facial recognition technology to identify the user.

[0914] Step 2:

[0915] When the device approaches a user, it greets them with "Hello! Is there anything you're looking for?"

[0916] Step 3:

[0917] The user replies, "I'm looking for a new phone."

[0918] Step 4:

[0919] The device converts the user's speech into text data using voice recognition technology and continues to ask questions such as, "What features of a smartphone are you looking for?" At this time, it presents options such as camera performance and battery life.

[0920] Step 5:

[0921] The user responds, "I want a smartphone with a good camera." The device converts this response back into text data and sends it to the server. The emotion engine analyzes the user's facial expressions and tone of voice, and sends the user's emotional data to the server.

[0922] Step 6:

[0923] The server analyzes the user's needs ("I want a smartphone with a good camera") and emotional data received. The server searches the database and uses an AI algorithm to generate an optimal product list. At this time, the server takes into account the emotional data and prioritizes products that are likely to interest the user.

[0924] Step 7:

[0925] The server transmits the generated product list to the terminal.

[0926] Step 8:

[0927] The device will advise the user, "This smartphone has excellent camera performance. Its high resolution makes it particularly good for taking night shots."

[0928] Step 9:

[0929] When a user asks, "That sounds good. Are there any other options?", the emotion engine analyzes the user's facial expressions and tone of voice to obtain emotional data.

[0930] Step 10:

[0931] The device again sends the new need ("more reasonable options") and emotion data to the server.

[0932] Step 11:

[0933] The server searches the database again based on the new needs and emotional data, and uses an AI algorithm to generate an even more optimal product list, taking into account the emotional data and including products that are likely to interest the user.

[0934] Step 12:

[0935] The server transmits the regenerated product list to the terminal.

[0936] Step 13:

[0937] The device will then make a new suggestion to the user, saying, "This smartphone also has a high-performance camera, but it's more reasonably priced."

[0938] Step 14:

[0939] The user responds, "That's great. Please tell me more about the specs." The emotion engine analyzes the user's facial expression and tone of voice at this time to obtain emotional data.

[0940] Step 15:

[0941] The device sends the user's reaction and emotional data to the server, which then generates detailed specification information and sends it to the device using the most appropriate expressions based on the emotional data.

[0942] Step 16:

[0943] The device provides detailed specifications to the user, and sends the user's reaction and emotional data to the server, which then accumulates all the data and updates the model to optimize the next proposal.

[0944] Example 2

[0945] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0946] Conventional customer service systems in shopping malls have had difficulty fully reflecting individual emotions and detailed requests when proposing products that meet users' needs. As a result, they have been unable to make appropriate proposals that stimulate purchasing motivation, and their effectiveness in improving customer satisfaction and promoting sales has been limited. The object of the present invention is to solve these problems by providing a system that recognizes users' emotions in real time and makes appropriate product proposals.

[0947] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0948] In this invention, the server includes means for detecting the approach of a user, means for starting a conversation with the user and collecting the user's needs, means for transmitting the collected user needs and emotion data to the server, means for the server to select optimal products based on the user's needs and emotion data and generate a proposal list, means for presenting the generated proposal list to the user, means for recollecting the user's reactions and transmitting them to the server to optimize the next proposal, and means for grasping the user's emotions in real time using an emotion recognition engine. This makes it possible to flexibly respond to the individual emotions and needs of the user and make optimal product proposals.

[0949] "User" refers to an individual who receives product information from the system.

[0950] "Server" refers to a central management unit that analyzes data sent by users and generates optimal product suggestions.

[0951] "Terminal" refers to a device that directly interacts with users, gathers their needs, and connects to a server.

[0952] "Emotional data" refers to information that indicates the emotional state of a user, as determined by facial expressions and vocal tones.

[0953] "Proposal list" refers to a list of optimal products generated by the server and presented to the user.

[0954] An "emotion recognition engine" refers to software that analyzes a user's facial expressions and tone of voice in real time to determine their emotions.

[0955] "Speech recognition technology" refers to technology that converts a user's speech into text data.

[0956] "Facial recognition technology" refers to technology that uses a camera to identify a user's face and determine whether it is familiar or new.

[0957] MODE FOR CARRYING OUT THE INVENTION

[0958] (System Overview)

[0959] This invention is an AI sales assistant robot system that guides users visiting a shopping mall to the most suitable products. This system operates through dialogue with the user and, by combining it with an emotion recognition engine, recognizes the user's emotions and optimizes the content of the recommendations. Specific embodiments for implementing this system are described below.

[0960] (Device configuration and operation)

[0961] The terminal will be installed in shopping malls as a sales assistant robot and is equipped with a built-in camera, infrared sensors, voice recognition technology, and an emotion recognition engine.

[0962] User approach detection

[0963] The device uses a built-in camera and infrared sensor to detect the user's movement when they approach within a certain distance. Facial recognition technology is used to determine whether the user is an existing or new user. For example, the built-in camera can be a standard webcam, and the infrared sensor can be a commercially available motion sensor. Facial recognition uses technologies such as Face++.

[0964] Starting the initial dialogue

[0965] When a user approaches, the device will speak a pre-programmed initial greeting (e.g., "Hello! Is there anything I can help you with?") using voice synthesis technology, which can use technologies such as Google Text-to-Speech.

[0966] Gathering needs

[0967] If the user responds, "I'm looking for a new smartphone," the device uses voice recognition software (e.g., Google Cloud Speech-to-Text) to convert the utterance into text data, then asks, "What features are you looking for in a smartphone?" and presents options such as "camera performance," "battery life," and "price range."

[0968] emotion recognition

[0969] When a user answers, "I want a smartphone with a good camera," the device's emotion recognition engine (e.g., Affectiva) is used to determine the user's emotion in real time from their facial expressions and tone of voice. This emotion data is then sent to the server along with the user's response.

[0970] (Server configuration and operation)

[0971] The server acts as a central control unit that analyzes the data sent from the terminals and generates product suggestions. The following processes are performed:

[0972] Data reception and analysis

[0973] The server receives the user's needs (e.g., "a smartphone with a good camera") and emotional data sent from the device. It uses an AI algorithm to search the database and generate the optimal product list. The AI ​​algorithm uses a natural language processing model using TensorFlow.

[0974] Generate and send a list of suggestions

[0975] The server then sends the generated product list to the device. It considers the user's emotional data and prioritizes expressions and products that are likely to interest the user. If the user is excited, the latest high-performance camera smartphones are placed at the top of the list.

[0976] (Suggestion and feedback handling)

[0977] The terminal makes suggestions to the user based on the product list sent from the server.

[0978] Proposal presentation

[0979] The device will suggest to the user, "This smartphone has excellent camera performance, especially with high resolution and excellent night photography." By emphasizing specific features (e.g., the night photography function "Night Sight"), it will attract the user's interest.

[0980] Collecting user responses

[0981] The user asks a follow-up question, "That sounds good. Are there any other options?" The device's emotion recognition engine again analyzes the user's facial expressions and tone of voice and sends the results to the server.

[0982] Generate and present re-proposals

[0983] The server searches the database again based on the new need (e.g., "reasonable price") and emotion data, generates a more suitable product list, and sends it to the device. The device then makes a new suggestion, saying, "This smartphone also has a high-performance camera, but at a more reasonable price."

[0984] (Continuous self-learning)

[0985] The server accumulates all conversation and emotion data and continually updates the AI ​​model to improve the accuracy of the next recommendation. For example, if a user frequently requests "reasonable prices," the server will prioritize presenting options with lowered prices to that user.

[0986] Example prompt

[0987] Example prompt: "User is looking for a new smartphone. Suggest a smartphone that focuses on camera performance, and if they inquire about price, suggest a more affordable option."

[0988] In this way, the present invention can flexibly respond to the individual needs and feelings of the user and make optimal product suggestions.

[0989] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0990] Program processing flow

[0991] Step 1: Detecting user approach

[0992] Input: The device receives data from the built-in camera and infrared sensor.

[0993] Data processing: Analyzes camera footage and reads motion sensor signals.

[0994] Output: Detects that the user is approaching within a certain distance.

[0995] Specific operation: The device analyzes camera footage in real time, identifies the user's face using facial recognition technology, and detects the user's approach from the infrared sensor signal to determine whether the user is a new or existing user.

[0996] Step 2: Initiating initial interactions

[0997] Input: Signal that detects the user's approach.

[0998] Data processing: Generation of greetings and speech synthesis.

[0999] Output: The device says "Hello! Is there anything I can help you with?"

[1000] Specific behavior: Uses the device's speech synthesis technology (e.g., Google Text-to-Speech) to audibly transmit a programmed greeting to the user.

[1001] Step 3: Gathering needs

[1002] Input: User utterance (e.g., "I'm looking for a new smartphone").

[1003] Data processing: Convert speech into text data using speech recognition technology.

[1004] Output: The text data is "Looking for a new smartphone."

[1005] What it does: The device uses voice recognition software (e.g., Google Cloud Speech-to-Text) to convert what the user says into text in real time, analyzes that text, and generates the next question (e.g., "What features of a smartphone are you looking for?").

[1006] Step 4: Emotion Recognition

[1007] Input: User's speech and facial expression data.

[1008] Data processing: Analyze user emotions using an emotion recognition engine.

[1009] Output: Sentiment data as the analysis result.

[1010] Specific operation: The device uses an emotion recognition engine (e.g., Affectiva) to determine the user's emotions (joy, anger, sadness, and happiness) from their facial expressions and vocal tones. The obtained emotional data is then sent to the server along with the user's needs.

[1011] Step 5: Server receives and analyzes data

[1012] Input: User needs and emotion data sent from the device.

[1013] Data processing: Using AI algorithms, we generate product lists that best suit the user's needs.

[1014] Output: A list of optimal products.

[1015] How it works: The server analyzes user needs using a natural language processing model powered by TensorFlow, searches the database to select the most suitable products, and adjusts the order of the list taking into account emotional data.

[1016] Step 6: Generate and submit a proposal list

[1017] Input: Product list generated by an AI algorithm.

[1018] Data processing: Adjust product priorities based on user sentiment.

[1019] Output: Sending the product list to the terminal.

[1020] Specific operation: The server sends the generated product list to the terminal. At this time, expressions that are likely to interest the user are given priority in the list.

[1021] Step 7: Proposal Presentation

[1022] Input: Product list sent from the server.

[1023] Data processing: Audio conversion of product list.

[1024] Output: Spoken product recommendations to the user.

[1025] Specific operation: The device will suggest to the user using a synthesized voice, "This smartphone has excellent camera performance. Its high resolution makes it particularly good for night photography."

[1026] Step 8: Collect user responses

[1027] Input: User's follow-up question (e.g., "That sounds good. Are there any other options I can choose?").

[1028] Data processing: The follow-up questions are converted into text data using voice recognition technology. The emotion data is then analyzed again using an emotion recognition engine.

[1029] Output: A new request to the server and sending emotion data.

[1030] Specific operation: The device performs voice recognition of the follow-up question, re-determines the user's current emotional state using the emotion recognition engine, and sends the analysis results to the server.

[1031] Step 9: Generate and present re-proposals

[1032] Input: A new product list from the server.

[1033] Data processing: Audio generation of new product listings.

[1034] Output: A spoken re-suggestion to the user.

[1035] Specific operation: The device makes a re-suggestion by synthesizing voice, saying, "This smartphone also has a high-performance camera, but it's more reasonably priced."

[1036] Prompt Sentence Examples

[1037] Example prompt: "User is looking for a new smartphone. Suggest a smartphone that focuses on camera performance, and if they inquire about price, suggest a more affordable option."

[1038] (Application example 2)

[1039] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1040] Conventional shopping support systems have difficulty accurately gathering user needs and suggesting optimal products. Furthermore, because they do not take the user's emotions into consideration, the content of the suggestions is unlikely to capture the user's interest and fail to stimulate purchasing motivation. Furthermore, there is a lack of personalized suggestions linked to past request data, making it impossible to achieve optimization for each user. There is a need to solve these problems and provide a more effective customer experience.

[1041] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1042] In this invention, the server includes means for detecting the approach of a user, means for initiating a conversation with the user and collecting the user's needs, means for transmitting the collected user needs to the server, means for the server to select optimal products based on the user's needs and generate a proposal list, means for presenting the generated proposal list to the user, means for recollecting the user's reactions and transmitting the data to the server to optimize the next proposal, means for recognizing the user's emotions in real time using emotion recognition technology and transmitting the data to the server, and means for optimizing the proposal content based on the user's emotion data. This makes it possible to make personalized proposals according to the user's needs and improve the customer experience.

[1043] "Means for detecting a user's approach" refers to technology or devices that use cameras or sensors to detect a user's approach to the system.

[1044] "Means for starting a conversation with a user and gathering information about the user's needs" refers to technology or devices that use voice recognition or natural language processing to engage in a dialogue with a user and gather information about the user's requests and wishes.

[1045] The "means for transmitting collected user needs to a server" refers to a communication technology or device for transferring the user needs collected by a terminal to a server as data.

[1046] "Means for the server to select optimal products based on the user's needs and generate a proposal list" refers to technology or software that allows the server to analyze the user's needs, select optimal products based on those needs, and create a list.

[1047] The "means for presenting the generated proposal list to the user" refers to a technique or device for visually or audibly presenting the generated proposal list to the user.

[1048] "Means for re-collecting user responses and sending them to a server to optimize the next proposal" refers to technology or devices for re-collecting user responses to a proposal, sending them to a server, and optimizing the content of the next proposal based on that data.

[1049] "Means for recognizing a user's emotions in real time using emotion recognition technology and transmitting that data to a server" refers to technology or devices that analyze a user's facial expressions and tone of voice to recognize emotions and transmit the results to a server in real time.

[1050] "Means for optimizing proposal content based on user emotion data" refers to technology or devices for optimizing the proposed products or presentation content by taking into account the recognized user emotion data.

[1051] This invention is a smart shopping assistant system that interacts with customers in physical stores and makes product recommendations. This system consists of the following main components:

[1052] Components

[1053] Hardware

[1054] 1. Sales assistant robot: A device that directly interacts with users and gathers their needs. It may use built-in cameras, microphones, and sensors to detect the user's approach and identify them using facial recognition technology.

[1055] 2. Built-in camera and microphone: Captures the user's facial expressions and voice, providing data for emotion recognition technology.

[1056] 3. Approach detection sensor: Sensor technology that detects the approach of a user.

[1057] software

[1058] 1. Emotion recognition engine: Technology that analyzes the user's facial expressions and vocal tone in real time to recognize their emotional state. Utilizing technologies such as "Affectiva" and "Microsoft Azure Emotion API."

[1059] 2. Speech recognition system: A technology that converts user speech into text data. Services such as "Google Cloud Speech-to-Text" and "Amazon Transcribe" are used.

[1060] 3. Database management systems: Used to store data on user needs and emotions and make optimal product recommendations. Examples include "MySQL" and "MongoDB."

[1061] 4. Natural language processing engine: A technology used to analyze collected data and make product suggestions based on user needs. It uses technologies such as "GPT-4 by OpenAI" and "BERT."

[1062] Operation explanation

[1063] Device behavior

[1064] 1. User approach detection: The sales assistant robot's built-in camera and sensors detect the user's approach and identify the user using facial recognition technology.

[1065] 2. Start of initial interaction: When the robot detects the user's approach, it greets them with "Hello! Is there anything I can help you with?"

[1066] 3. Gathering needs: If the user responds, "I'm looking for a new smartphone," the robot will ask for more specific needs, such as, "Are you looking for a smartphone with a good camera?"

[1067] 4. Emotion recognition: During this speech, the emotion recognition engine processes the user's facial expressions and tone of voice in real time and sends the emotional data to the server.

[1068] Server Operation

[1069] 1. Data reception and analysis: The server receives the user's needs and emotion data sent from the device and analyzes the data using a natural language processing engine.

[1070] 2. Generate and send a list of proposals: Search the database, generate an optimal product list based on the user's needs and emotional data, and send it to the terminal.

[1071] Handling suggestions and feedback

[1072] 1. Proposal presentation: The sales assistant robot makes a proposal to the user, saying, "This smartphone has excellent camera performance. It has high resolution and is particularly good at taking night shots."

[1073] 2. Collecting user responses: If the user asks, "That sounds good. Are there any other options?", the robot sends a new request and emotional data to the server.

[1074] 3. Generation and presentation of a new recommendation: The server generates a more optimal product list based on the new needs and emotional data and sends it to the device. The robot makes a new recommendation, saying, "This smartphone also has a high-performance camera, but it's more reasonably priced."

[1075] Examples and prompts

[1076] Examples:

[1077] The user approaches and the approach is detected.

[1078] Robot: "Hello! Is there anything you're looking for?"

[1079] User: "Hi. I'm looking for a new smartphone."

[1080] Example prompt sentence:

[1081] The user approaches and the approach is detected.

[1082] Robot: "Hello! Is there anything you're looking for?"

[1083] User: "Hi. I'm looking for a new smartphone."

[1084] Robot: "What features are you looking for in a smartphone?"

[1085] User: "I want a smartphone with a good camera."

[1086] The robot sends this request and emotional data to the server.

[1087] The server generates a list of optimal smartphones and sends it to the robot.

[1088] Robot: "The camera on this smartphone is very good. It's especially good at taking high-resolution photos at night."

[1089] User: "That sounds good. Are there any other options?"

[1090] The robot sends a new request and emotion data to the server and gets the suggestion list again.

[1091] Robot: "This smartphone also has a high-performance camera, but at a reasonable price."

[1092] In this way, the smart shopping assistant system can highly personalize the customer experience in physical stores and efficiently make optimal product recommendations.

[1093] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1094] Step 1:

[1095] User approach detection

[1096] Input: Real-time video data from cameras and sensors.

[1097] How it works: The sales assistant robot (terminal) monitors its surroundings using a built-in camera and proximity detection sensors, and detects when a user approaches. It uses facial recognition technology to determine whether the user is a known or new user.

[1098] Output: Information on the detection of the user's approach and the user ID identified as the face recognition result.

[1099] Step 2:

[1100] Starting the initial dialogue

[1101] Input: Information on the user's approach detection and user ID.

[1102] How it works: The device greets the user with a voice message saying, "Hello! Is there anything I can help you with?" and begins the conversation.

[1103] Output: A spoken response from the user.

[1104] Step 3:

[1105] Gathering needs

[1106] Input: A spoken response from the user.

[1107] How it works: The device uses a voice recognition system (Google Cloud Speech-to-Text) to convert the user's voice into text data, then asks, "What kind of smartphone are you looking for?"

[1108] Output: User needs information as converted text data.

[1109] Step 4:

[1110] emotion recognition

[1111] Input: User's voice response and video data.

[1112] How it works: The device uses an emotion recognition engine (Affectiva or Microsoft Azure Emotion API) to analyze the user's facial expressions and voice tone in real time and generate emotion data.

[1113] Output: User emotion data.

[1114] Step 5:

[1115] Sending data

[1116] Input: User needs information and emotion data.

[1117] Operation: The terminal sends the collected data to the server.

[1118] Output: User needs information and emotion data sent to the server.

[1119] Step 6:

[1120] Generate a list of suggestions

[1121] Input: User needs information and emotion data sent to the server.

[1122] How it works: The server analyzes the data using a natural language processing engine (GPT-4 by OpenAI, BERT, etc.), searches for relevant product information in a database management system (MySQL or MongoDB), generates an optimal product list, and prioritizes it taking into account sentiment data.

[1123] Output: The generated optimal product list.

[1124] Step 7:

[1125] Presenting the proposal

[1126] Input: Generated optimal product list.

[1127] What it does: The device recommends the product to the user, saying, "This smartphone has excellent camera performance. Its high resolution makes it particularly good for nighttime photography."

[1128] Output: Product information presented to the user.

[1129] Step 8:

[1130] Collecting user responses

[1131] Input: User response (voice, facial expression).

[1132] How it works: The device uses its voice recognition system and emotion recognition engine to again collect the user's additional requests and reactions and sends this data to the server.

[1133] Output: Newly collected user needs information and emotion data.

[1134] Step 9:

[1135] Generate and present re-proposals

[1136] Input: Newly collected user needs information and emotion data.

[1137] How it works: The server again analyzes the data using a natural language processing engine, searches the database, and generates a new list of suggestions. The list is then sent to the device, which then makes a new suggestion, saying, "This smartphone also has a high-performance camera, but it's more reasonably priced."

[1138] Output: Product information re-presented to the user.

[1139] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1140] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1141] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1142] [Third embodiment]

[1143] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1144] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1145] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1146] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1147] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1148] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1149] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1150] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1151] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1152] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1153] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1154] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1155] This invention is an AI sales assistant robot system that guides users visiting shopping malls about products such as smartphones and water dispensers. This system operates mainly through interactions with terminals, servers, and users. Below, we provide a detailed explanation of the operation of each element and provide specific examples.

[1156] System configuration

[1157] 1. Terminal (Sales Assistant Robot):

[1158] Dialogue directly with users to gather their needs.

[1159] Built-in sensors and cameras may be used to detect the user's approach and facial recognition technology may be used to identify the user.

[1160] Using voice recognition technology, the user's questions and requests are converted into text data and sent to the server.

[1161] 2. Server:

[1162] The user's needs data transmitted from the terminal is received and analyzed.

[1163] Using AI algorithms, we generate product lists that best suit the user's needs.

[1164] The generated proposal list is sent to the terminal.

[1165] It accumulates user response data and performs self-learning to improve the accuracy of its next suggestions.

[1166] Program processing explanation

[1167] Device behavior

[1168] 1. User approach detection:

[1169] The device uses its built-in camera and sensors to detect when a user approaches, along with facial recognition technology to determine whether the user is a known or new user.

[1170] 2. Initiating the initial dialogue:

[1171] When the device detects a user approaching, it greets them with "Hello! Is there anything you're looking for?"

[1172] 3. Gathering needs:

[1173] If the user answers, "I'm looking for a new smartphone," the device will then ask, "What features are you looking for in a smartphone?" If the user answers, "I want a smartphone with a good camera," the device will send this information to the server.

[1174] Server Operation

[1175] 1. Data Receipt and Analysis:

[1176] The server receives the user's needs ("a smartphone with a good camera") sent from the device, searches the database using an AI algorithm, and generates the optimal product list.

[1177] 2. Generate and send the proposal list:

[1178] The server transmits the generated product list to the terminal.

[1179] Handling suggestions and feedback

[1180] 1. Presentation of the proposal:

[1181] The device advises the user, "This smartphone has excellent camera performance. Its high resolution makes it particularly good for night photography."

[1182] 2. Collecting user responses:

[1183] If the user responds with, "That sounds good. Are there any other options?" the device sends that information back to the server and generates additional suggestions.

[1184] 3. Generate and present re-proposals:

[1185] The server searches the database again based on the new needs (price is also important), generates an optimal product list, and sends it to the device. The device then re-suggests to the user, "This smartphone also has a high-performance camera, but it's more reasonably priced."

[1186] Specific examples

[1187] 1. The user approaches and is detected:

[1188] Terminal: "Hello! Is there anything you're looking for?"

[1189] User: "Hi. I'm looking for a new smartphone."

[1190] 2. Make suggestions based on needs:

[1191] Device: "What features of a smartphone are you looking for?"

[1192] User: "I want a smartphone with a good camera."

[1193] The terminal sends this request to the server.

[1194] The server generates a list of optimal smartphones and sends it to the device.

[1195] Device: "The camera on this smartphone is excellent. It's particularly high-resolution and great for night photography."

[1196] 3. The user makes an additional request:

[1197] User: "That sounds good. Are there any other options?"

[1198] The device sends a new request to the server, gets another product list, and makes another suggestion.

[1199] Device: "This phone also has a great camera, but it's more affordable."

[1200] Continuous self-learning

[1201] The server accumulates all interaction data and continuously updates the model to optimize the next suggestion, thus providing each user with the best possible suggestions in real time.

[1202] In this way, the present invention efficiently delivers personalized offers tailored to each customer's individual needs, enhancing the customer experience.

[1203] The processing flow will be explained below.

[1204] Step 1:

[1205] The device uses its built-in camera and sensors to detect the user's approach, and may then use facial recognition technology to identify the user.

[1206] Step 2:

[1207] When the device approaches a user, it greets them with "Hello! Is there anything you're looking for?"

[1208] Step 3:

[1209] The user replies, "I'm looking for a new phone."

[1210] Step 4:

[1211] The device converts the user's speech into text data using voice recognition technology and continues to ask questions such as, "What features of a smartphone are you looking for?" At this time, it presents options such as camera performance and battery life.

[1212] Step 5:

[1213] The user responds, "I want a smartphone with a good camera." The device converts this response back into text data and sends it to the server.

[1214] Step 6:

[1215] The server analyzes the user's needs ("I want a smartphone with a good camera") and searches the database to generate the optimal product list.

[1216] Step 7:

[1217] The server transmits the generated product list to the terminal.

[1218] Step 8:

[1219] The device will advise the user, "This smartphone has excellent camera performance. Its high resolution makes it particularly good for taking night shots."

[1220] Step 9:

[1221] The user asks, "That sounds good. What other options do I have?"

[1222] Step 10:

[1223] The device converts the new need ("more reasonable options") into text data and sends it to the server.

[1224] Step 11:

[1225] The server searches the database again based on the new needs and generates a further product list.

[1226] Step 12:

[1227] The server transmits the regenerated product list to the terminal.

[1228] Step 13:

[1229] The device will then make a new suggestion to the user, saying, "This smartphone also has a high-performance camera, but it's more reasonably priced."

[1230] Step 14:

[1231] The user responds, "That's great. Can you give me more detailed specs?"

[1232] Step 15:

[1233] The device sends the user's response to the server, and the server generates detailed specification information and sends it to the device.

[1234] Step 16:

[1235] The device provides detailed information to the user, and the server stores all interaction data and updates the model to further optimize the next suggestion.

[1236] Example 1

[1237] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1238] Traditional product guidance in shopping malls relies on face-to-face human resources and is inefficient. It is also difficult to provide personalized suggestions tailored to user needs, making it difficult to quickly and accurately guide users to the products they are looking for. Furthermore, there is a lack of a mechanism for efficiently incorporating user feedback, making it difficult to improve the quality of next suggestions.

[1239] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1240] In this invention, the server includes means for detecting the approach of a user, means for starting a conversation with the user and collecting the user's needs, means for transmitting the collected user needs to the server, means for the server to select optimal products based on the user's needs and generate a proposal list, means for presenting the generated proposal list to the user, means for recollecting the user's reactions and transmitting them to the server to optimize the next proposal, and means for the server to accumulate dialogue data and perform self-learning to optimize the next proposal, thereby enabling efficient and personalized product guidance according to the user's needs.

[1241] "Means for detecting the approach of a user" refers to a device or technology that uses a built-in camera or sensor to detect when a user approaches within a certain range.

[1242] "Means for starting a conversation with the user and gathering user needs" refers to the process of using speech synthesis technology and natural language processing technology to converse with the user and gather information and product requests that the user is looking for.

[1243] The "means for transmitting the collected user needs to the server" refers to a communication means or protocol for transmitting the information collected from the user to the server via a network.

[1244] "Means by which the server selects optimal products based on the user's needs and generates a proposal list" refers to algorithms or technologies by which the server analyzes collected user information, selects relevant products from a database, and generates a list.

[1245] The "means for presenting the generated proposal list to the user" refers to a device such as a display or a speaker that visually or audibly presents the generated product proposal list to the user.

[1246] "Means for collecting user responses again and sending them to the server to optimize future suggestions" refers to the process of collecting the responses that users give to suggestions and sending that information back to the server to improve the quality of future suggestions.

[1247] "Means for the server to accumulate dialogue data and self-learn to optimize the next proposal" refers to a technology in which the server saves past dialogue data with the user, uses that data to apply machine learning technology, and improves the accuracy of the next proposal.

[1248] This invention is an AI sales assistant robot system that guides users visiting shopping malls about products such as smartphones and water dispensers. This system operates mainly through interactions between terminals, servers, and users.

[1249] System configuration

[1250] 1. Terminal (Sales Assistant Robot):

[1251] Approach Detection:

[1252] The device uses a built-in camera (e.g., Logitech HD Pro Webcam C920) and sensors (e.g., HC-SR501 PIR motion sensor) to detect the approach of a user, and uses facial recognition technology (e.g., OpenCV library) to determine whether the user is a known or new user.

[1253] Example: When a user approaches within 2 meters of a robot, the device's sensors detect this and the camera captures the user's face.

[1254] Beginning the initial interaction:

[1255] When the device detects the user's approach, it uses speech synthesis software (e.g., Google Text-to-Speech API) to greet the user, saying, "Hello! Is there anything I can help you with?"

[1256] Example: When the device says, "Hello! Is there anything I can help you with?" the user turns towards the device in surprise.

[1257] Gathering needs:

[1258] When the user responds by saying, "I'm looking for a new smartphone," the device uses voice recognition technology (e.g., Google Cloud Speech-to-Text API) to convert this request into text data and send it to the server.

[1259] Example: When a user says, "I'm looking for a new smartphone," the device converts it into text and sends it to the server as "I'm looking for a new smartphone."

[1260] 2. Server:

[1261] Data reception and analysis:

[1262] The server receives user needs (e.g., "a smartphone with a good camera") sent from the device, searches the database using AI algorithms (e.g., Python and the Scikit-learn library), and generates the optimal product list. This uses a generative AI model such as the XGBoost model.

[1263] Example: The server receives the need "Looking for a new smartphone" and selects a suitable smartphone from the database.

[1264] Generate and send a list of suggestions:

[1265] The server transmits the generated product list to the terminal.

[1266] Example: The server generates a list of "smartphones with good camera performance" and sends it to the device.

[1267] 3. Handling Suggestions and Feedback:

[1268] Presentation of proposal:

[1269] The device advises the user, "This smartphone has excellent camera performance. Its high resolution makes it particularly good for night photography."

[1270] Example: The device suggests to the user, "This smartphone has excellent camera performance."

[1271] User response collection and re-proposal:

[1272] If the user responds with, "That sounds good. Are there any other options?" the device sends that information back to the server and generates additional suggestions.

[1273] Example: When a user asks, "Are there any other options available?", the device sends the request to the server.

[1274] Generate and present re-proposals:

[1275] The server receives the new needs, generates a new appropriate product list, and sends it to the terminal, which then makes new suggestions to the user based on this list.

[1276] Example: The device re-proposes, "This smartphone also has a high-performance camera, but at a more reasonable price."

[1277] 4. Continuous self-learning:

[1278] The server stores all interaction data and continuously trains the AI ​​model to optimize the next suggestion, resulting in more personalized suggestions for the next user.

[1279] Example: The server uses user interaction data to update the model to improve the accuracy of the next suggestion.

[1280] The overall process of this system makes it possible to efficiently gather user needs and make optimal product recommendations in real time. By using the above-mentioned means, shopping mall visitors can receive prompt and appropriate product information, providing a more satisfying shopping experience.

[1281] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1282] System program processing flow

[1283] Step 1:

[1284] User approach detection

[1285] The device uses its built-in camera and sensors to constantly monitor whether a user is approaching. When the user approaches within a certain range, the device uses facial recognition technology (e.g., OpenCV library) to recognize the user. At this time, the device's camera captures video and the sensor receives distance data as input. Based on this data, the device runs a facial recognition algorithm to output facial information.

[1286] Specific operations: camera image capture, sensor data acquisition, face recognition

[1287] Input: Camera image data, sensor data

[1288] Output: Face recognition result (user is new or known)

[1289] Step 2:

[1290] Starting the initial dialogue

[1291] When the device detects the user's proximity, it uses speech synthesis software (e.g., Google Text-to-Speech API) to greet the user with "Hello! Is there anything I can help you with?" This speech is transmitted directly to the user through the speaker.

[1292] Specific operations: speech synthesis, speech output from speaker

[1293] Input: Approach detection result

[1294] Output: Synthesized speech greeting

[1295] Step 3:

[1296] Gathering needs

[1297] When the user responds by saying "I'm looking for a new smartphone," the device uses voice recognition technology (e.g., Google Cloud Speech-to-Text API) to convert this request into text data, which is then sent to the server for further processing.

[1298] Specific operations: speech recognition, text data generation

[1299] Input: User voice input

[1300] Output: Text data (e.g. "Looking for a new smartphone")

[1301] Step 4:

[1302] Sending data

[1303] The terminal sends the converted text data to the server, often using HTTP or WebSocket as the communication protocol.

[1304] Specific operations: sending data, using communication protocols

[1305] Input: Text data

[1306] Output: Send text data to the server

[1307] Step 5:

[1308] Data reception and analysis

[1309] The server receives the user needs data sent from the device and analyzes it using an AI algorithm (e.g., Python and the Scikit-learn library). The server then accesses a database and generates a list of products that best fit the user's needs. This uses a generative AI model such as the XGBoost model.

[1310] Specific operations: receiving data, analyzing it with AI algorithms, and executing database queries

[1311] Input: User needs data

[1312] Output: Optimal product list

[1313] Step 6:

[1314] Generate and send a list of suggestions

[1315] The server generates an optimal product list based on the analysis results and sends it to the terminal. This list includes detailed product information (e.g., product name, specifications, price, etc.).

[1316] Specific operations: Generate product list, send data

[1317] Input: Analysis results of the AI ​​algorithm

[1318] Output: Send product list to terminal

[1319] Step 7:

[1320] Proposal presentation

[1321] Based on the product list received from the server, the device makes suggestions to the user via voice, providing specific explanations such as, "This smartphone has excellent camera performance."

[1322] Specific operations: speech synthesis, speech output from speaker

[1323] Input: Product list

[1324] Output: Spoken suggestions

[1325] Step 8:

[1326] Collecting user responses and re-proposing

[1327] When the user makes additional requests, the device converts their responses into text using voice recognition technology again and sends it to the server.

[1328] Specific operations: speech recognition, text data generation, data transmission

[1329] Input: User's voice response

[1330] Output: Send text data to the server

[1331] Step 9:

[1332] Generate and present re-proposals

[1333] The server receives the new request, generates a new appropriate product list, and sends it to the terminal, which then makes new suggestions to the user based on this list.

[1334] Specific operations: receiving data, analyzing with AI algorithms, generating product list, sending data, and synthesizing voice

[1335] Input: New user request

[1336] Output: Re-proposed product list

[1337] Step 10:

[1338] Data accumulation and learning

[1339] The server stores all interaction data and continuously trains the AI ​​model to optimize the next suggestion, resulting in more personalized suggestions for the next user.

[1340] Specific actions: Data accumulation, AI model training, performance improvement

[1341] Input: User interaction data

[1342] Output: Optimized AI model

[1343] (Application example 1)

[1344] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1345] In today's brick-and-mortar stores, it is difficult to quickly propose products that accurately meet customer needs. Furthermore, there is a lack of systems to effectively utilize the information obtained through dialogue with customers, and the accuracy of proposals and the optimization of future proposals are not sufficiently implemented. As a result, improvements to the customer experience are hindered, leading to a decline in store sales and customer satisfaction.

[1346] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1347] In this invention, the server includes means for detecting the approach of a user, means for initiating a conversation with the user and collecting the user's needs, means for transmitting the collected user needs to the server, means for the server to select optimal information based on the user's needs and generate a proposal list, means for presenting the generated proposal list to the user, means for recollecting the user's responses and transmitting them to the server to optimize the next proposal, means for analyzing the user's needs using an AI algorithm and searching a database, and means for self-learning based on the collected feedback to improve the accuracy of proposals from the next time onwards. This enables product proposals that quickly and accurately meet customer needs, improving the customer experience and increasing store sales.

[1348] "Means for detecting the approach of a user" refers to technology that uses a camera or sensor to detect the presence of a user when they arrive within a pre-set distance.

[1349] "Means for starting a conversation with the user and gathering information about the user's needs" refers to a function that uses voice recognition technology and natural language processing to converse with the user and understand their requests and wishes.

[1350] The "means for transmitting collected user needs to a server" is a communication means for transferring collected data to a central database via the Internet.

[1351] "Means for the server to select the most appropriate information based on the user's needs and generate a recommendation list" refers to a function that uses AI algorithms and data analysis technology to find the products and services that best meet the user's requirements and create a recommendation list.

[1352] The "means for presenting the generated proposal list to the user" is a function that uses a display or voice synthesis technology to present information that is determined to be most suitable to the user in an easy-to-understand manner.

[1353] "Means for recollecting user responses and sending them to a server to optimize the next proposal" refers to a technology that collects the interest and feedback that users have shown in the proposal and sends that data to a server to improve the quality of future proposals.

[1354] "Means of analyzing user needs using AI algorithms and searching a database" refers to a function that analyzes needs data collected using machine learning and deep learning, and searches a database for appropriate information.

[1355] "Means for self-learning based on collected feedback to improve the accuracy of future proposals" refers to a system that uses accumulated user feedback data as learning material to improve the AI ​​model and fine-tune the algorithm, thereby improving the accuracy of proposals.

[1356] This invention is an AI sales assistant system for suggesting products according to customer needs in shopping malls and brick-and-mortar stores, and it mainly operates using a terminal (e.g., a smartphone or robot) and a server. The main function of this system is to detect when a user approaches, collect and analyze the user's needs through dialogue, and generate a suggestion list.

[1357] System configuration

[1358] Device configuration

[1359] 1. User approach detection:

[1360] The device uses its built-in camera and sensors to detect approaching users in real time, and uses facial recognition technology (e.g., TensorFlow) to distinguish between existing and new users.

[1361] 2. Initiating conversations and gathering needs:

[1362] When the user approaches, the device greets the user through voice output (e.g., Google Cloud Text-to-Speech) and initiates a dialogue. It then uses voice recognition technology (e.g., Google Cloud Speech-to-Text) to convert the user's request into text data and transmits that data to the server.

[1363] 3. Present the list of proposals and gather feedback:

[1364] The device that receives the proposal list presents it to the user via voice output or a display (e.g., a smartphone or robot screen). The user's reactions and feedback are collected again and sent to the server.

[1365] Server Configuration

[1366] 1. Data Receipt and Analysis:

[1367] The server receives the user's needs data sent from the device and searches the database using an AI algorithm. Specifically, it analyzes the data using a machine learning model (e.g., a generative AI model) to generate an optimal product list.

[1368] 2. Generate the proposal list:

[1369] Based on the analysis results, a product list that best suits the user's needs is generated and sent to the terminal.

[1370] 3. Feedback accumulation and self-learning:

[1371] The server accumulates user feedback data sent from the device and performs self-learning to improve the accuracy of suggestions from the next time onwards, thereby enabling continuous improvement of the accuracy of suggestions.

[1372] Hardware and software used

[1373] Cameras and sensors: Used to detect the user's approach.

[1374] Microphone and speaker: Used for audio input and output.

[1375] Display: Used to visually display the suggestion list.

[1376] TensorFlow: Used to provide facial recognition technology.

[1377] Google Cloud Speech-to-Text: Used to provide speech recognition technology.

[1378] Google Cloud Text-to-Speech: Used to provide audio output.

[1379] Firebase or AWS DynamoDB: Used to manage data and manage AI models.

[1380] Specific examples

[1381] 1. User A approaches:

[1382] Terminal: "Hello! Is there anything you're looking for?"

[1383] User: "Hi. I'm looking for a new smartphone."

[1384] 2. Make suggestions based on needs:

[1385] Device: "What features of a smartphone are you looking for?"

[1386] User: "I want a smartphone with a good camera."

[1387] The terminal sends this request to the server.

[1388] The server generates a list of optimal smartphones and sends it to the device.

[1389] Device: "The camera on this smartphone is excellent. It's particularly high-resolution and great for night photography."

[1390] 3. The user makes an additional request:

[1391] User: "That sounds good. Are there any other options?"

[1392] The device sends a new request to the server, gets another product list, and makes another suggestion.

[1393] Device: "This phone also has a great camera, but it's more affordable."

[1394] Prompt Sentence Examples

[1395] "Design an AI assistant for a physical store that detects the user's proximity and suggests the best products based on their needs. Write a Python program that recognizes the customer's voice input and generates and presents a list of products."

[1396] As a result, the present invention makes it possible to quickly grasp the needs of customers in physical stores and provide the most suitable products to meet those needs, thereby improving the customer experience and increasing store sales.

[1397] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1398] Step 1:

[1399] The device uses its built-in camera and sensors to detect when a user approaches. Specifically, it analyzes the frames captured by the camera and uses facial recognition technology (e.g., TensorFlow) to identify whether the user is an existing or new user. Here, the input is the camera image frame, and the output is the user approach detection information.

[1400] Step 2:

[1401] When the device detects the user's approach, it uses voice output (e.g., Google Cloud Text-to-Speech) to address the user, saying, "Hello! Is there anything I can help you with?", and begins a dialogue. Here, the input is user detection information, and the output is the start of a conversation with the user.

[1402] Step 3:

[1403] When the user responds, "I'm looking for a new smartphone," the device uses voice recognition technology (e.g., Google Cloud Speech-to-Text) to convert this speech into text data. It then asks, "What features of a smartphone are you looking for?" Here, the input is voice data, and the output is text data.

[1404] Step 4:

[1405] If the user answers, "I want a smartphone with a good camera," the device sends this need data to the server. Here, the input is text data, and the output is the transmission of the need data to the server.

[1406] Step 5:

[1407] The server receives the received needs data and analyzes it using an AI algorithm (e.g., a generative AI model). Specifically, it uses a machine learning model to generate an optimal smartphone list. Here, the input is the needs data and the output is a product list.

[1408] Step 6:

[1409] The server sends the generated product list to the terminal, which then uses voice output and a display to suggest to the user, "This smartphone has excellent camera performance. Its high resolution makes it particularly good for nighttime photography." Here, the input is the product list, and the output is the suggestion to the user.

[1410] Step 7:

[1411] The user responds with "That sounds good. Are there any other options?" and the device sends this feedback back to the server. Here, the input is the user's feedback and the output is sending the feedback to the server.

[1412] Step 8:

[1413] The server analyzes the feedback data and generates an additional proposal list. The server then sends the generated proposal list to the device, which then makes a re-proposal, saying, "This smartphone also has a high-performance camera, but it's more reasonably priced." Here, the input is the feedback data, and the output is the new product list and the re-proposal.

[1414] Step 9:

[1415] The server accumulates all the interaction data and performs self-learning to optimize the next suggestion, thereby improving the accuracy of suggestions for the user's future requests. Here, the input is all the interaction data, and the output is the updated AI model.

[1416] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1417] This invention is an AI sales assistant robot system that guides users visiting shopping malls about products such as smartphones and water dispensers. This system operates primarily through dialogue between terminals, servers, and users, and by combining it with an emotion engine, it recognizes the user's emotions and optimizes the content of its recommendations. Below, we provide a detailed explanation of the operation of each element and provide specific examples.

[1418] System configuration

[1419] 1. Terminal (Sales Assistant Robot):

[1420] Dialogue directly with users to gather their needs.

[1421] Built-in sensors and cameras may be used to detect the user's approach and facial recognition technology may be used to identify the user.

[1422] Using voice recognition technology, the user's questions and requests are converted into text data and sent to the server.

[1423] Equipped with an emotion engine, it recognizes emotions in real time from the user's facial expressions and tone of voice.

[1424] 2. Server:

[1425] The system receives and analyzes the user's needs data and emotion data transmitted from the terminal.

[1426] Using AI algorithms, we generate product lists that best suit the user's needs.

[1427] Based on emotional data, suggestions are optimized to suit the user's emotional state.

[1428] The generated proposal list is sent to the terminal.

[1429] It accumulates user response data and performs self-learning to improve the accuracy of its next suggestions.

[1430] Program processing explanation

[1431] Device behavior

[1432] 1. User approach detection:

[1433] The device uses its built-in camera and sensors to detect when a user approaches, along with facial recognition technology to determine whether the user is a known or new user.

[1434] 2. Initiating the initial dialogue:

[1435] When the device detects a user approaching, it greets them with "Hello! Is there anything you're looking for?"

[1436] 3. Gathering needs:

[1437] The user responds, "I'm looking for a new smartphone." The device then asks, "What features are you looking for in a smartphone?" and offers options such as camera performance and battery life.

[1438] 4. Emotion recognition:

[1439] The user responds, "I want a smartphone with a good camera." During this utterance, the device's emotion engine analyzes the user's emotions in real time from their facial expressions and tone of voice, and sends that data to the server.

[1440] Server Operation

[1441] 1. Data Receipt and Analysis:

[1442] The server receives the user's needs ("a smartphone with a good camera") and emotional data sent from the device, searches the database using an AI algorithm, and generates the optimal product list.

[1443] 2. Generate and send the proposal list:

[1444] The server then sends the generated product list to the terminal, taking into consideration the emotional data and prioritizing expressions and products that are likely to interest the user.

[1445] Handling suggestions and feedback

[1446] 1. Presentation of the proposal:

[1447] The device advises the user, "This smartphone has excellent camera performance. Its high resolution makes it particularly good for night photography."

[1448] 2. Collecting user responses:

[1449] The user asks, "That sounds good. Are there any other options?" The device's emotion engine again analyzes the user's emotions and sends the results to the server.

[1450] 3. Generate and present re-proposals:

[1451] The server searches the database again based on the new need ("more reasonable price") and emotion data, generates a more optimal product list, and sends it to the device. The device then makes a new suggestion to the user, saying, "This smartphone also has a high-performance camera, but at a more reasonable price."

[1452] Specific examples

[1453] 1. The user approaches and is detected:

[1454] Terminal: "Hello! Is there anything you're looking for?"

[1455] User: "Hi. I'm looking for a new smartphone."

[1456] 2. Make suggestions based on needs:

[1457] Device: "What features of a smartphone are you looking for?"

[1458] User: "I want a smartphone with a good camera."

[1459] The terminal transmits this request and emotion data to the server.

[1460] The server generates a list of optimal smartphones and sends it to the device.

[1461] Device: "The camera on this smartphone is excellent. It's particularly high-resolution and great for night photography."

[1462] 3. The user makes an additional request:

[1463] User: "That sounds good. Are there any other options?"

[1464] The device sends a new request and emotion data to the server, retrieves another product list, and makes another suggestion.

[1465] Device: "This phone also has a great camera, but it's more affordable."

[1466] Continuous self-learning

[1467] The server accumulates all conversation and emotion data and continuously updates the model to optimize the next recommendation, thereby providing each user with the best possible recommendation in real time, providing a better purchasing experience.

[1468] In this way, the present invention efficiently delivers personalized offers tailored to each customer's individual needs and emotions, enhancing the customer experience.

[1469] The processing flow will be explained below.

[1470] Step 1:

[1471] The device uses its built-in camera and sensors to detect the user's approach, and may then use facial recognition technology to identify the user.

[1472] Step 2:

[1473] When the device approaches a user, it greets them with "Hello! Is there anything you're looking for?"

[1474] Step 3:

[1475] The user replies, "I'm looking for a new phone."

[1476] Step 4:

[1477] The device converts the user's speech into text data using voice recognition technology and continues to ask questions such as, "What features of a smartphone are you looking for?" At this time, it presents options such as camera performance and battery life.

[1478] Step 5:

[1479] The user responds, "I want a smartphone with a good camera." The device converts this response back into text data and sends it to the server. The emotion engine analyzes the user's facial expressions and tone of voice, and sends the user's emotional data to the server.

[1480] Step 6:

[1481] The server analyzes the user's needs ("I want a smartphone with a good camera") and emotional data received. The server searches the database and uses an AI algorithm to generate an optimal product list. At this time, the server takes into account the emotional data and prioritizes products that are likely to interest the user.

[1482] Step 7:

[1483] The server transmits the generated product list to the terminal.

[1484] Step 8:

[1485] The device will advise the user, "This smartphone has excellent camera performance. Its high resolution makes it particularly good for taking night shots."

[1486] Step 9:

[1487] When a user asks, "That sounds good. Are there any other options?", the emotion engine analyzes the user's facial expressions and tone of voice to obtain emotional data.

[1488] Step 10:

[1489] The device again sends the new need ("more reasonable options") and emotion data to the server.

[1490] Step 11:

[1491] The server searches the database again based on the new needs and emotional data, and uses an AI algorithm to generate an even more optimal product list, taking into account the emotional data and including products that are likely to interest the user.

[1492] Step 12:

[1493] The server transmits the regenerated product list to the terminal.

[1494] Step 13:

[1495] The device will then make a new suggestion to the user, saying, "This smartphone also has a high-performance camera, but it's more reasonably priced."

[1496] Step 14:

[1497] The user responds, "That's great. Please tell me more about the specs." The emotion engine analyzes the user's facial expression and tone of voice at this time to obtain emotional data.

[1498] Step 15:

[1499] The device sends the user's reaction and emotional data to the server, which then generates detailed specification information and sends it to the device using the most appropriate expressions based on the emotional data.

[1500] Step 16:

[1501] The device provides detailed specifications to the user, and sends the user's reaction and emotional data to the server, which then accumulates all the data and updates the model to optimize the next proposal.

[1502] Example 2

[1503] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1504] Conventional customer service systems in shopping malls have had difficulty fully reflecting individual emotions and detailed requests when proposing products that meet users' needs. As a result, they have been unable to make appropriate proposals that stimulate purchasing motivation, and their effectiveness in improving customer satisfaction and promoting sales has been limited. The object of the present invention is to solve these problems by providing a system that recognizes users' emotions in real time and makes appropriate product proposals.

[1505] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1506] In this invention, the server includes means for detecting the approach of a user, means for starting a conversation with the user and collecting the user's needs, means for transmitting the collected user needs and emotion data to the server, means for the server to select optimal products based on the user's needs and emotion data and generate a proposal list, means for presenting the generated proposal list to the user, means for recollecting the user's reactions and transmitting them to the server to optimize the next proposal, and means for grasping the user's emotions in real time using an emotion recognition engine. This makes it possible to flexibly respond to the individual emotions and needs of the user and make optimal product proposals.

[1507] "User" refers to an individual who receives product information from the system.

[1508] "Server" refers to a central management unit that analyzes data sent by users and generates optimal product suggestions.

[1509] "Terminal" refers to a device that directly interacts with users, gathers their needs, and connects to a server.

[1510] "Emotional data" refers to information that indicates the emotional state of a user, as determined by facial expressions and vocal tones.

[1511] "Proposal list" refers to a list of optimal products generated by the server and presented to the user.

[1512] An "emotion recognition engine" refers to software that analyzes a user's facial expressions and tone of voice in real time to determine their emotions.

[1513] "Speech recognition technology" refers to technology that converts a user's speech into text data.

[1514] "Facial recognition technology" refers to technology that uses a camera to identify a user's face and determine whether it is familiar or new.

[1515] MODE FOR CARRYING OUT THE INVENTION

[1516] (System Overview)

[1517] This invention is an AI sales assistant robot system that guides users visiting a shopping mall to the most suitable products. This system operates through dialogue with the user and, by combining it with an emotion recognition engine, recognizes the user's emotions and optimizes the content of the recommendations. Specific embodiments for implementing this system are described below.

[1518] (Device configuration and operation)

[1519] The terminal will be installed in shopping malls as a sales assistant robot and is equipped with a built-in camera, infrared sensors, voice recognition technology, and an emotion recognition engine.

[1520] User approach detection

[1521] The device uses a built-in camera and infrared sensor to detect the user's movement when they approach within a certain distance. Facial recognition technology is used to determine whether the user is an existing or new user. For example, the built-in camera can be a standard webcam, and the infrared sensor can be a commercially available motion sensor. Facial recognition uses technologies such as Face++.

[1522] Starting the initial dialogue

[1523] When a user approaches, the device will speak a pre-programmed initial greeting (e.g., "Hello! Is there anything I can help you with?") using voice synthesis technology, which can use technologies such as Google Text-to-Speech.

[1524] Gathering needs

[1525] If the user responds, "I'm looking for a new smartphone," the device uses voice recognition software (e.g., Google Cloud Speech-to-Text) to convert the utterance into text data, then asks, "What features are you looking for in a smartphone?" and presents options such as "camera performance," "battery life," and "price range."

[1526] emotion recognition

[1527] When a user answers, "I want a smartphone with a good camera," the device's emotion recognition engine (e.g., Affectiva) is used to determine the user's emotion in real time from their facial expressions and tone of voice. This emotion data is then sent to the server along with the user's response.

[1528] (Server configuration and operation)

[1529] The server acts as a central control unit that analyzes the data sent from the terminals and generates product suggestions. The following processes are performed:

[1530] Data reception and analysis

[1531] The server receives the user's needs (e.g., "a smartphone with a good camera") and emotional data sent from the device. It uses an AI algorithm to search the database and generate the optimal product list. The AI ​​algorithm uses a natural language processing model using TensorFlow.

[1532] Generate and send a list of suggestions

[1533] The server then sends the generated product list to the device. It considers the user's emotional data and prioritizes expressions and products that are likely to interest the user. If the user is excited, the latest high-performance camera smartphones are placed at the top of the list.

[1534] (Suggestion and feedback handling)

[1535] The terminal makes suggestions to the user based on the product list sent from the server.

[1536] Proposal presentation

[1537] The device will suggest to the user, "This smartphone has excellent camera performance, especially with high resolution and excellent night photography." By emphasizing specific features (e.g., the night photography function "Night Sight"), it will attract the user's interest.

[1538] Collecting user responses

[1539] The user asks a follow-up question, "That sounds good. Are there any other options?" The device's emotion recognition engine again analyzes the user's facial expressions and tone of voice and sends the results to the server.

[1540] Generate and present re-proposals

[1541] The server searches the database again based on the new need (e.g., "reasonable price") and emotion data, generates a more suitable product list, and sends it to the device. The device then makes a new suggestion, saying, "This smartphone also has a high-performance camera, but at a more reasonable price."

[1542] (Continuous self-learning)

[1543] The server accumulates all conversation and emotion data and continually updates the AI ​​model to improve the accuracy of the next recommendation. For example, if a user frequently requests "reasonable prices," the server will prioritize presenting options with lowered prices to that user.

[1544] Example prompt

[1545] Example prompt: "User is looking for a new smartphone. Suggest a smartphone that focuses on camera performance, and if they inquire about price, suggest a more affordable option."

[1546] In this way, the present invention can flexibly respond to the individual needs and feelings of the user and make optimal product suggestions.

[1547] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1548] Program processing flow

[1549] Step 1: Detecting user approach

[1550] Input: The device receives data from the built-in camera and infrared sensor.

[1551] Data processing: Analyzes camera footage and reads motion sensor signals.

[1552] Output: Detects that the user is approaching within a certain distance.

[1553] Specific operation: The device analyzes camera footage in real time, identifies the user's face using facial recognition technology, and detects the user's approach from the infrared sensor signal to determine whether the user is a new or existing user.

[1554] Step 2: Initiating initial interactions

[1555] Input: Signal that detects the user's approach.

[1556] Data processing: Generation of greetings and speech synthesis.

[1557] Output: The device says "Hello! Is there anything I can help you with?"

[1558] Specific behavior: Uses the device's speech synthesis technology (e.g., Google Text-to-Speech) to audibly transmit a programmed greeting to the user.

[1559] Step 3: Gathering needs

[1560] Input: User utterance (e.g., "I'm looking for a new smartphone").

[1561] Data processing: Convert speech into text data using speech recognition technology.

[1562] Output: The text data is "Looking for a new smartphone."

[1563] What it does: The device uses voice recognition software (e.g., Google Cloud Speech-to-Text) to convert what the user says into text in real time, analyzes that text, and generates the next question (e.g., "What features of a smartphone are you looking for?").

[1564] Step 4: Emotion Recognition

[1565] Input: User's speech and facial expression data.

[1566] Data processing: Analyze user emotions using an emotion recognition engine.

[1567] Output: Sentiment data as the analysis result.

[1568] Specific operation: The device uses an emotion recognition engine (e.g., Affectiva) to determine the user's emotions (joy, anger, sadness, and happiness) from their facial expressions and vocal tones. The obtained emotional data is then sent to the server along with the user's needs.

[1569] Step 5: Server receives and analyzes data

[1570] Input: User needs and emotion data sent from the device.

[1571] Data processing: Using AI algorithms, we generate product lists that best suit the user's needs.

[1572] Output: A list of optimal products.

[1573] How it works: The server analyzes user needs using a natural language processing model powered by TensorFlow, searches the database to select the most suitable products, and adjusts the order of the list taking into account emotional data.

[1574] Step 6: Generate and submit a proposal list

[1575] Input: Product list generated by an AI algorithm.

[1576] Data processing: Adjust product priorities based on user sentiment.

[1577] Output: Sending the product list to the terminal.

[1578] Specific operation: The server sends the generated product list to the terminal. At this time, expressions that are likely to interest the user are given priority in the list.

[1579] Step 7: Proposal Presentation

[1580] Input: Product list sent from the server.

[1581] Data processing: Audio conversion of product list.

[1582] Output: Spoken product recommendations to the user.

[1583] Specific operation: The device will suggest to the user using a synthesized voice, "This smartphone has excellent camera performance. Its high resolution makes it particularly good for night photography."

[1584] Step 8: Collect user responses

[1585] Input: User's follow-up question (e.g., "That sounds good. Are there any other options I can choose?").

[1586] Data processing: The follow-up questions are converted into text data using voice recognition technology. The emotion data is then analyzed again using an emotion recognition engine.

[1587] Output: A new request to the server and sending emotion data.

[1588] Specific operation: The device performs voice recognition of the follow-up question, re-determines the user's current emotional state using the emotion recognition engine, and sends the analysis results to the server.

[1589] Step 9: Generate and present re-proposals

[1590] Input: A new product list from the server.

[1591] Data processing: Audio generation of new product listings.

[1592] Output: A spoken re-suggestion to the user.

[1593] Specific operation: The device makes a re-suggestion by synthesizing voice, saying, "This smartphone also has a high-performance camera, but it's more reasonably priced."

[1594] Prompt Sentence Examples

[1595] Example prompt: "User is looking for a new smartphone. Suggest a smartphone that focuses on camera performance, and if they inquire about price, suggest a more affordable option."

[1596] (Application example 2)

[1597] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1598] Conventional shopping support systems have difficulty accurately gathering user needs and suggesting optimal products. Furthermore, because they do not take the user's emotions into consideration, the content of the suggestions is unlikely to capture the user's interest and fail to stimulate purchasing motivation. Furthermore, there is a lack of personalized suggestions linked to past request data, making it impossible to achieve optimization for each user. There is a need to solve these problems and provide a more effective customer experience.

[1599] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1600] In this invention, the server includes means for detecting the approach of a user, means for initiating a conversation with the user and collecting the user's needs, means for transmitting the collected user needs to the server, means for the server to select optimal products based on the user's needs and generate a proposal list, means for presenting the generated proposal list to the user, means for recollecting the user's reactions and transmitting the data to the server to optimize the next proposal, means for recognizing the user's emotions in real time using emotion recognition technology and transmitting the data to the server, and means for optimizing the proposal content based on the user's emotion data. This makes it possible to make personalized proposals according to the user's needs and improve the customer experience.

[1601] "Means for detecting a user's approach" refers to technology or devices that use cameras or sensors to detect a user's approach to the system.

[1602] "Means for starting a conversation with a user and gathering information about the user's needs" refers to technology or devices that use voice recognition or natural language processing to engage in a dialogue with a user and gather information about the user's requests and wishes.

[1603] The "means for transmitting collected user needs to a server" refers to a communication technology or device for transferring the user needs collected by a terminal to a server as data.

[1604] "Means for the server to select optimal products based on the user's needs and generate a proposal list" refers to technology or software that allows the server to analyze the user's needs, select optimal products based on those needs, and create a list.

[1605] The "means for presenting the generated proposal list to the user" refers to a technique or device for visually or audibly presenting the generated proposal list to the user.

[1606] "Means for re-collecting user responses and sending them to a server to optimize the next proposal" refers to technology or devices for re-collecting user responses to a proposal, sending them to a server, and optimizing the content of the next proposal based on that data.

[1607] "Means for recognizing a user's emotions in real time using emotion recognition technology and transmitting that data to a server" refers to technology or devices that analyze a user's facial expressions and tone of voice to recognize emotions and transmit the results to a server in real time.

[1608] "Means for optimizing proposal content based on user emotion data" refers to technology or devices for optimizing the proposed products or presentation content by taking into account the recognized user emotion data.

[1609] This invention is a smart shopping assistant system that interacts with customers in physical stores and makes product recommendations. This system consists of the following main components:

[1610] Components

[1611] Hardware

[1612] 1. Sales assistant robot: A device that directly interacts with users and gathers their needs. It may use built-in cameras, microphones, and sensors to detect the user's approach and identify them using facial recognition technology.

[1613] 2. Built-in camera and microphone: Captures the user's facial expressions and voice, providing data for emotion recognition technology.

[1614] 3. Approach detection sensor: Sensor technology that detects the approach of a user.

[1615] software

[1616] 1. Emotion recognition engine: Technology that analyzes the user's facial expressions and vocal tone in real time to recognize their emotional state. Utilizing technologies such as "Affectiva" and "Microsoft Azure Emotion API."

[1617] 2. Speech recognition system: A technology that converts user speech into text data. Services such as "Google Cloud Speech-to-Text" and "Amazon Transcribe" are used.

[1618] 3. Database management systems: Used to store data on user needs and emotions and make optimal product recommendations. Examples include "MySQL" and "MongoDB."

[1619] 4. Natural language processing engine: A technology used to analyze collected data and make product suggestions based on user needs. It uses technologies such as "GPT-4 by OpenAI" and "BERT."

[1620] Operation explanation

[1621] Device behavior

[1622] 1. User approach detection: The sales assistant robot's built-in camera and sensors detect the user's approach and identify the user using facial recognition technology.

[1623] 2. Start of initial interaction: When the robot detects the user's approach, it greets them with "Hello! Is there anything I can help you with?"

[1624] 3. Gathering needs: If the user responds, "I'm looking for a new smartphone," the robot will ask for more specific needs, such as, "Are you looking for a smartphone with a good camera?"

[1625] 4. Emotion recognition: During this speech, the emotion recognition engine processes the user's facial expressions and tone of voice in real time and sends the emotional data to the server.

[1626] Server Operation

[1627] 1. Data reception and analysis: The server receives the user's needs and emotion data sent from the device and analyzes the data using a natural language processing engine.

[1628] 2. Generate and send a list of proposals: Search the database, generate an optimal product list based on the user's needs and emotional data, and send it to the terminal.

[1629] Handling suggestions and feedback

[1630] 1. Proposal presentation: The sales assistant robot makes a proposal to the user, saying, "This smartphone has excellent camera performance. It has high resolution and is particularly good at taking night shots."

[1631] 2. Collecting user responses: If the user asks, "That sounds good. Are there any other options?", the robot sends a new request and emotional data to the server.

[1632] 3. Generation and presentation of a new recommendation: The server generates a more optimal product list based on the new needs and emotional data and sends it to the device. The robot makes a new recommendation, saying, "This smartphone also has a high-performance camera, but it's more reasonably priced."

[1633] Examples and prompts

[1634] Examples:

[1635] The user approaches and the approach is detected.

[1636] Robot: "Hello! Is there anything you're looking for?"

[1637] User: "Hi. I'm looking for a new smartphone."

[1638] Example prompt sentence:

[1639] The user approaches and the approach is detected.

[1640] Robot: "Hello! Is there anything you're looking for?"

[1641] User: "Hi. I'm looking for a new smartphone."

[1642] Robot: "What features are you looking for in a smartphone?"

[1643] User: "I want a smartphone with a good camera."

[1644] The robot sends this request and emotional data to the server.

[1645] The server generates a list of optimal smartphones and sends it to the robot.

[1646] Robot: "The camera on this smartphone is very good. It's especially good at taking high-resolution photos at night."

[1647] User: "That sounds good. Are there any other options?"

[1648] The robot sends a new request and emotion data to the server and gets the suggestion list again.

[1649] Robot: "This smartphone also has a high-performance camera, but at a reasonable price."

[1650] In this way, the smart shopping assistant system can highly personalize the customer experience in physical stores and efficiently make optimal product recommendations.

[1651] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1652] Step 1:

[1653] User approach detection

[1654] Input: Real-time video data from cameras and sensors.

[1655] How it works: The sales assistant robot (terminal) monitors its surroundings using a built-in camera and proximity detection sensors, and detects when a user approaches. It uses facial recognition technology to determine whether the user is a known or new user.

[1656] Output: Information on the detection of the user's approach and the user ID identified as the face recognition result.

[1657] Step 2:

[1658] Starting the initial dialogue

[1659] Input: Information on the user's approach detection and user ID.

[1660] How it works: The device greets the user with a voice message saying, "Hello! Is there anything I can help you with?" and begins the conversation.

[1661] Output: A spoken response from the user.

[1662] Step 3:

[1663] Gathering needs

[1664] Input: A spoken response from the user.

[1665] How it works: The device uses a voice recognition system (Google Cloud Speech-to-Text) to convert the user's voice into text data, then asks, "What kind of smartphone are you looking for?"

[1666] Output: User needs information as converted text data.

[1667] Step 4:

[1668] emotion recognition

[1669] Input: User's voice response and video data.

[1670] How it works: The device uses an emotion recognition engine (Affectiva or Microsoft Azure Emotion API) to analyze the user's facial expressions and voice tone in real time and generate emotion data.

[1671] Output: User emotion data.

[1672] Step 5:

[1673] Sending data

[1674] Input: User needs information and emotion data.

[1675] Operation: The terminal sends the collected data to the server.

[1676] Output: User needs information and emotion data sent to the server.

[1677] Step 6:

[1678] Generate a list of suggestions

[1679] Input: User needs information and emotion data sent to the server.

[1680] How it works: The server analyzes the data using a natural language processing engine (GPT-4 by OpenAI, BERT, etc.), searches for relevant product information in a database management system (MySQL or MongoDB), generates an optimal product list, and prioritizes it taking into account sentiment data.

[1681] Output: The generated optimal product list.

[1682] Step 7:

[1683] Presenting the proposal

[1684] Input: Generated optimal product list.

[1685] What it does: The device recommends the product to the user, saying, "This smartphone has excellent camera performance. Its high resolution makes it particularly good for nighttime photography."

[1686] Output: Product information presented to the user.

[1687] Step 8:

[1688] Collecting user responses

[1689] Input: User response (voice, facial expression).

[1690] How it works: The device uses its voice recognition system and emotion recognition engine to again collect the user's additional requests and reactions and sends this data to the server.

[1691] Output: Newly collected user needs information and emotion data.

[1692] Step 9:

[1693] Generate and present re-proposals

[1694] Input: Newly collected user needs information and emotion data.

[1695] How it works: The server again analyzes the data using a natural language processing engine, searches the database, and generates a new list of suggestions. The list is then sent to the device, which then makes a new suggestion, saying, "This smartphone also has a high-performance camera, but it's more reasonably priced."

[1696] Output: Product information re-presented to the user.

[1697] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1698] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1699] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1700] [Fourth embodiment]

[1701] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1702] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1703] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1704] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1705] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1706] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1707] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1708] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1709] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1710] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1711] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1712] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1713] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1714] This invention is an AI sales assistant robot system that guides users visiting shopping malls about products such as smartphones and water dispensers. This system operates mainly through interactions with terminals, servers, and users. Below, we provide a detailed explanation of the operation of each element and provide specific examples.

[1715] System configuration

[1716] 1. Terminal (Sales Assistant Robot):

[1717] Dialogue directly with users to gather their needs.

[1718] Built-in sensors and cameras may be used to detect the user's approach and facial recognition technology may be used to identify the user.

[1719] Using voice recognition technology, the user's questions and requests are converted into text data and sent to the server.

[1720] 2. Server:

[1721] The user's needs data transmitted from the terminal is received and analyzed.

[1722] Using AI algorithms, we generate product lists that best suit the user's needs.

[1723] The generated proposal list is sent to the terminal.

[1724] It accumulates user response data and performs self-learning to improve the accuracy of its next suggestions.

[1725] Program processing explanation

[1726] Device behavior

[1727] 1. User approach detection:

[1728] The device uses its built-in camera and sensors to detect when a user approaches, along with facial recognition technology to determine whether the user is a known or new user.

[1729] 2. Initiating the initial dialogue:

[1730] When the device detects a user approaching, it greets them with "Hello! Is there anything you're looking for?"

[1731] 3. Gathering needs:

[1732] If the user answers, "I'm looking for a new smartphone," the device will then ask, "What features are you looking for in a smartphone?" If the user answers, "I want a smartphone with a good camera," the device will send this information to the server.

[1733] Server Operation

[1734] 1. Data Receipt and Analysis:

[1735] The server receives the user's needs ("a smartphone with a good camera") sent from the device, searches the database using an AI algorithm, and generates the optimal product list.

[1736] 2. Generate and send the proposal list:

[1737] The server transmits the generated product list to the terminal.

[1738] Handling suggestions and feedback

[1739] 1. Presentation of the proposal:

[1740] The device advises the user, "This smartphone has excellent camera performance. Its high resolution makes it particularly good for night photography."

[1741] 2. Collecting user responses:

[1742] If the user responds with, "That sounds good. Are there any other options?" the device sends that information back to the server and generates additional suggestions.

[1743] 3. Generate and present re-proposals:

[1744] The server searches the database again based on the new needs (price is also important), generates an optimal product list, and sends it to the device. The device then re-suggests to the user, "This smartphone also has a high-performance camera, but it's more reasonably priced."

[1745] Specific examples

[1746] 1. The user approaches and is detected:

[1747] Terminal: "Hello! Is there anything you're looking for?"

[1748] User: "Hi. I'm looking for a new smartphone."

[1749] 2. Make suggestions based on needs:

[1750] Device: "What features of a smartphone are you looking for?"

[1751] User: "I want a smartphone with a good camera."

[1752] The terminal sends this request to the server.

[1753] The server generates a list of optimal smartphones and sends it to the device.

[1754] Device: "The camera on this smartphone is excellent. It's particularly high-resolution and great for night photography."

[1755] 3. The user makes an additional request:

[1756] User: "That sounds good. Are there any other options?"

[1757] The device sends a new request to the server, gets another product list, and makes another suggestion.

[1758] Device: "This phone also has a great camera, but it's more affordable."

[1759] Continuous self-learning

[1760] The server accumulates all interaction data and continuously updates the model to optimize the next suggestion, thus providing each user with the best possible suggestions in real time.

[1761] In this way, the present invention efficiently delivers personalized offers tailored to each customer's individual needs, enhancing the customer experience.

[1762] The processing flow will be explained below.

[1763] Step 1:

[1764] The device uses its built-in camera and sensors to detect the user's approach, and may then use facial recognition technology to identify the user.

[1765] Step 2:

[1766] When the device approaches a user, it greets them with "Hello! Is there anything you're looking for?"

[1767] Step 3:

[1768] The user replies, "I'm looking for a new phone."

[1769] Step 4:

[1770] The device converts the user's speech into text data using voice recognition technology and continues to ask questions such as, "What features of a smartphone are you looking for?" At this time, it presents options such as camera performance and battery life.

[1771] Step 5:

[1772] The user responds, "I want a smartphone with a good camera." The device converts this response back into text data and sends it to the server.

[1773] Step 6:

[1774] The server analyzes the user's needs ("I want a smartphone with a good camera") and searches the database to generate the optimal product list.

[1775] Step 7:

[1776] The server transmits the generated product list to the terminal.

[1777] Step 8:

[1778] The device will advise the user, "This smartphone has excellent camera performance. Its high resolution makes it particularly good for taking night shots."

[1779] Step 9:

[1780] The user asks, "That sounds good. What other options do I have?"

[1781] Step 10:

[1782] The device converts the new need ("more reasonable options") into text data and sends it to the server.

[1783] Step 11:

[1784] The server searches the database again based on the new needs and generates a further product list.

[1785] Step 12:

[1786] The server transmits the regenerated product list to the terminal.

[1787] Step 13:

[1788] The device will then make a new suggestion to the user, saying, "This smartphone also has a high-performance camera, but it's more reasonably priced."

[1789] Step 14:

[1790] The user responds, "That's great. Can you give me more detailed specs?"

[1791] Step 15:

[1792] The device sends the user's response to the server, and the server generates detailed specification information and sends it to the device.

[1793] Step 16:

[1794] The device provides detailed information to the user, and the server stores all interaction data and updates the model to further optimize the next suggestion.

[1795] Example 1

[1796] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1797] Traditional product guidance in shopping malls relies on face-to-face human resources and is inefficient. It is also difficult to provide personalized suggestions tailored to user needs, making it difficult to quickly and accurately guide users to the products they are looking for. Furthermore, there is a lack of a mechanism for efficiently incorporating user feedback, making it difficult to improve the quality of next suggestions.

[1798] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1799] In this invention, the server includes means for detecting the approach of a user, means for starting a conversation with the user and collecting the user's needs, means for transmitting the collected user needs to the server, means for the server to select optimal products based on the user's needs and generate a proposal list, means for presenting the generated proposal list to the user, means for recollecting the user's reactions and transmitting them to the server to optimize the next proposal, and means for the server to accumulate dialogue data and perform self-learning to optimize the next proposal, thereby enabling efficient and personalized product guidance according to the user's needs.

[1800] "Means for detecting the approach of a user" refers to a device or technology that uses a built-in camera or sensor to detect when a user approaches within a certain range.

[1801] "Means for starting a conversation with the user and gathering user needs" refers to the process of using speech synthesis technology and natural language processing technology to converse with the user and gather information and product requests that the user is looking for.

[1802] The "means for transmitting the collected user needs to the server" refers to a communication means or protocol for transmitting the information collected from the user to the server via a network.

[1803] "Means by which the server selects optimal products based on the user's needs and generates a proposal list" refers to algorithms or technologies by which the server analyzes collected user information, selects relevant products from a database, and generates a list.

[1804] The "means for presenting the generated proposal list to the user" refers to a device such as a display or a speaker that visually or audibly presents the generated product proposal list to the user.

[1805] "Means for collecting user responses again and sending them to the server to optimize future suggestions" refers to the process of collecting the responses that users give to suggestions and sending that information back to the server to improve the quality of future suggestions.

[1806] "Means for the server to accumulate dialogue data and self-learn to optimize the next proposal" refers to a technology in which the server saves past dialogue data with the user, uses that data to apply machine learning technology, and improves the accuracy of the next proposal.

[1807] This invention is an AI sales assistant robot system that guides users visiting shopping malls about products such as smartphones and water dispensers. This system operates mainly through interactions between terminals, servers, and users.

[1808] System configuration

[1809] 1. Terminal (Sales Assistant Robot):

[1810] Approach Detection:

[1811] The device uses a built-in camera (e.g., Logitech HD Pro Webcam C920) and sensors (e.g., HC-SR501 PIR motion sensor) to detect the approach of a user, and uses facial recognition technology (e.g., OpenCV library) to determine whether the user is a known or new user.

[1812] Example: When a user approaches within 2 meters of a robot, the device's sensors detect this and the camera captures the user's face.

[1813] Beginning the initial interaction:

[1814] When the device detects the user's approach, it uses speech synthesis software (e.g., Google Text-to-Speech API) to greet the user, saying, "Hello! Is there anything I can help you with?"

[1815] Example: When the device says, "Hello! Is there anything I can help you with?" the user turns towards the device in surprise.

[1816] Gathering needs:

[1817] When the user responds by saying, "I'm looking for a new smartphone," the device uses voice recognition technology (e.g., Google Cloud Speech-to-Text API) to convert this request into text data and send it to the server.

[1818] Example: When a user says, "I'm looking for a new smartphone," the device converts it into text and sends it to the server as "I'm looking for a new smartphone."

[1819] 2. Server:

[1820] Data reception and analysis:

[1821] The server receives user needs (e.g., "a smartphone with a good camera") sent from the device, searches the database using AI algorithms (e.g., Python and the Scikit-learn library), and generates the optimal product list. This uses a generative AI model such as the XGBoost model.

[1822] Example: The server receives the need "Looking for a new smartphone" and selects a suitable smartphone from the database.

[1823] Generate and send a list of suggestions:

[1824] The server transmits the generated product list to the terminal.

[1825] Example: The server generates a list of "smartphones with good camera performance" and sends it to the device.

[1826] 3. Handling Suggestions and Feedback:

[1827] Presentation of proposal:

[1828] The device advises the user, "This smartphone has excellent camera performance. Its high resolution makes it particularly good for night photography."

[1829] Example: The device suggests to the user, "This smartphone has excellent camera performance."

[1830] User response collection and re-proposal:

[1831] If the user responds with, "That sounds good. Are there any other options?" the device sends that information back to the server and generates additional suggestions.

[1832] Example: When a user asks, "Are there any other options available?", the device sends the request to the server.

[1833] Generate and present re-proposals:

[1834] The server receives the new needs, generates a new appropriate product list, and sends it to the terminal, which then makes new suggestions to the user based on this list.

[1835] Example: The device re-proposes, "This smartphone also has a high-performance camera, but at a more reasonable price."

[1836] 4. Continuous self-learning:

[1837] The server stores all interaction data and continuously trains the AI ​​model to optimize the next suggestion, resulting in more personalized suggestions for the next user.

[1838] Example: The server uses user interaction data to update the model to improve the accuracy of the next suggestion.

[1839] The overall process of this system makes it possible to efficiently gather user needs and make optimal product recommendations in real time. By using the above-mentioned means, shopping mall visitors can receive prompt and appropriate product information, providing a more satisfying shopping experience.

[1840] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1841] System program processing flow

[1842] Step 1:

[1843] User approach detection

[1844] The device uses its built-in camera and sensors to constantly monitor whether a user is approaching. When the user approaches within a certain range, the device uses facial recognition technology (e.g., OpenCV library) to recognize the user. At this time, the device's camera captures video and the sensor receives distance data as input. Based on this data, the device runs a facial recognition algorithm to output facial information.

[1845] Specific operations: camera image capture, sensor data acquisition, face recognition

[1846] Input: Camera image data, sensor data

[1847] Output: Face recognition result (user is new or known)

[1848] Step 2:

[1849] Starting the initial dialogue

[1850] When the device detects the user's proximity, it uses speech synthesis software (e.g., Google Text-to-Speech API) to greet the user with "Hello! Is there anything I can help you with?" This speech is transmitted directly to the user through the speaker.

[1851] Specific operations: speech synthesis, speech output from speaker

[1852] Input: Approach detection result

[1853] Output: Synthesized speech greeting

[1854] Step 3:

[1855] Gathering needs

[1856] When the user responds by saying "I'm looking for a new smartphone," the device uses voice recognition technology (e.g., Google Cloud Speech-to-Text API) to convert this request into text data, which is then sent to the server for further processing.

[1857] Specific operations: speech recognition, text data generation

[1858] Input: User voice input

[1859] Output: Text data (e.g. "Looking for a new smartphone")

[1860] Step 4:

[1861] Sending data

[1862] The terminal sends the converted text data to the server, often using HTTP or WebSocket as the communication protocol.

[1863] Specific operations: sending data, using communication protocols

[1864] Input: Text data

[1865] Output: Send text data to the server

[1866] Step 5:

[1867] Data reception and analysis

[1868] The server receives the user needs data sent from the device and analyzes it using an AI algorithm (e.g., Python and the Scikit-learn library). The server then accesses a database and generates a list of products that best fit the user's needs. This uses a generative AI model such as the XGBoost model.

[1869] Specific operations: receiving data, analyzing it with AI algorithms, and executing database queries

[1870] Input: User needs data

[1871] Output: Optimal product list

[1872] Step 6:

[1873] Generate and send a list of suggestions

[1874] The server generates an optimal product list based on the analysis results and sends it to the terminal. This list includes detailed product information (e.g., product name, specifications, price, etc.).

[1875] Specific operations: Generate product list, send data

[1876] Input: Analysis results of the AI ​​algorithm

[1877] Output: Send product list to terminal

[1878] Step 7:

[1879] Proposal presentation

[1880] Based on the product list received from the server, the device makes suggestions to the user via voice, providing specific explanations such as, "This smartphone has excellent camera performance."

[1881] Specific operations: speech synthesis, speech output from speaker

[1882] Input: Product list

[1883] Output: Spoken suggestions

[1884] Step 8:

[1885] Collecting user responses and re-proposing

[1886] When the user makes additional requests, the device converts their responses into text using voice recognition technology again and sends it to the server.

[1887] Specific operations: speech recognition, text data generation, data transmission

[1888] Input: User's voice response

[1889] Output: Send text data to the server

[1890] Step 9:

[1891] Generate and present re-proposals

[1892] The server receives the new request, generates a new appropriate product list, and sends it to the terminal, which then makes new suggestions to the user based on this list.

[1893] Specific operations: receiving data, analyzing with AI algorithms, generating product list, sending data, and synthesizing voice

[1894] Input: New user request

[1895] Output: Re-proposed product list

[1896] Step 10:

[1897] Data accumulation and learning

[1898] The server stores all interaction data and continuously trains the AI ​​model to optimize the next suggestion, resulting in more personalized suggestions for the next user.

[1899] Specific actions: Data accumulation, AI model training, performance improvement

[1900] Input: User interaction data

[1901] Output: Optimized AI model

[1902] (Application example 1)

[1903] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1904] In today's brick-and-mortar stores, it is difficult to quickly propose products that accurately meet customer needs. Furthermore, there is a lack of systems to effectively utilize the information obtained through dialogue with customers, and the accuracy of proposals and the optimization of future proposals are not sufficiently implemented. As a result, improvements to the customer experience are hindered, leading to a decline in store sales and customer satisfaction.

[1905] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1906] In this invention, the server includes means for detecting the approach of a user, means for initiating a conversation with the user and collecting the user's needs, means for transmitting the collected user needs to the server, means for the server to select optimal information based on the user's needs and generate a proposal list, means for presenting the generated proposal list to the user, means for recollecting the user's responses and transmitting them to the server to optimize the next proposal, means for analyzing the user's needs using an AI algorithm and searching a database, and means for self-learning based on the collected feedback to improve the accuracy of proposals from the next time onwards. This enables product proposals that quickly and accurately meet customer needs, improving the customer experience and increasing store sales.

[1907] "Means for detecting the approach of a user" refers to technology that uses a camera or sensor to detect the presence of a user when they arrive within a pre-set distance.

[1908] "Means for starting a conversation with the user and gathering information about the user's needs" refers to a function that uses voice recognition technology and natural language processing to converse with the user and understand their requests and wishes.

[1909] The "means for transmitting collected user needs to a server" is a communication means for transferring collected data to a central database via the Internet.

[1910] "Means for the server to select the most appropriate information based on the user's needs and generate a recommendation list" refers to a function that uses AI algorithms and data analysis technology to find the products and services that best meet the user's requirements and create a recommendation list.

[1911] The "means for presenting the generated proposal list to the user" is a function that uses a display or voice synthesis technology to present information that is determined to be most suitable to the user in an easy-to-understand manner.

[1912] "Means for recollecting user responses and sending them to a server to optimize the next proposal" refers to a technology that collects the interest and feedback that users have shown in the proposal and sends that data to a server to improve the quality of future proposals.

[1913] "Means of analyzing user needs using AI algorithms and searching a database" refers to a function that analyzes needs data collected using machine learning and deep learning, and searches a database for appropriate information.

[1914] "Means for self-learning based on collected feedback to improve the accuracy of future proposals" refers to a system that uses accumulated user feedback data as learning material to improve the AI ​​model and fine-tune the algorithm, thereby improving the accuracy of proposals.

[1915] This invention is an AI sales assistant system for suggesting products according to customer needs in shopping malls and brick-and-mortar stores, and it mainly operates using a terminal (e.g., a smartphone or robot) and a server. The main function of this system is to detect when a user approaches, collect and analyze the user's needs through dialogue, and generate a suggestion list.

[1916] System configuration

[1917] Device configuration

[1918] 1. User approach detection:

[1919] The device uses its built-in camera and sensors to detect approaching users in real time, and uses facial recognition technology (e.g., TensorFlow) to distinguish between existing and new users.

[1920] 2. Initiating conversations and gathering needs:

[1921] When the user approaches, the device greets the user through voice output (e.g., Google Cloud Text-to-Speech) and initiates a dialogue. It then uses voice recognition technology (e.g., Google Cloud Speech-to-Text) to convert the user's request into text data and transmits that data to the server.

[1922] 3. Present the list of proposals and gather feedback:

[1923] The device that receives the proposal list presents it to the user via voice output or a display (e.g., a smartphone or robot screen). The user's reactions and feedback are collected again and sent to the server.

[1924] Server Configuration

[1925] 1. Data Receipt and Analysis:

[1926] The server receives the user's needs data sent from the device and searches the database using an AI algorithm. Specifically, it analyzes the data using a machine learning model (e.g., a generative AI model) to generate an optimal product list.

[1927] 2. Generate the proposal list:

[1928] Based on the analysis results, a product list that best suits the user's needs is generated and sent to the terminal.

[1929] 3. Feedback accumulation and self-learning:

[1930] The server accumulates user feedback data sent from the device and performs self-learning to improve the accuracy of suggestions from the next time onwards, thereby enabling continuous improvement of the accuracy of suggestions.

[1931] Hardware and software used

[1932] Cameras and sensors: Used to detect the user's approach.

[1933] Microphone and speaker: Used for audio input and output.

[1934] Display: Used to visually display the suggestion list.

[1935] TensorFlow: Used to provide facial recognition technology.

[1936] Google Cloud Speech-to-Text: Used to provide speech recognition technology.

[1937] Google Cloud Text-to-Speech: Used to provide audio output.

[1938] Firebase or AWS DynamoDB: Used to manage data and manage AI models.

[1939] Specific examples

[1940] 1. User A approaches:

[1941] Terminal: "Hello! Is there anything you're looking for?"

[1942] User: "Hi. I'm looking for a new smartphone."

[1943] 2. Make suggestions based on needs:

[1944] Device: "What features of a smartphone are you looking for?"

[1945] User: "I want a smartphone with a good camera."

[1946] The terminal sends this request to the server.

[1947] The server generates a list of optimal smartphones and sends it to the device.

[1948] Device: "The camera on this smartphone is excellent. It's particularly high-resolution and great for night photography."

[1949] 3. The user makes an additional request:

[1950] User: "That sounds good. Are there any other options?"

[1951] The device sends a new request to the server, gets another product list, and makes another suggestion.

[1952] Device: "This phone also has a great camera, but it's more affordable."

[1953] Prompt Sentence Examples

[1954] "Design an AI assistant for a physical store that detects the user's proximity and suggests the best products based on their needs. Write a Python program that recognizes the customer's voice input and generates and presents a list of products."

[1955] As a result, the present invention makes it possible to quickly grasp the needs of customers in physical stores and provide the most suitable products to meet those needs, thereby improving the customer experience and increasing store sales.

[1956] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1957] Step 1:

[1958] The device uses its built-in camera and sensors to detect when a user approaches. Specifically, it analyzes the frames captured by the camera and uses facial recognition technology (e.g., TensorFlow) to identify whether the user is an existing or new user. Here, the input is the camera image frame, and the output is the user approach detection information.

[1959] Step 2:

[1960] When the device detects the user's approach, it uses voice output (e.g., Google Cloud Text-to-Speech) to address the user, saying, "Hello! Is there anything I can help you with?", and begins a dialogue. Here, the input is user detection information, and the output is the start of a conversation with the user.

[1961] Step 3:

[1962] When the user responds, "I'm looking for a new smartphone," the device uses voice recognition technology (e.g., Google Cloud Speech-to-Text) to convert this speech into text data. It then asks, "What features of a smartphone are you looking for?" Here, the input is voice data, and the output is text data.

[1963] Step 4:

[1964] If the user answers, "I want a smartphone with a good camera," the device sends this need data to the server. Here, the input is text data, and the output is the transmission of the need data to the server.

[1965] Step 5:

[1966] The server receives the received needs data and analyzes it using an AI algorithm (e.g., a generative AI model). Specifically, it uses a machine learning model to generate an optimal smartphone list. Here, the input is the needs data and the output is a product list.

[1967] Step 6:

[1968] The server sends the generated product list to the terminal, which then uses voice output and a display to suggest to the user, "This smartphone has excellent camera performance. Its high resolution makes it particularly good for nighttime photography." Here, the input is the product list, and the output is the suggestion to the user.

[1969] Step 7:

[1970] The user responds with "That sounds good. Are there any other options?" and the device sends this feedback back to the server. Here, the input is the user's feedback and the output is sending the feedback to the server.

[1971] Step 8:

[1972] The server analyzes the feedback data and generates an additional proposal list. The server then sends the generated proposal list to the device, which then makes a re-proposal, saying, "This smartphone also has a high-performance camera, but it's more reasonably priced." Here, the input is the feedback data, and the output is the new product list and the re-proposal.

[1973] Step 9:

[1974] The server accumulates all the interaction data and performs self-learning to optimize the next suggestion, thereby improving the accuracy of suggestions for the user's future requests. Here, the input is all the interaction data, and the output is the updated AI model.

[1975] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1976] This invention is an AI sales assistant robot system that guides users visiting shopping malls about products such as smartphones and water dispensers. This system operates primarily through dialogue between terminals, servers, and users, and by combining it with an emotion engine, it recognizes the user's emotions and optimizes the content of its recommendations. Below, we provide a detailed explanation of the operation of each element and provide specific examples.

[1977] System configuration

[1978] 1. Terminal (Sales Assistant Robot):

[1979] Dialogue directly with users to gather their needs.

[1980] Built-in sensors and cameras may be used to detect the user's approach and facial recognition technology may be used to identify the user.

[1981] Using voice recognition technology, the user's questions and requests are converted into text data and sent to the server.

[1982] Equipped with an emotion engine, it recognizes emotions in real time from the user's facial expressions and tone of voice.

[1983] 2. Server:

[1984] The system receives and analyzes the user's needs data and emotion data transmitted from the terminal.

[1985] Using AI algorithms, we generate product lists that best suit the user's needs.

[1986] Based on emotional data, suggestions are optimized to suit the user's emotional state.

[1987] The generated proposal list is sent to the terminal.

[1988] It accumulates user response data and performs self-learning to improve the accuracy of its next suggestions.

[1989] Program processing explanation

[1990] Device behavior

[1991] 1. User approach detection:

[1992] The device uses its built-in camera and sensors to detect when a user approaches, along with facial recognition technology to determine whether the user is a known or new user.

[1993] 2. Initiating the initial dialogue:

[1994] When the device detects a user approaching, it greets them with "Hello! Is there anything you're looking for?"

[1995] 3. Gathering needs:

[1996] The user responds, "I'm looking for a new smartphone." The device then asks, "What features are you looking for in a smartphone?" and offers options such as camera performance and battery life.

[1997] 4. Emotion recognition:

[1998] The user responds, "I want a smartphone with a good camera." During this utterance, the device's emotion engine analyzes the user's emotions in real time from their facial expressions and tone of voice, and sends that data to the server.

[1999] Server Operation

[2000] 1. Data Receipt and Analysis:

[2001] The server receives the user's needs ("a smartphone with a good camera") and emotional data sent from the device, searches the database using an AI algorithm, and generates the optimal product list.

[2002] 2. Generate and send the proposal list:

[2003] The server then sends the generated product list to the terminal, taking into consideration the emotional data and prioritizing expressions and products that are likely to interest the user.

[2004] Handling suggestions and feedback

[2005] 1. Presentation of the proposal:

[2006] The device advises the user, "This smartphone has excellent camera performance. Its high resolution makes it particularly good for night photography."

[2007] 2. Collecting user responses:

[2008] The user asks, "That sounds good. Are there any other options?" The device's emotion engine again analyzes the user's emotions and sends the results to the server.

[2009] 3. Generate and present re-proposals:

[2010] The server searches the database again based on the new need ("more reasonable price") and emotion data, generates a more optimal product list, and sends it to the device. The device then makes a new suggestion to the user, saying, "This smartphone also has a high-performance camera, but at a more reasonable price."

[2011] Specific examples

[2012] 1. The user approaches and is detected:

[2013] Terminal: "Hello! Is there anything you're looking for?"

[2014] User: "Hi. I'm looking for a new smartphone."

[2015] 2. Make suggestions based on needs:

[2016] Device: "What features of a smartphone are you looking for?"

[2017] User: "I want a smartphone with a good camera."

[2018] The terminal transmits this request and emotion data to the server.

[2019] The server generates a list of optimal smartphones and sends it to the device.

[2020] Device: "The camera on this smartphone is excellent. It's particularly high-resolution and great for night photography."

[2021] 3. The user makes an additional request:

[2022] User: "That sounds good. Are there any other options?"

[2023] The device sends a new request and emotion data to the server, retrieves another product list, and makes another suggestion.

[2024] Device: "This phone also has a great camera, but it's more affordable."

[2025] Continuous self-learning

[2026] The server accumulates all conversation and emotion data and continuously updates the model to optimize the next recommendation, thereby providing each user with the best possible recommendation in real time, providing a better purchasing experience.

[2027] In this way, the present invention efficiently delivers personalized offers tailored to each customer's individual needs and emotions, enhancing the customer experience.

[2028] The processing flow will be explained below.

[2029] Step 1:

[2030] The device uses its built-in camera and sensors to detect the user's approach, and may then use facial recognition technology to identify the user.

[2031] Step 2:

[2032] When the device approaches a user, it greets them with "Hello! Is there anything you're looking for?"

[2033] Step 3:

[2034] The user replies, "I'm looking for a new phone."

[2035] Step 4:

[2036] The device converts the user's speech into text data using voice recognition technology and continues to ask questions such as, "What features of a smartphone are you looking for?" At this time, it presents options such as camera performance and battery life.

[2037] Step 5:

[2038] The user responds, "I want a smartphone with a good camera." The device converts this response back into text data and sends it to the server. The emotion engine analyzes the user's facial expressions and tone of voice, and sends the user's emotional data to the server.

[2039] Step 6:

[2040] The server analyzes the user's needs ("I want a smartphone with a good camera") and emotional data received. The server searches the database and uses an AI algorithm to generate an optimal product list. At this time, the server takes into account the emotional data and prioritizes products that are likely to interest the user.

[2041] Step 7:

[2042] The server transmits the generated product list to the terminal.

[2043] Step 8:

[2044] The device will advise the user, "This smartphone has excellent camera performance. Its high resolution makes it particularly good for taking night shots."

[2045] Step 9:

[2046] When a user asks, "That sounds good. Are there any other options?", the emotion engine analyzes the user's facial expressions and tone of voice to obtain emotional data.

[2047] Step 10:

[2048] The device again sends the new need ("more reasonable options") and emotion data to the server.

[2049] Step 11:

[2050] The server searches the database again based on the new needs and emotional data, and uses an AI algorithm to generate an even more optimal product list, taking into account the emotional data and including products that are likely to interest the user.

[2051] Step 12:

[2052] The server transmits the regenerated product list to the terminal.

[2053] Step 13:

[2054] The device will then make a new suggestion to the user, saying, "This smartphone also has a high-performance camera, but it's more reasonably priced."

[2055] Step 14:

[2056] The user responds, "That's great. Please tell me more about the specs." The emotion engine analyzes the user's facial expression and tone of voice at this time to obtain emotional data.

[2057] Step 15:

[2058] The device sends the user's reaction and emotional data to the server, which then generates detailed specification information and sends it to the device using the most appropriate expressions based on the emotional data.

[2059] Step 16:

[2060] The device provides detailed specifications to the user, and sends the user's reaction and emotional data to the server, which then accumulates all the data and updates the model to optimize the next proposal.

[2061] Example 2

[2062] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2063] Conventional customer service systems in shopping malls have had difficulty fully reflecting individual emotions and detailed requests when proposing products that meet users' needs. As a result, they have been unable to make appropriate proposals that stimulate purchasing motivation, and their effectiveness in improving customer satisfaction and promoting sales has been limited. The object of the present invention is to solve these problems by providing a system that recognizes users' emotions in real time and makes appropriate product proposals.

[2064] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2065] In this invention, the server includes means for detecting the approach of a user, means for starting a conversation with the user and collecting the user's needs, means for transmitting the collected user needs and emotion data to the server, means for the server to select optimal products based on the user's needs and emotion data and generate a proposal list, means for presenting the generated proposal list to the user, means for recollecting the user's reactions and transmitting them to the server to optimize the next proposal, and means for grasping the user's emotions in real time using an emotion recognition engine. This makes it possible to flexibly respond to the individual emotions and needs of the user and make optimal product proposals.

[2066] "User" refers to an individual who receives product information from the system.

[2067] "Server" refers to a central management unit that analyzes data sent by users and generates optimal product suggestions.

[2068] "Terminal" refers to a device that directly interacts with users, gathers their needs, and connects to a server.

[2069] "Emotional data" refers to information that indicates the emotional state of a user, as determined by facial expressions and vocal tones.

[2070] "Proposal list" refers to a list of optimal products generated by the server and presented to the user.

[2071] An "emotion recognition engine" refers to software that analyzes a user's facial expressions and tone of voice in real time to determine their emotions.

[2072] "Speech recognition technology" refers to technology that converts a user's speech into text data.

[2073] "Facial recognition technology" refers to technology that uses a camera to identify a user's face and determine whether it is familiar or new.

[2074] MODE FOR CARRYING OUT THE INVENTION

[2075] (System Overview)

[2076] This invention is an AI sales assistant robot system that guides users visiting a shopping mall to the most suitable products. This system operates through dialogue with the user and, by combining it with an emotion recognition engine, recognizes the user's emotions and optimizes the content of the recommendations. Specific embodiments for implementing this system are described below.

[2077] (Device configuration and operation)

[2078] The terminal will be installed in shopping malls as a sales assistant robot and is equipped with a built-in camera, infrared sensors, voice recognition technology, and an emotion recognition engine.

[2079] User approach detection

[2080] The device uses a built-in camera and infrared sensor to detect the user's movement when they approach within a certain distance. Facial recognition technology is used to determine whether the user is an existing or new user. For example, the built-in camera can be a standard webcam, and the infrared sensor can be a commercially available motion sensor. Facial recognition uses technologies such as Face++.

[2081] Starting the initial dialogue

[2082] When a user approaches, the device will speak a pre-programmed initial greeting (e.g., "Hello! Is there anything I can help you with?") using voice synthesis technology, which can use technologies such as Google Text-to-Speech.

[2083] Gathering needs

[2084] If the user responds, "I'm looking for a new smartphone," the device uses voice recognition software (e.g., Google Cloud Speech-to-Text) to convert the utterance into text data, then asks, "What features are you looking for in a smartphone?" and presents options such as "camera performance," "battery life," and "price range."

[2085] emotion recognition

[2086] When a user answers, "I want a smartphone with a good camera," the device's emotion recognition engine (e.g., Affectiva) is used to determine the user's emotion in real time from their facial expressions and tone of voice. This emotion data is then sent to the server along with the user's response.

[2087] (Server configuration and operation)

[2088] The server acts as a central control unit that analyzes the data sent from the terminals and generates product suggestions. The following processes are performed:

[2089] Data reception and analysis

[2090] The server receives the user's needs (e.g., "a smartphone with a good camera") and emotional data sent from the device. It uses an AI algorithm to search the database and generate the optimal product list. The AI ​​algorithm uses a natural language processing model using TensorFlow.

[2091] Generate and send a list of suggestions

[2092] The server then sends the generated product list to the device. It considers the user's emotional data and prioritizes expressions and products that are likely to interest the user. If the user is excited, the latest high-performance camera smartphones are placed at the top of the list.

[2093] (Suggestion and feedback handling)

[2094] The terminal makes suggestions to the user based on the product list sent from the server.

[2095] Proposal presentation

[2096] The device will suggest to the user, "This smartphone has excellent camera performance, especially with high resolution and excellent night photography." By emphasizing specific features (e.g., the night photography function "Night Sight"), it will attract the user's interest.

[2097] Collecting user responses

[2098] The user asks a follow-up question, "That sounds good. Are there any other options?" The device's emotion recognition engine again analyzes the user's facial expressions and tone of voice and sends the results to the server.

[2099] Generate and present re-proposals

[2100] The server searches the database again based on the new need (e.g., "reasonable price") and emotion data, generates a more suitable product list, and sends it to the device. The device then makes a new suggestion, saying, "This smartphone also has a high-performance camera, but at a more reasonable price."

[2101] (Continuous self-learning)

[2102] The server accumulates all conversation and emotion data and continually updates the AI ​​model to improve the accuracy of the next recommendation. For example, if a user frequently requests "reasonable prices," the server will prioritize presenting options with lowered prices to that user.

[2103] Example prompt

[2104] Example prompt: "User is looking for a new smartphone. Suggest a smartphone that focuses on camera performance, and if they inquire about price, suggest a more affordable option."

[2105] In this way, the present invention can flexibly respond to the individual needs and feelings of the user and make optimal product suggestions.

[2106] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2107] Program processing flow

[2108] Step 1: Detecting user approach

[2109] Input: The device receives data from the built-in camera and infrared sensor.

[2110] Data processing: Analyzes camera footage and reads motion sensor signals.

[2111] Output: Detects that the user is approaching within a certain distance.

[2112] Specific operation: The device analyzes camera footage in real time, identifies the user's face using facial recognition technology, and detects the user's approach from the infrared sensor signal to determine whether the user is a new or existing user.

[2113] Step 2: Initiating initial interactions

[2114] Input: Signal that detects the user's approach.

[2115] Data processing: Generation of greetings and speech synthesis.

[2116] Output: The device says "Hello! Is there anything I can help you with?"

[2117] Specific behavior: Uses the device's speech synthesis technology (e.g., Google Text-to-Speech) to audibly transmit a programmed greeting to the user.

[2118] Step 3: Gathering needs

[2119] Input: User utterance (e.g., "I'm looking for a new smartphone").

[2120] Data processing: Convert speech into text data using speech recognition technology.

[2121] Output: The text data is "Looking for a new smartphone."

[2122] What it does: The device uses voice recognition software (e.g., Google Cloud Speech-to-Text) to convert what the user says into text in real time, analyzes that text, and generates the next question (e.g., "What features of a smartphone are you looking for?").

[2123] Step 4: Emotion Recognition

[2124] Input: User's speech and facial expression data.

[2125] Data processing: Analyze user emotions using an emotion recognition engine.

[2126] Output: Sentiment data as the analysis result.

[2127] Specific operation: The device uses an emotion recognition engine (e.g., Affectiva) to determine the user's emotions (joy, anger, sadness, and happiness) from their facial expressions and vocal tones. The obtained emotional data is then sent to the server along with the user's needs.

[2128] Step 5: Server receives and analyzes data

[2129] Input: User needs and emotion data sent from the device.

[2130] Data processing: Using AI algorithms, we generate product lists that best suit the user's needs.

[2131] Output: A list of optimal products.

[2132] How it works: The server analyzes user needs using a natural language processing model powered by TensorFlow, searches the database to select the most suitable products, and adjusts the order of the list taking into account emotional data.

[2133] Step 6: Generate and submit a proposal list

[2134] Input: Product list generated by an AI algorithm.

[2135] Data processing: Adjust product priorities based on user sentiment.

[2136] Output: Sending the product list to the terminal.

[2137] Specific operation: The server sends the generated product list to the terminal. At this time, expressions that are likely to interest the user are given priority in the list.

[2138] Step 7: Proposal Presentation

[2139] Input: Product list sent from the server.

[2140] Data processing: Audio conversion of product list.

[2141] Output: Spoken product recommendations to the user.

[2142] Specific operation: The device will suggest to the user using a synthesized voice, "This smartphone has excellent camera performance. Its high resolution makes it particularly good for night photography."

[2143] Step 8: Collect user responses

[2144] Input: User's follow-up question (e.g., "That sounds good. Are there any other options I can choose?").

[2145] Data processing: The follow-up questions are converted into text data using voice recognition technology. The emotion data is then analyzed again using an emotion recognition engine.

[2146] Output: A new request to the server and sending emotion data.

[2147] Specific operation: The device performs voice recognition of the follow-up question, re-determines the user's current emotional state using the emotion recognition engine, and sends the analysis results to the server.

[2148] Step 9: Generate and present re-proposals

[2149] Input: A new product list from the server.

[2150] Data processing: Audio generation of new product listings.

[2151] Output: A spoken re-suggestion to the user.

[2152] Specific operation: The device makes a re-suggestion by synthesizing voice, saying, "This smartphone also has a high-performance camera, but it's more reasonably priced."

[2153] Prompt Sentence Examples

[2154] Example prompt: "User is looking for a new smartphone. Suggest a smartphone that focuses on camera performance, and if they inquire about price, suggest a more affordable option."

[2155] (Application example 2)

[2156] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2157] Conventional shopping support systems have difficulty accurately gathering user needs and suggesting optimal products. Furthermore, because they do not take the user's emotions into consideration, the content of the suggestions is unlikely to capture the user's interest and fail to stimulate purchasing motivation. Furthermore, there is a lack of personalized suggestions linked to past request data, making it impossible to achieve optimization for each user. There is a need to solve these problems and provide a more effective customer experience.

[2158] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[2159] In this invention, the server includes means for detecting the approach of a user, means for initiating a conversation with the user and collecting the user's needs, means for transmitting the collected user needs to the server, means for the server to select optimal products based on the user's needs and generate a proposal list, means for presenting the generated proposal list to the user, means for recollecting the user's reactions and transmitting the data to the server to optimize the next proposal, means for recognizing the user's emotions in real time using emotion recognition technology and transmitting the data to the server, and means for optimizing the proposal content based on the user's emotion data. This makes it possible to make personalized proposals according to the user's needs and improve the customer experience.

[2160] "Means for detecting a user's approach" refers to technology or devices that use cameras or sensors to detect a user's approach to the system.

[2161] "Means for starting a conversation with a user and gathering information about the user's needs" refers to technology or devices that use voice recognition or natural language processing to engage in a dialogue with a user and gather information about the user's requests and wishes.

[2162] The "means for transmitting collected user needs to a server" refers to a communication technology or device for transferring the user needs collected by a terminal to a server as data.

[2163] "Means for the server to select optimal products based on the user's needs and generate a proposal list" refers to technology or software that allows the server to analyze the user's needs, select optimal products based on those needs, and create a list.

[2164] The "means for presenting the generated proposal list to the user" refers to a technique or device for visually or audibly presenting the generated proposal list to the user.

[2165] "Means for re-collecting user responses and sending them to a server to optimize the next proposal" refers to technology or devices for re-collecting user responses to a proposal, sending them to a server, and optimizing the content of the next proposal based on that data.

[2166] "Means for recognizing a user's emotions in real time using emotion recognition technology and transmitting that data to a server" refers to technology or devices that analyze a user's facial expressions and tone of voice to recognize emotions and transmit the results to a server in real time.

[2167] "Means for optimizing proposal content based on user emotion data" refers to technology or devices for optimizing the proposed products or presentation content by taking into account the recognized user emotion data.

[2168] This invention is a smart shopping assistant system that interacts with customers in physical stores and makes product recommendations. This system consists of the following main components:

[2169] Components

[2170] Hardware

[2171] 1. Sales assistant robot: A device that directly interacts with users and gathers their needs. It may use built-in cameras, microphones, and sensors to detect the user's approach and identify them using facial recognition technology.

[2172] 2. Built-in camera and microphone: Captures the user's facial expressions and voice, providing data for emotion recognition technology.

[2173] 3. Approach detection sensor: Sensor technology that detects the approach of a user.

[2174] software

[2175] 1. Emotion recognition engine: Technology that analyzes the user's facial expressions and vocal tone in real time to recognize their emotional state. Utilizing technologies such as "Affectiva" and "Microsoft Azure Emotion API."

[2176] 2. Speech recognition system: A technology that converts user speech into text data. Services such as "Google Cloud Speech-to-Text" and "Amazon Transcribe" are used.

[2177] 3. Database management systems: Used to store data on user needs and emotions and make optimal product recommendations. Examples include "MySQL" and "MongoDB."

[2178] 4. Natural language processing engine: A technology used to analyze collected data and make product suggestions based on user needs. It uses technologies such as "GPT-4 by OpenAI" and "BERT."

[2179] Operation explanation

[2180] Device behavior

[2181] 1. User approach detection: The sales assistant robot's built-in camera and sensors detect the user's approach and identify the user using facial recognition technology.

[2182] 2. Start of initial interaction: When the robot detects the user's approach, it greets them with "Hello! Is there anything I can help you with?"

[2183] 3. Gathering needs: If the user responds, "I'm looking for a new smartphone," the robot will ask for more specific needs, such as, "Are you looking for a smartphone with a good camera?"

[2184] 4. Emotion recognition: During this speech, the emotion recognition engine processes the user's facial expressions and tone of voice in real time and sends the emotional data to the server.

[2185] Server Operation

[2186] 1. Data reception and analysis: The server receives the user's needs and emotion data sent from the device and analyzes the data using a natural language processing engine.

[2187] 2. Generate and send a list of proposals: Search the database, generate an optimal product list based on the user's needs and emotional data, and send it to the terminal.

[2188] Handling suggestions and feedback

[2189] 1. Proposal presentation: The sales assistant robot makes a proposal to the user, saying, "This smartphone has excellent camera performance. It has high resolution and is particularly good at taking night shots."

[2190] 2. Collecting user responses: If the user asks, "That sounds good. Are there any other options?", the robot sends a new request and emotional data to the server.

[2191] 3. Generation and presentation of a new recommendation: The server generates a more optimal product list based on the new needs and emotional data and sends it to the device. The robot makes a new recommendation, saying, "This smartphone also has a high-performance camera, but it's more reasonably priced."

[2192] Examples and prompts

[2193] Examples:

[2194] The user approaches and the approach is detected.

[2195] Robot: "Hello! Is there anything you're looking for?"

[2196] User: "Hi. I'm looking for a new smartphone."

[2197] Example prompt sentence:

[2198] The user approaches and the approach is detected.

[2199] Robot: "Hello! Is there anything you're looking for?"

[2200] User: "Hi. I'm looking for a new smartphone."

[2201] Robot: "What features are you looking for in a smartphone?"

[2202] User: "I want a smartphone with a good camera."

[2203] The robot sends this request and emotional data to the server.

[2204] The server generates a list of optimal smartphones and sends it to the robot.

[2205] Robot: "The camera on this smartphone is very good. It's especially good at taking high-resolution photos at night."

[2206] User: "That sounds good. Are there any other options?"

[2207] The robot sends a new request and emotion data to the server and gets the suggestion list again.

[2208] Robot: "This smartphone also has a high-performance camera, but at a reasonable price."

[2209] In this way, the smart shopping assistant system can highly personalize the customer experience in physical stores and efficiently make optimal product recommendations.

[2210] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2211] Step 1:

[2212] User approach detection

[2213] Input: Real-time video data from cameras and sensors.

[2214] How it works: The sales assistant robot (terminal) monitors its surroundings using a built-in camera and proximity detection sensors, and detects when a user approaches. It uses facial recognition technology to determine whether the user is a known or new user.

[2215] Output: Information on the detection of the user's approach and the user ID identified as the face recognition result.

[2216] Step 2:

[2217] Starting the initial dialogue

[2218] Input: Information on the user's approach detection and user ID.

[2219] How it works: The device greets the user with a voice message saying, "Hello! Is there anything I can help you with?" and begins the conversation.

[2220] Output: A spoken response from the user.

[2221] Step 3:

[2222] Gathering needs

[2223] Input: A spoken response from the user.

[2224] How it works: The device uses a voice recognition system (Google Cloud Speech-to-Text) to convert the user's voice into text data, then asks, "What kind of smartphone are you looking for?"

[2225] Output: User needs information as converted text data.

[2226] Step 4:

[2227] emotion recognition

[2228] Input: User's voice response and video data.

[2229] How it works: The device uses an emotion recognition engine (Affectiva or Microsoft Azure Emotion API) to analyze the user's facial expressions and voice tone in real time and generate emotion data.

[2230] Output: User emotion data.

[2231] Step 5:

[2232] Sending data

[2233] Input: User needs information and emotion data.

[2234] Operation: The terminal sends the collected data to the server.

[2235] Output: User needs information and emotion data sent to the server.

[2236] Step 6:

[2237] Generate a list of suggestions

[2238] Input: User needs information and emotion data sent to the server.

[2239] How it works: The server analyzes the data using a natural language processing engine (GPT-4 by OpenAI, BERT, etc.), searches for relevant product information in a database management system (MySQL or MongoDB), generates an optimal product list, and prioritizes it taking into account sentiment data.

[2240] Output: The generated optimal product list.

[2241] Step 7:

[2242] Presenting the proposal

[2243] Input: Generated optimal product list.

[2244] What it does: The device recommends the product to the user, saying, "This smartphone has excellent camera performance. Its high resolution makes it particularly good for nighttime photography."

[2245] Output: Product information presented to the user.

[2246] Step 8:

[2247] Collecting user responses

[2248] Input: User response (voice, facial expression).

[2249] How it works: The device uses its voice recognition system and emotion recognition engine to again collect the user's additional requests and reactions and sends this data to the server.

[2250] Output: Newly collected user needs information and emotion data.

[2251] Step 9:

[2252] Generate and present re-proposals

[2253] Input: Newly collected user needs information and emotion data.

[2254] How it works: The server again analyzes the data using a natural language processing engine, searches the database, and generates a new list of suggestions. The list is then sent to the device, which then makes a new suggestion, saying, "This smartphone also has a high-performance camera, but it's more reasonably priced."

[2255] Output: Product information re-presented to the user.

[2256] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2257] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2258] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2259] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2260] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2261] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2262] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2263] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, ...

Claims

1. means for detecting the approach of a user; a means for initiating a conversation with the user and gathering information about the user's needs; means for transmitting the collected user needs to a server; A means for the server to select optimal products based on the user's needs and generate a proposal list; means for presenting the generated suggestion list to a user; A means to collect user responses again and send them to the server to optimize the next proposal; A system including:

2. 2. The system according to claim 1, further comprising means for converting a conversation with the user into text data using a voice recognition technique and transmitting the text data to the server.

3. The system according to claim 1 , further comprising means for recognizing a user's face and identifying the user by linking it with past request data.

Images