System

Abstract

To provide a system.SOLUTION: A system includes: first means that collects customer use information having information on a visit frequency, a stay time, and a purchased commodity of a customer; second means that generates a first prompt sentence for outputting a suggestion content to be suggested to the customer based on the collected customer use information; third means that generates the suggestion content by inputting the generated first prompt sentence to a generation AI model; and fourth means that displays the generated suggestion content on a tablet of a crew that provides a service to the customer.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including a description and related instruction sentence regarding the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-180282 A Summary of the Invention [Problem to be solved by the invention]

[0004] With the conventional system, there was a difference in the level of suggestions made by new and experienced crew members, resulting in inconsistent service quality for customers. [Means for solving the problem]

[0005] The system of the present invention collects customer usage information and shares the data. It then diagnoses and analyzes customers based on the collected customer usage information and the shared data, and displays products and conversations to be recommended based on the results of the diagnosis and analysis. This allows the same level of suggestions to be made to new and experienced crew members alike, and maintains a consistent quality of service to customers. [Brief description of the drawings]

[0006] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Diagram 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. FIG. [Diagram 3] FIG. 11 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Diagram 5] FIG. 13 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 13 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 13 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 2 is a sequence diagram showing a process flow of the data processing system in the first embodiment of the first form example. [Figure 12] FIG. 11 is a sequence diagram showing a process flow of the data processing system in the application example 1 of the first embodiment. [Figure 13] FIG. 11 is a sequence diagram showing a process flow of the data processing system in the second embodiment of the second embodiment. [Figure 14] FIG. 13 is a sequence diagram showing a process flow of the data processing system in application example 2 of embodiment 2. [Figure 15] FIG. 13 is a sequence diagram showing a process flow of a data processing system in a third embodiment of the present invention. [Figure 16]FIG. 13 is a sequence diagram showing a process flow of the data processing system in the third application example of the third embodiment. [Figure 17] FIG. 13 is a sequence diagram showing the flow of processing in the data processing system in the first embodiment of the first form example when an emotion engine is combined. [Figure 18] FIG. 13 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Form Example 1 when combined with an emotion engine. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0008] First, the terms used in the following description will be explained.

[0009] In the following embodiments, a signed processor (hereinafter simply referred to as a "processor") may be one arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be one type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (TENSOR PROCESSING UNIT (registered trademark)).

[0010] In the following embodiments, a signed RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by the processor.

[0011] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0012] In the following embodiments, a communication I / F (Interface) with a code is an interface including a communication processor and an antenna. The communication I / F controls communication between multiple computers. An example of a communication standard applied to the communication I / F is a wireless communication standard including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0013] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. In addition, in this specification, the same idea as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0014] [First embodiment]

[0015] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0016] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0017] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0018] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0019] The reception device 38 includes a touch panel 38A and a microphone 38B, and receives user input. The touch panel 38A detects contact with a pointer (e.g., a pen or a finger) to receive user input by the touch of the pointer. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0020] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (e.g., voice and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs voice according to instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a Complementary Metal-Oxide-Semiconductor (CMOS) image sensor or a Charge Coupled Device (CCD) image sensor.

[0021] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54.

[0022] FIG. 2 shows an example of main functions of the data processing device 12 and the smart device 14.

[0023] As shown in Fig. 2, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32. The specific process program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific process program 56 from the storage 32, and executes the read specific process program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific process program 56 executed on the RAM 30.

[0024] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0025] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores a reception output program 60. The reception output program 60 is used together with the specific processing program 56 by the data processing system 10. The processor 46 reads out the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0026] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0027] "Example 1"

[0028] The system of the present invention uses sensors and cameras installed in stores, or applications used by customers, as a means for collecting customer usage information. Through these means, information such as the frequency of customers' visits, the duration of their stay, and the products purchased is collected. In addition, as a means for data linkage, the collected customer usage information is sent to a cloud server, etc., and linked to an analysis system. The analysis system diagnoses and analyzes customers based on the collected customer usage information and data linkage, and determines the products and talks to be proposed based on the diagnosis and analysis results. The determined proposal content is then displayed on the store's display or the crew's tablet, etc. This allows proposals to be made at the same level by new and veteran crew members. As a specific example, when a new crew member proposes the most suitable product based on information such as the purchasing tendency and frequency of visits of a customer they are serving for the first time, by using this system, the new crew member can make proposals equivalent to those of a veteran crew member.

[0029] "Example 2"

[0030] The system of the present invention uses sensors and cameras installed in stores, or applications used by customers, as a means for collecting customer usage information. Through these means, information such as the frequency of customers' visits, the duration of their stay, and the products purchased is collected. In addition, as a means for data linkage, the collected customer usage information is sent to a cloud server, etc., and linked to an analysis system. The analysis system diagnoses and analyzes customers based on the collected customer usage information and data linkage, and determines the products and talks to be proposed based on the diagnosis and analysis results. The determined proposal content is then displayed on the store's display or the crew's tablet, etc. This allows proposals to be made at the same level by new and veteran crew members. As a specific example, when a new crew member proposes the most suitable product based on information such as the purchasing tendency and frequency of visits of a customer they are serving for the first time, by using this system, the new crew member can make proposals equivalent to those of a veteran crew member.

[0031] The process flow of each embodiment will be described below.

[0032] "Example 1"

[0033] Step 1: Using sensors and cameras installed in the store, or applications used by customers, collect information such as how often customers visit the store, how long they stay, and what items they purchase.

[0034] Step 2: The collected customer usage information is sent to a cloud server, etc., and linked to the analysis system.

[0035] Step 3: The analysis system diagnoses and analyzes customers based on the collected customer usage information and data linkages.

[0036] Step 4: Determine the products and conversations to be proposed based on the results of the diagnosis and analysis.

[0037] Step 5: The finalized proposal content is displayed on the store's display and on the crew's tablets, etc. This allows new and experienced crew members to make proposals at the same level.

[0038] "Example 2"

[0039] Step 1: Using sensors and cameras installed in the store, or applications used by customers, collect information such as how often customers visit the store, how long they stay, and what items they purchase.

[0040] Step 2: The collected customer usage information is sent to a cloud server, etc., and linked to the analysis system.

[0041] Step 3: The analysis system diagnoses and analyzes customers based on the collected customer usage information and data linkages.

[0042] Step 4: Determine the products and conversations to be proposed based on the results of the diagnosis and analysis.

[0043] Step 5: The finalized proposal content is displayed on the store's display and on the crew's tablets, etc. This allows new and experienced crew members to make proposals at the same level.

[0044] Example 1

[0045] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal".

[0046] With conventional customer support systems, it was difficult to effectively collect and analyze customer usage information and make appropriate product suggestions. In addition, there was often a difference in the quality of suggestions between new and experienced crew members, making it difficult to improve customer satisfaction.

[0047] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0048] In this invention, the server includes a means for collecting customer usage information, a means for transmitting the collected customer usage information to the cloud server, a means for preprocessing the customer usage information received from the cloud server, a means for analyzing the preprocessed customer usage information, a means for generating products and talks to be proposed based on the analysis results, and a means for displaying the generated proposal contents. This makes it possible to effectively collect and analyze customer usage information and to make appropriate product proposals at the same level to new and experienced crew members alike.

[0049] "Customer usage information" is information generated when a customer uses a store, such as the customer's frequency of visits to the store, length of stay, and purchased items.

[0050] "Means of collection" refers to devices and software for obtaining customer usage information using sensors, cameras, applications, etc.

[0051] A "cloud server" is a remote server for storing and managing data via the Internet.

[0052] A "preprocessing means" is a device or software that performs processing to convert the received data into a format suitable for analysis.

[0053] "Means of analysis" refers to devices or software that perform statistical analysis and machine learning on preprocessed data to derive customer behavior patterns and trends.

[0054] "Means for generating" refers to devices or software that automatically create products and conversations to be suggested based on the analysis results.

[0055] The "display means" refers to a device such as a display or tablet for visually displaying the generated proposal content.

[0056] The present invention is a system for collecting and analyzing customer usage information and making appropriate product suggestions. A specific embodiment of this system will be described below.

[0057] Collection of customer usage information

[0058] When a user visits a store, the terminal activates sensors and cameras installed in the store. Specific hardware used is network cameras and motion sensors. These devices collect information such as the frequency of the customer's visits to the store, the length of time they stay, and the products they purchase. Similar information is also collected from applications used by the user. The applications send the customer's purchase history and store visit history to a server.

[0059] Sending data to cloud server

[0060] Before sending the data collected by the terminal to the cloud server, the data is formatted. Specifically, the data is converted to JSON format. The formatted data is securely sent to the cloud server using the HTTPS protocol. A general cloud service is used as the cloud server.

[0061] Analyzing data

[0062] The server receives the data sent from the cloud server. The received data undergoes pre-processing, such as missing value completion and data normalization. The pre-processed data is analyzed using known data analysis tools. The analysis system derives customer behavior patterns and trends.

[0063] Proposal Generation

[0064] The server generates suggested products and chats based on the analysis results. For example, it identifies products that are likely to be purchased next based on the customer's purchase history. The generated suggestions are displayed on store displays and crew tablets.

[0065] View Proposals

[0066] The suggestions generated by the server are displayed on the store's display. Specific hardware used is digital signage and tablets. The suggestions are displayed on the display as "recommended products," and on the tablet it says, "Please suggest this product to this customer."

[0067] Examples

[0068] When a new crew member comes to the store to serve a customer for the first time, sensors and cameras detect the customer's arrival and collect their length of stay and purchase history. The device sends this data to a cloud server, which analyzes it. Based on the analysis results, the server generates a suggestion such as "Please suggest product A to this customer, as he or she has purchased it in the past," and displays it on the crew member's tablet. This allows new crew members to make suggestions on a par with those of veteran crew members.

[0069] Examples of prompt statements

[0070] "Based on how often customers visit the store and their purchase history, what products should we recommend to them next?"

[0071] The system enables stores to provide consistent, high-quality service to their customers.

[0072] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0073] Step 1:

[0074] When a user visits a store, the terminal activates sensors and cameras installed in the store. The sensors and cameras detect the customer's visit and collect information such as frequency of visits, length of stay, and purchased items. The input is the customer's actions and behavior, and the output is the collected customer usage information. In concrete terms, the camera recognizes the customer's face, and the sensor measures the length of stay.

[0075] Step 2:

[0076] Before sending the customer usage information collected by the terminal to the cloud server, the data is formatted. The input is the collected raw data, and the output is the formatted data. Specifically, the data is converted to JSON format and sent to the cloud server using the HTTPS protocol.

[0077] Step 3:

[0078] The server receives the data sent from the cloud server. The input is the data from the cloud server, and the output is the received data. In concrete terms, the server receives the data and stores it in a database.

[0079] Step 4:

[0080] The server preprocesses the data it receives. The input is the received data, and the output is the preprocessed data. Specific operations include filling in missing values ​​and normalizing the data.

[0081] Step 5:

[0082] The server analyzes the pre-processed data. The input is the pre-processed data, and the output is the analysis results. Specifically, the server analyzes the data using known techniques to derive customer behavior patterns and trends.

[0083] Step 6:

[0084] The server generates suggested products and tokens based on the analysis results. The input is the analysis results, and the output is the generated suggestions. Specifically, the server identifies the products that are likely to be purchased next based on the customer's purchase history.

[0085] Step 7:

[0086] The proposal content generated by the server is displayed on the store's display and on the crew's tablet. The input is the generated proposal content, and the output is the displayed proposal content. The specific operation is to display it as a "recommended product" on the display and to display "Please suggest this product to this customer" on the tablet.

[0087] (Application example 1)

[0088] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0089] In conventional customer service at stores, it was difficult for new staff to make optimal product recommendations based on the customer's purchasing habits and frequency of visits, resulting in a difference in the quality of service between new and experienced staff. In addition, there was a lack of tools to grasp customer information in real time and make appropriate recommendations.

[0090] The identification process by the identification processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means. In this invention, the server includes a means for collecting customer usage information, a means for data linking, a means for diagnosing and analyzing visiting customers based on the collected customer usage information and the data linking, a means for displaying products and talks to be proposed based on the diagnosis and analysis results, a means for displaying on the smart glasses, and a means for generating proposal contents using a generative AI model. This enables even a new crew member to make optimal product proposals to customers at the same level as a veteran crew member.

[0091] "Customer usage information" refers to information about a customer's behavior and preferences when using a store, such as the frequency of the customer's visit, the length of time they stay, and the products they purchase.

[0092] "Data integration" means sending collected customer usage information to an external system such as a cloud server and sharing the information with other systems.

[0093] "Diagnosis and analysis" refers to the process of evaluating customer behavior and preferences based on collected customer usage information and determining appropriate product suggestions and services.

[0094] "Products and conversations to be proposed" refers to products recommended to customers based on the results of diagnosis and analysis, as well as topics and explanations to be used in communication with customers.

[0095] "Smart glasses" are wearable devices that have a built-in display and provide users with visual information when worn.

[0096] A "generative AI model" is an algorithm or system that uses artificial intelligence technology to generate optimal suggestions based on input data.

[0097] The system for implementing this invention performs a series of processes to collect customer usage information, link the data, and display products and talks to be proposed based on the diagnosis and analysis results. Specifically, the following hardware and software are used.

[0098] Hardware

[0099] 1. Sensors and cameras: Installed in stores, they collect information such as how often customers visit, how long they stay, and what items they purchase.

[0100] 2. Cloud server: Stores collected customer usage information and connects to the analysis system.

[0101] 3. Smart glasses: Worn by crew members, they display real-time suggestions based on diagnostic and analysis results.

[0102] software

[0103] 1. Data collection module: Collects customer usage information from sensors and cameras and transmits it to the cloud server.

[0104] 2. Data linking module: Links data between the cloud server and the analysis system.

[0105] 3. Analysis system: Diagnosis and analysis are performed based on collected customer usage information, and proposals are generated using a generative AI model.

[0106] 4. Display module: displays the suggestions on the smart glasses.

[0107] Processing flow

[0108] 1. Data collection: Sensors and cameras collect information such as how often customers visit the store, how long they stay, and what items they purchase.

[0109] 2. Data transmission: The collected data is transmitted to the cloud server through the data collection module.

[0110] 3. Data linkage: The cloud server shares data with the analysis system via the data linkage module.

[0111] 4. Diagnosis and analysis: The analysis system performs diagnosis and analysis based on the collected data, and generates optimal proposals using a generative AI model.

[0112] 5. Displaying the Suggestions: The generated suggestions are displayed on the smart glasses through the display module.

[0113] Examples

[0114] For example, if the customer ID is "12345", the customer visits the store twice a week and the products they have purchased in the past are "coffee" and "sandwiches". Based on this information, the following prompt sentence is input to the generative AI model.

[0115] Example prompt:

[0116] Customer visit frequency: twice a week, purchases: coffee, sandwiches

[0117] Based on this prompt, the generative AI model generates suggestions such as the following:

[0118] Examples of suggestions:

[0119] "This customer comes in twice a week and buys a coffee and a sandwich. The next time they come, you could suggest a new seasonal drink or sandwich. You could also encourage them to use their loyalty card to encourage them to visit more frequently."

[0120] These suggestions are then displayed on the smart glasses' display, allowing the crew to make optimal product suggestions to the customer based on that information.

[0121] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0122] Step 1:

[0123] Sensors and cameras collect information such as how often customers visit the store, how long they stay, what products they purchase, etc. These devices monitor customers' behavior from the moment they enter the store and collect data in real time. The input is customer behavior data, and the output is collected customer usage information.

[0124] Step 2:

[0125] The data collection module sends the collected customer usage information to the cloud server. The data collection module organizes the data obtained from the sensors and cameras and uploads it to the cloud server. The input is the collected customer usage information, and the output is the data stored in the cloud server.

[0126] Step 3:

[0127] The cloud server shares data with the analysis system via the data linking module. The cloud server transmits collected customer usage information to the analysis system and links the data. The input is the data stored in the cloud server, and the output is the data transmitted to the analysis system.

[0128] Step 4:

[0129] The analysis system performs diagnosis and analysis based on the data collected. The analysis system analyzes information such as the frequency of customer visits to the store and the products purchased, and generates optimal proposals using a generative AI model. The input is the data sent to the analysis system, and the output is the generated proposals.

[0130] Step 5:

[0131] The generative AI model generates suggestions based on the prompt text. For example, the generative AI model generates appropriate suggestions based on the prompt text "Customer visit frequency: twice a week, Purchased items: coffee, sandwiches." The input is the prompt text, and the output is the generated suggestions.

[0132] Step 6:

[0133] The display module displays the generated proposal on the smart glasses. The display module displays the proposal obtained from the generative AI model on the display of the smart glasses for the crew to review in real time. The input is the generated proposal, and the output is the information displayed on the smart glasses.

[0134] Step 7:

[0135] The crew makes optimal product proposals to the customer based on the proposals displayed on the smart glasses. The crew uses the information displayed on the smart glasses to propose appropriate products and services to the customer. The input is the information displayed on the smart glasses, and the output is the proposals to the customer.

[0136] Example 2

[0137] Next, a second embodiment of the second embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0138] With conventional customer support systems, it was difficult to effectively collect and analyze customer usage information and make optimal product proposals. It was also difficult for new crew members to make proposals to customers at the same level as veteran crew members. This resulted in issues such as insufficient improvement of customer satisfaction and sales promotion.

[0139] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0140] In this invention, the server includes a means for collecting customer usage information, a means for transmitting the collected customer usage information to a cloud server, a means for linking the customer usage information transmitted to the cloud server to an analysis system, a means for diagnosing and analyzing customers based on the collected customer usage information and data linkage, a means for determining products and conversations to be proposed based on the diagnosis and analysis results using a generative AI model, and a means for displaying the determined proposal content on a store display or a crew tablet. This makes it possible to effectively collect and analyze customer usage information and make optimal product proposals. In addition, even new crew members can make proposals to customers at the same level as veteran crew members, which is expected to improve customer satisfaction and promote sales.

[0141] "Customer usage information" refers to information such as the frequency of a customer's visits to a store, the length of their stay, and the items purchased.

[0142] A "cloud server" refers to a remote server that stores and processes data over the Internet.

[0143] An "analysis system" refers to a system that analyzes collected data and derives customer behavior patterns and trends.

[0144] A "generative AI model" refers to an algorithm that uses artificial intelligence to generate optimal suggestions or conversations from data.

[0145] "Display" refers to a display device for visually displaying information.

[0146] "Tablet" refers to a portable, touch-screen computing device.

[0147] "Proposal content" refers to the content of the products or talk to be proposed to the customer.

[0148] "Diagnosis and Analysis" refers to the process of evaluating customer behavior and trends based on collected Customer Usage Information.

[0149] The system of the present invention is for collecting and analyzing customer usage information and making optimal product proposals. A specific embodiment of this system will be described below.

[0150] Data collection

[0151] The terminals use sensors and cameras installed in the store, or applications used by customers, to collect information such as the frequency of customers' visits, length of stay, and purchased items. Specific hardware used is a "general-purpose sensor" for the sensor, a "general-purpose camera" for the camera, and a "store-specific application" for the application. These terminals record customer behavior in real time and collect data.

[0152] Data transmission

[0153] The terminal transmits the collected customer usage information to a cloud server. A "general-purpose cloud service" is used as the cloud server. The secure communication protocol "HTTPS" is used for data transmission. This allows the collected data to be safely stored on the cloud server.

[0154] Data analysis

[0155] The server receives the data sent to the cloud server and connects to the analysis system. The analysis system uses a "general-purpose AI platform." The server diagnoses and analyzes customers based on the collected customer usage information. Specifically, it analyzes data such as the frequency of customers' visits, length of stay, and purchased items to derive customers' purchasing patterns and store visit tendencies.

[0156] Deciding on the proposal content

[0157] The server decides which products and conversations to suggest based on the analysis results. As for the specific software, a "general-purpose generative AI model" is used as the generative AI model. The server inputs a prompt text into the generative AI model and generates optimal product suggestions and conversations. For example, the following is used as a prompt text:

[0158] "Customer ID: 12345 visits the store twice a week, stays for an average of 30 minutes, and recently purchased coffee and sandwiches. Please generate a conversation that will provide optimal product suggestions to this customer."

[0159] View Proposals

[0160] The server displays the determined proposals on the store's displays and on the crew's tablets. A "general-purpose display" is used for the display and a "general-purpose tablet" is used for the tablet. This allows the crew to make optimal product proposals to customers.

[0161] Examples

[0162] By using this system, new crew members can make suggestions on par with those made by veteran crew members when suggesting optimal products to customers they are serving for the first time, based on their purchasing habits and frequency of visits. For example, if a customer frequently purchases coffee, the system will generate a suggestion such as "Would you like to try a new coffee flavor?" and display it on the crew member's tablet.

[0163] The above is a specific embodiment for implementing the system of the present invention. This system makes it possible to effectively collect and analyze customer usage information and make optimal product proposals. In addition, even new crew members can make proposals to customers at the same level as veteran crew members, which is expected to improve customer satisfaction and promote sales.

[0164] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0165] Step 1: Data collection

[0166] The terminal uses sensors and cameras installed in the store, or an application used by the customer, to collect information such as the frequency of customer visits, length of stay, and purchased items. In concrete terms, the sensors detect when customers enter and leave the store, and the cameras record the length of time the customer stays. The application saves the customer's purchase history. The input is customer behavior data, and the output is collected customer usage information.

[0167] Step 2: Send data

[0168] The terminal transmits the collected customer usage information to the cloud server. Specifically, the terminal transmits the collected data to the cloud server using the HTTPS protocol. The input is the collected customer usage information, and the output is the data stored in the cloud server.

[0169] Step 3: Data analysis

[0170] The server receives the data sent to the cloud server and connects it to the analysis system. Specifically, the server uses the AI ​​platform to analyze data such as the frequency of customer visits, length of stay, and purchased items. The input is customer usage information stored in the cloud server, and the output is the analysis results.

[0171] Step 4: Decide on the proposal

[0172] The server decides which products and conversations to suggest based on the analysis results. Specifically, the server inputs a prompt text into the generative AI model to generate optimal product suggestions and conversations. The inputs are the analysis results and the prompt text, and the output is the generated suggestions. For example, the prompt text could be "Customer ID: 12345 visits the store twice a week, spends an average of 30 minutes, and most recently purchased coffee and sandwiches. Please generate a conversation to suggest optimal products to this customer."

[0173] Step 5: View the proposal

[0174] The server displays the determined proposal content on the store's display or on the crew's tablet. In concrete terms, the server sends the generated proposal content to the display or tablet and displays it in real time. The input is the generated proposal content, and the output is the proposal content displayed on the display or tablet. This enables the crew to make optimal product proposals to the customer.

[0175] (Application example 2)

[0176] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0177] Customer service in conventional stores was heavily dependent on the experience and skills of staff, making it difficult for new staff to provide the same level of service as veteran staff. In addition, there was a lack of effective means to utilize information such as customers' purchase history and frequency of visits to the store, making it difficult to propose optimal products and services to customers. Furthermore, the inability to analyze and propose customer information in real time made improving customer satisfaction an issue.

[0178] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0179] In this invention, the server includes a means for collecting customer usage information, a means for data linking, a means for diagnosing and analyzing customers visiting the store based on the collected customer usage information and the data linking, a means for generating proposal contents using a generative AI model, and a means for displaying the generated proposal contents on the mobile terminal of the store staff. This allows even new staff to propose optimal products and services to customers at the same level as veteran staff. In addition, since customer information can be analyzed and proposed in real time, it is expected that customer satisfaction will improve.

[0180] "Customer usage information" refers to information such as the frequency of a customer's visit to a store, the length of time they stay, and the products they purchase.

[0181] "Data integration" means sending collected customer usage information to a cloud server or the like and sharing the information with other systems.

[0182] "Diagnosis and analysis" refers to the process of evaluating customer behavior and trends based on collected customer usage information and making appropriate suggestions.

[0183] The "products and conversations to be proposed" refer to the products and services recommended to the customer, as well as the related explanations and conversation content.

[0184] A "generative AI model" is an algorithm or system that uses artificial intelligence to analyze data and generate optimal suggestions.

[0185] A "mobile terminal" is a device that can be carried by store staff, including a smartphone or tablet.

[0186] "Real-time" refers to data collection, analysis, and recommendations being made immediately.

[0187] "Store staff" refers to employees who deal with customers in stores.

[0188] "New staff" refers to employees who have little experience working in a store.

[0189] "Veteran staff" refers to employees with extensive experience working in stores.

[0190] The system for implementing this invention collects customer usage information, links the data, performs diagnosis and analysis based on the collected information, generates proposal content using a generative AI model, and displays the proposal content on the mobile devices of store staff.

[0191] To collect customer usage information, the server uses sensors and cameras installed in the store, or applications used by customers, to collect information such as the frequency of customers' visits, length of stay, and purchased items. The collected information is sent to a cloud server, where it is shared with other systems.

[0192] The server performs diagnosis and analysis based on the collected customer usage information. For this analysis, for example, the TENSORFLOW (registered trademark) AI model is used. The AI ​​model analyzes data such as the customer's purchase history and frequency of visits to the store, and predicts the product that the customer is likely to purchase next.

[0193] The generative AI model generates proposals based on the analysis results. The generated proposals are displayed as prompts on the store staff's mobile devices. This enables even new staff to propose optimal products and services to customers at the same level as experienced staff.

[0194] As a concrete example, the following prompt sentence is input to the generative AI model:

[0195] "Customer ID: 12345

[0196] Visit frequency: Once a week

[0197] Purchase history: [Product A, Product B, Product C]

[0198] Duration: Average 30 minutes

[0199] Based on this prompt, the AI ​​model generates the following suggestions:

[0200] "The product we recommend to you is Product D. Since you have previously purchased Products A, B, and C, we think Product D will also suit your taste."

[0201] In this way, store staff can recommend the most suitable products to customers.

[0202] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0203] Step 1:

[0204] The server collects customer usage information from sensors and cameras installed in the store or through applications used by customers. Specifically, it obtains data such as the frequency of customers' visits to the store, the length of their stay, and the products they purchase. The input is real-time data from the sensors and cameras, and the output is the collected customer usage information.

[0205] Step 2:

[0206] The server sends the collected customer usage information to a cloud server and shares data with other systems. Specifically, the collected data is stored in a database on the cloud and shared with other analysis systems as necessary. The input is the collected customer usage information, and the output is the data stored in the cloud server.

[0207] Step 3:

[0208] The server performs diagnosis and analysis based on the customer usage information stored in the cloud server. Specifically, it uses an AI model (using TensorFlow, for example) to analyze customer behavior patterns and purchasing trends. The input is the customer usage information stored in the cloud server, and the output is the diagnosis and analysis results.

[0209] Step 4:

[0210] The server uses a generative AI model based on the diagnosis and analysis results to generate proposals. Specifically, the AI ​​model generates prompts that recommend optimal products and services to customers based on the analysis results. The input is the diagnosis and analysis results, and the output is the generated prompts.

[0211] Step 5:

[0212] The server sends the generated prompt text to the mobile device of the store staff. Specifically, the generated suggestion content is displayed in real time on the staff's smartphone or tablet. The input is the generated prompt text, and the output is the suggestion content displayed on the mobile device.

[0213] Step 6:

[0214] The terminal proposes products and services to the customer based on the received proposal content. Specifically, the staff member refers to the prompt displayed on the mobile terminal and introduces appropriate products and services to the customer. The input is the proposal content displayed on the mobile terminal, and the output is the proposal to the customer.

[0215] Furthermore, an emotion engine that estimates the emotion of the user may be combined. That is, the identification processing unit 290 may estimate the emotion of the user using the emotion identification model 59, and perform identification processing using the emotion of the user.

[0216] "Example 1"

[0217] In one embodiment of the present invention, an emotion engine is incorporated. The emotion engine recognizes emotions from the user's facial expressions, tone of voice, choice of words, and the like. Specifically, when a user visits a store, the user's facial expressions and voice are captured through cameras and microphones installed in the store, and the emotion engine analyzes this information. The emotion engine can recognize the user's emotions, such as joy, anger, surprise, and sadness, and provide services according to those emotions. For example, if it is recognized that the user is happy to see a product, it will suggest products and services related to that product. Also, if it is recognized that the user is in trouble, it will provide appropriate assistance. This makes it possible to further improve the user's satisfaction.

[0218] "Example 2"

[0219] In one embodiment of the present invention, an emotion engine is incorporated. The emotion engine recognizes emotions from the user's facial expressions, tone of voice, choice of words, and the like. Specifically, when a user visits a store, the user's facial expressions and voice are captured through cameras and microphones installed in the store, and the emotion engine analyzes this information. The emotion engine can recognize the user's emotions, such as joy, anger, surprise, and sadness, and provide services according to those emotions. For example, if it is recognized that the user is happy to see a product, it will suggest products and services related to that product. Also, if it is recognized that the user is in trouble, it will provide appropriate assistance. This makes it possible to further improve the user's satisfaction.

[0220] The process flow of each embodiment will be described below.

[0221] "Example 1"

[0222] Step 1: The user visits the store. Cameras and microphones installed in the store capture the user's facial expressions and voice.

[0223] Step 2: The captured information is sent to the emotion engine, which recognizes emotions from the user's facial expressions, tone of voice, choice of words, etc.

[0224] Step 3: The emotion engine decides which products and services to suggest based on the recognized emotion. For example, if it recognizes that the user is happy to see a product, it will suggest products and services related to that product. Also, if it recognizes that the user is in trouble, it will provide appropriate assistance.

[0225] Step 4: Information about the proposed products and services is communicated to the store staff, who then make the proposal to the user. This makes it possible to further improve user satisfaction.

[0226] "Example 2"

[0227] Step 1: The user visits the store. Cameras and microphones installed in the store capture the user's facial expressions and voice.

[0228] Step 2: The captured information is sent to the emotion engine, which recognizes emotions from the user's facial expressions, tone of voice, choice of words, etc.

[0229] Step 3: The emotion engine decides which products and services to suggest based on the recognized emotion. For example, if it recognizes that the user is happy to see a product, it will suggest products and services related to that product. Also, if it recognizes that the user is in trouble, it will provide appropriate assistance.

[0230] Step 4: Information about the proposed products and services is communicated to the store staff, who then make the proposal to the user. This makes it possible to further improve user satisfaction.

[0231] Example 1

[0232] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal".

[0233] With conventional customer service systems, it was difficult to effectively collect and analyze customer usage information and make appropriate suggestions. In addition, because it depended on the experience and skills of the crew, it was difficult for a new crew member to make the same suggestions as a veteran crew member. Furthermore, it was not possible to provide services that took into account the customer's feelings, making it difficult to improve customer satisfaction.

[0234] The identification process by the identification processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means. In this invention, the server includes a means for collecting customer usage information, a means for transmitting the collected customer usage information to a cloud server, a means for analyzing the customer usage information in cooperation with the cloud server, a means for determining products and conversations to be proposed based on the analysis results, a means for displaying the determined proposal contents, a means for recognizing the user's facial expression and voice and analyzing emotions, and a means for providing appropriate services based on the emotion analysis results. This makes it possible to effectively collect and analyze customer usage information and make appropriate suggestions. In addition, even a new crew member can make suggestions equivalent to those of a veteran crew member, and it becomes possible to provide services that take into account the emotions of the customer.

[0235] "Customer usage information" is data generated when a customer uses a store, such as the customer's frequency of visits, length of stay, and purchased items.

[0236] "Means of collection" refers to devices and software for obtaining customer usage information using sensors, cameras, applications, etc.

[0237] A "cloud server" is a remote server that stores and manages data via the Internet and connects with other systems.

[0238] "Analytical tools" are software and algorithms used to process and analyze collected data and understand customer behavior and trends.

[0239] The "means of determining the products and conversations to be proposed" refers to a system for selecting the most suitable products and services for each customer based on the analysis results and generating proposal content.

[0240] The "display means" refers to a device such as a display or tablet for visually displaying the determined proposal content.

[0241] "Means for analyzing emotions" refers to software or algorithms that analyze a user's facial expressions and tone of voice to recognize emotions.

[0242] The "means for providing appropriate services" is a system for providing optimal responses and services to users based on the results of sentiment analysis.

[0243] The present invention is a system for effectively collecting and analyzing customer usage information and making appropriate proposals. A specific embodiment of this system will be described below.

[0244] The server collects customer usage information, such as the frequency of visits, length of stay, and purchased items, from sensors and cameras installed in the store, or through an application used by the customer. The sensors detect the moment a customer enters the store, and the cameras track the customer's movements in real time. The application automatically records data on the items purchased by the customer.

[0245] The collected customer usage information is sent from the server to a cloud server. The cloud server securely stores this data and connects it to the analysis system. Specifically, the data is sent and stored using the cloud infrastructure of Amazon Web Services (AWS (registered trademark)).

[0246] The server diagnoses and analyzes customers based on the customer usage information collected using the analysis system. It uses known techniques to analyze the frequency of customers' visits and purchasing trends. For example, it formats and cleanses the data and models the frequency of customers' visits and purchasing trends.

[0247] Based on the analysis results, the server decides which products and conversations should be proposed. The decided proposals are displayed on the store's displays and crew members' tablets. Specifically, the proposals are displayed using iPads (registered trademarks) and ANDROID (registered trademark) tablets. The proposals generated by the server are sent to the store's displays and crew members' tablets, and the devices display the proposals.

[0248] Furthermore, the server uses an emotion engine to recognize emotions from the user's facial expressions, tone of voice, choice of words, etc. The emotion engine analyzes the user's facial expressions and voice captured by cameras and microphones installed in the store. Specifically, it uses Microsoft® Azure® emotion recognition API to recognize the user's emotions such as joy, anger, surprise, and sadness.

[0249] The server provides appropriate services depending on the user's emotions recognized by the emotion engine. For example, if it recognizes that the user is happy when looking at a product, it will suggest products or services related to that product. If it recognizes that the user is in trouble, it will provide appropriate assistance. The server determines the appropriate service based on the emotion recognition results and sends it to the store's display or the crew's tablet. The terminal displays the service details, and the crew responds to the user.

[0250] As a concrete example, when a new staff member is serving a customer for the first time, and is trying to recommend the best product based on the customer's purchasing habits and frequency of visits, this system will enable the new staff member to make the same suggestions as a veteran staff member. For example, the system can suggest related products based on the customer's past purchase data.

[0251] An example of a prompt sentence is, "Please tell us how to suggest the best products to customers based on their frequency of visits to the store and their purchasing tendencies."

[0252] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0253] Step 1: Collect customer usage information

[0254] The server collects information such as the frequency of customers' visits, length of stay, and purchased items from sensors and cameras installed in the store, or from applications used by customers. The sensors detect the moment a customer enters the store, and the cameras track the customer's movements in real time. The application automatically records data on the items purchased by the customer. The input is data from the sensors, cameras, and applications, and the output is the collected customer usage information.

[0255] Specific behavior:

[0256] The sensor detects when a customer enters the store and notifies the server.

[0257] The cameras track customer movements and transmit video data to a server.

[0258] The application records the purchase data and sends it to the server.

[0259] Step 2: Send and link data

[0260] The server transmits the collected customer usage information to the cloud server. The cloud server securely stores the data and connects it to the analysis system. The input is the collected customer usage information, and the output is the data stored in the cloud server.

[0261] Specific behavior:

[0262] The server transmits the collected data to a cloud server.

[0263] The cloud server receives and stores the data.

[0264] Step 3: Analyze customer information

[0265] The server diagnoses and analyzes customers based on the customer usage information collected using the analysis system. It uses known technology to analyze the frequency of customers' visits and their purchasing trends. The input is the data stored in the cloud server, and the output is the analysis results.

[0266] Specific behavior:

[0267] The server formats and cleanses the data.

[0268] The server models customers' frequency of visits and purchasing habits.

[0269] Step 4: Decide and present your proposal

[0270] The server decides what products and conversations to suggest based on the diagnosis and analysis results obtained from the analysis system. The proposed content is displayed on the store's display and on the crew's tablets. The input is the analysis results, and the output is the proposed content.

[0271] Specific behavior:

[0272] The server generates suggestions based on the analysis results.

[0273] The suggestions generated by the server are sent to store displays and crew tablets.

[0274] The device will display the suggestions.

[0275] Step 5: Use the Emotion Engine

[0276] The server uses an emotion engine to recognize emotions from the user's facial expressions, tone of voice, choice of words, etc. The emotion engine analyzes the user's facial expressions and voice captured by cameras and microphones installed in the store. The input is data from the cameras and microphones, and the output is the emotion analysis results.

[0277] Specific behavior:

[0278] The camera captures the user's facial expressions and transmits them to the server.

[0279] The microphone records the user's voice and transmits it to the server.

[0280] The server calls the emotion recognition API and analyzes the user's emotions.

[0281] Step 6: Deliver emotional service

[0282] The server provides appropriate services according to the user's emotions recognized by the emotion engine. For example, if it recognizes that the user is happy when looking at a product, it will suggest products and services related to that product. If it recognizes that the user is in trouble, it will provide appropriate assistance. The input is the emotion analysis results, and the output is the service content to be provided.

[0283] Specific behavior:

[0284] The server determines the appropriate service based on the emotion recognition results.

[0285] The server transmits the determined service content to the store's display and crew members' tablets.

[0286] The terminal displays the service details, and the crew responds to the user.

[0287] (Application example 1)

[0288] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0289] Conventional customer service systems can collect and analyze customer usage information, but it is difficult to recognize customer emotions in real time and provide optimal product suggestions and services based on that. In addition, it is difficult for new crew members to provide customer service at the same level as veteran crew members, making it difficult to improve customer satisfaction.

[0290] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0291] In this invention, the server includes a means for collecting customer usage information, a means for linking data, a means for diagnosing and analyzing customers based on the collected customer usage information and the data linking, a means for displaying products and conversations to be proposed based on the results of the diagnosis and analysis, a means for recognizing the customer's facial expression and tone of voice and analyzing their emotions, and a means for determining and displaying the contents of proposals based on the results of the emotion analysis. This makes it possible to provide optimal product proposals and services in real time according to the customer's emotions, and enables even new crew members to handle customers at the same level as veteran crew members.

[0292] "Customer usage information" refers to information such as the frequency of a customer's visit to a store, the length of time they stay, and the products they purchase.

[0293] "Data integration" means sending collected customer usage information to a cloud server or the like and sharing the information with other systems.

[0294] "Diagnosis and analysis" refers to the process of analyzing the behavior and trends of customers visiting the store based on collected customer usage information and proposing appropriate products and services.

[0295] "Means of display" refers to devices or methods for displaying products and conversations to be suggested based on the results of diagnosis and analysis on store displays or staff tablets, etc.

[0296] The "means for recognizing facial expressions and tone of voice" refers to a device or method for capturing a customer's facial expressions and tone of voice using a camera or microphone and analyzing them.

[0297] The "means for analyzing emotions" refers to a device or method for analyzing the emotions of a customer from recognized facial expressions and tone of voice, and identifying emotions such as joy, anger, surprise, sadness, etc.

[0298] The "means for determining the content of proposals based on the results of emotion analysis" refers to a device or method for determining the optimal product or service proposal content according to the analyzed emotion.

[0299] The system for implementing this invention collects customer usage information, links the data, and diagnoses and analyzes customers based on the collected information.Furthermore, it displays products and conversations to be suggested based on the results of the diagnosis and analysis, recognizes the customer's facial expression and tone of voice to analyze their emotions, and determines the content of the suggestions based on those emotions.

[0300] Hardware and Software Used

[0301] Hardware:

[0302] Smart glasses (e.g. Google® Glass®)

[0303] camera

[0304] microphone

[0305] software:

[0306] Image Processing Library

[0307] Emotion Recognition Library

[0308] Customer Data Management Library

[0309] Cloud server integration library

[0310] System processing overview

[0311] The server first collects customer usage information, including the frequency of customers' visits, the length of their stay, and the products they purchase. This information is collected from sensors and cameras installed in the store, or through applications used by customers. The collected information is sent to a cloud server, where it is shared with other systems.

[0312] Next, the server diagnoses and analyzes the customer based on the collected customer usage information. Based on the results of the diagnosis and analysis, it decides which products and conversations to recommend, and displays them on the store's displays and on staff tablets. This allows even new staff members to serve customers at the same level as veteran staff members.

[0313] In addition, the server recognizes the customer's facial expressions and tone of voice and analyzes their emotions. To do this, the server uses a camera and microphone to capture the customer's facial expressions and voice, and analyzes them using an emotion recognition engine. Based on the analyzed emotions, the server determines and displays the most suitable product suggestions and services.

[0314] Examples

[0315] For example, when a customer visits a store and a staff member wearing smart glasses captures the customer's facial expression, if the staff member recognizes that the customer looks "happy," the store will suggest new products related to those that the customer previously purchased.

[0316] Examples of prompt statements

[0317] Customer ID: 12345

[0318] Customer facial expressions: Delighted

[0319] Recommendation: We also recommend this product!

[0320] In this way, it becomes possible to provide optimal product suggestions and services in real time according to the customer's emotions, thereby improving customer satisfaction.

[0321] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0322] Step 1:

[0323] The server collects customer usage information, such as the frequency of customer visits, length of stay, and purchased items, from sensors and cameras installed in the store or through applications used by customers. The input is data from the sensors and cameras, and the output is the collected customer usage information.

[0324] Step 2:

[0325] The server sends the collected customer usage information to the cloud server and shares data with other systems. The input is the collected customer usage information, and the output is the data stored in the cloud server.

[0326] Step 3:

[0327] The server diagnoses and analyzes customers who visit the store based on the customer usage information stored in the cloud server. The input is the customer usage information obtained from the cloud server, and the output is the diagnosis and analysis results. Specifically, the server uses a data analysis algorithm to analyze customer behavior and trends.

[0328] Step 4:

[0329] The server determines the products and conversations to be proposed based on the diagnosis and analysis results, and displays them on the store's displays and staff tablets. The input is the diagnosis and analysis results, and the output is the displayed proposal content. In concrete terms, the proposal content is generated using a generative AI model and sent to the display or tablet.

[0330] Step 5:

[0331] The terminal (a staff member wearing smart glasses) captures the customer's facial expressions and voice using a camera and microphone. The input is the data from the camera and microphone, and the output is the captured facial expression and voice data.

[0332] Step 6:

[0333] The server uses the captured facial and voice data to analyze the customer's emotions with an emotion recognition engine. The input is the captured facial and voice data, and the output is the analyzed emotion. The specific operation is to analyze facial expressions and tone of voice using an emotion recognition algorithm.

[0334] Step 7:

[0335] The server determines optimal product suggestions and services based on the analyzed emotions and displays them on the smart glasses. The input is the analyzed emotions, and the output is the displayed suggestions. Specifically, the server uses a generative AI model to generate suggestions based on the emotions and sends them to the smart glasses.

[0336] Step 8:

[0337] The terminal (a staff member wearing smart glasses) responds to the customer based on the displayed proposal. The input is the proposal displayed on the smart glasses, and the output is the response to the customer. In concrete terms, the staff member conveys the proposal to the customer.

[0338] Example 2

[0339] Next, a second embodiment of the second embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0340] Conventional customer service systems could collect and analyze customer usage information, but they could not recognize customer emotions and provide services based on them. This made it difficult to respond appropriately to improve customer satisfaction. It was also difficult for new crew members to make suggestions to customers at the same level as veteran crew members.

[0341] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0342] In this invention, the server includes a means for collecting customer usage information, a means for linking data, a means for diagnosing and analyzing customers based on the collected customer usage information and the data linking, a means for displaying products and conversations to be proposed based on the results of the diagnosis and analysis, a means for recognizing the user's facial expressions and voice and analyzing emotions, and a means for providing services based on emotions. This makes it possible to provide services according to the customer's emotions, improving customer satisfaction and enabling new crew members to make suggestions equivalent to those of veteran crew members.

[0343] "Customer usage information" is data generated when a customer uses a store, such as the customer's frequency of visits, length of stay, and purchased items.

[0344] "Data integration" is the process of sending collected customer usage information to an external system such as a cloud server and sharing the information with other systems.

[0345] "Diagnosis and analysis" is the process of analyzing customer behavior and trends based on collected customer usage information and deriving appropriate proposals and services.

[0346] "Products and conversations to be proposed" refer to products recommended to customers based on the results of diagnosis and analysis, as well as topics and phrases to be used in communication with customers.

[0347] "Means of display" refers to devices and software for showing suggested products and conversations on store displays, crew tablets, etc.

[0348] "Means for recognizing a user's facial expressions and voice and analyzing emotions" refers to technology that uses a camera and microphone to capture a user's facial expressions and voice, and analyzes them to identify the user's emotions.

[0349] A "means for providing services based on emotions" is a process or system for providing appropriate product suggestions and assistance based on the results of user emotion analysis.

[0350] The present invention is a system that collects and analyzes customer usage information and makes appropriate suggestions. This system collects information such as the frequency of customers' visits to a store, the length of their stay, and the products they purchase, and uses this information to analyze customer behavior and trends. It can also provide services based on emotions by recognizing the user's facial expressions and voice and analyzing their emotions.

[0351] Collection of customer usage information

[0352] The terminals collect information such as the frequency of customers' visits, the length of their stay, and the products they purchase through sensors and cameras installed in the store, or through an application used by the customer. Specifically, the sensors detect when customers enter and leave the store, and the cameras track their movements. The application records data on the products customers purchase.

[0353] Sending data to cloud server

[0354] The terminal transmits the collected customer usage information to a cloud server. The cloud server receives the data and links it to an analysis system. Specifically, it uses cloud services.

[0355] Analysis of customer usage information

[0356] The server diagnoses and analyzes customers based on the collected customer usage information. The analysis system uses machine learning algorithms to analyze customers' purchasing trends and frequency of visits to the store. Specifically, it uses publicly known libraries.

[0357] Deciding and displaying the proposal contents

[0358] The server decides which products and conversations should be suggested based on the results of the diagnosis and analysis. The decided suggestions are displayed on the store's displays and on the crew's tablets. This allows suggestions to be made at the same level to all crew members, from newbies to veterans. Specifically, iPads and Android tablets are used.

[0359] Emotion recognition by emotion engine

[0360] The server uses an emotion engine to recognize emotions from the user's facial expressions, tone of voice, choice of words, etc. The emotion engine analyzes the user's facial expressions and voice captured by cameras and microphones installed in the store. Specifically, it uses an emotion recognition API.

[0361] Emotionally responsive service

[0362] The server provides services according to the user's emotions. For example, if it recognizes that the user is happy when looking at a product, it will suggest products and services related to that product. Also, if it recognizes that the user is in trouble, it will provide appropriate assistance.

[0363] Examples

[0364] By using this system, new staff members can suggest the most suitable products to customers they are serving for the first time, based on information such as their purchasing habits and frequency of visits to the store, and can make suggestions on the same level as experienced staff members.

[0365] Examples of prompt statements

[0366] "How can we suggest the best products to customers based on their frequency of visits to the store and their purchasing habits?"

[0367] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0368] Step 1:

[0369] The terminal collects information such as the frequency of customers' visits, the length of their stay, and the products they purchase through sensors and cameras installed in the store, or through an application used by the customer. The sensors detect when customers enter and leave the store, and the cameras track their movements. The application records data on the products the customer purchases. This data is then entered into the terminal.

[0370] Step 2:

[0371] The terminal transmits the collected customer usage information to the cloud server. Specifically, the data is uploaded using known techniques. The cloud server receives the data and stores it in a database. The input is the customer usage information from the terminal, and the output is the data stored in the cloud server.

[0372] Step 3:

[0373] The server performs preprocessing of the data based on the customer usage information stored in the cloud server. Specifically, it performs completion of missing values ​​and normalization of the data. It formats the data using a publicly known library. The input is the raw data stored in the cloud server, and the output is the preprocessed data.

[0374] Step 4:

[0375] The server runs a machine learning algorithm using the preprocessed data. Specifically, it analyzes customer purchasing habits and store visit frequency using known techniques. The input is the preprocessed data, and the output is the analysis result.

[0376] Step 5:

[0377] The server decides which products and phrases to suggest based on the analysis results. Specifically, it selects the most suitable products by taking into account the customer's past purchase history and frequency of visits to the store. The input is the analysis results, and the output is the suggestions.

[0378] Step 6:

[0379] The server displays the determined proposal on the store's display or on the crew's tablet. Specifically, the information is displayed using an iPad or an Android tablet. The input is the proposal, and the output is the displayed information.

[0380] Step 7:

[0381] The terminal collects the user's facial expressions and voice using a camera and microphone installed in the store. The camera captures the user's face, and the microphone records the tone of the voice. These data are input into the terminal.

[0382] Step 8:

[0383] The server inputs the collected data into the emotion engine to analyze the user's emotions. Specifically, it uses an emotion recognition API to recognize the user's emotions such as joy, anger, surprise, and sadness. The input is data from the camera and microphone, and the output is the emotion analysis result.

[0384] Step 9:

[0385] The server provides services according to the user's emotions. For example, if it recognizes that the user is happy when looking at a product, it will suggest products or services related to that product. Also, if it recognizes that the user is in trouble, it will provide appropriate assistance. The input is the emotion analysis result, and the output is the service to be provided.

[0386] (Application example 2)

[0387] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0388] Conventional customer service systems in stores were able to collect and analyze information such as the frequency of customer visits and purchase history, but it was difficult to recognize customers' emotions in real time and provide optimal product suggestions and conversations based on that. In addition, there was often a difference in the quality of suggestions between new and experienced crew members, making it difficult to improve customer satisfaction. Furthermore, there were limited ways to visually display the content of suggestions, making it difficult to provide efficient customer service.

[0389] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0390] In this invention, the server includes a means for collecting customer usage information, a means for data linking, a means for diagnosing and analyzing customers based on the collected customer usage information and the data linking, a means for displaying products and phrases to be proposed based on the diagnosis and analysis results, a means for recognizing emotions from the customer's facial expressions and tone of voice, a means for determining the content of the proposal based on the recognized emotion, and a means for displaying the determined content of the proposal. This enables optimal product proposals and talks according to the customer's emotions, and enables even new crew members to make proposals equivalent to those of veteran crew members, thereby improving customer satisfaction. In addition, efficient customer service can be achieved by displaying the content of the proposal on the display of the smart glasses.

[0391] "Customer usage information" is information generated when a customer uses a store, such as the customer's frequency of visits to the store, length of stay, and purchased items.

[0392] "Data integration" means sending collected customer usage information to an external system such as a cloud server and sharing the information with other systems.

[0393] "Diagnosis and analysis" refers to analyzing customer behavior and trends based on collected customer usage information, and generating information for appropriate product proposals and service provision.

[0394] "Products and conversations to be proposed" refers to products recommended to customers based on the results of diagnosis and analysis, as well as topics and explanations to be used in communication with customers.

[0395] "Means of recognizing emotions from facial expressions and tone of voice" refers to technology that uses a camera and microphone to capture a customer's facial expressions and voice, and analyzes this data to identify the customer's emotions.

[0396] The "means for determining the proposal content" refers to an algorithm or system for determining the optimal product or service proposal content based on recognized customer emotions and collected customer usage information.

[0397] The "display means" refers to a device or interface for visually displaying the determined proposal content, and specifically includes the display of smart glasses.

[0398] The system for carrying out the present invention includes a series of processes for collecting customer usage information, linking data, performing diagnosis and analysis, and displaying proposal contents. Specific embodiments are described below.

[0399] System Configuration

[0400] The system consists of the following main components:

[0401] 1. How we collect customer usage information:

[0402] Smart glasses equipped with a camera and microphone

[0403] Sensors that collect information such as customer visit frequency, length of stay, and purchased items

[0404] 2. Means of data integration:

[0405] Cloud Server

[0406] A communication module that transmits data over an Internet connection

[0407] 3. Diagnostic and analytical methods:

[0408] Analysis system on a cloud server

[0409] Algorithms for analyzing collected customer usage information

[0410] 4. To view the suggestions:

[0411] Smart Glasses Display

[0412] Tablets and store displays

[0413] 5. Ways to recognize emotions:

[0414] Emotion recognition engine that analyzes customer facial expressions and tone of voice

[0415] 6. How the proposal will be determined:

[0416] An algorithm that generates optimal proposals based on emotion recognition results and customer usage information

[0417] Program Processing

[0418] The server does the following:

[0419] 1. Collection of Customer Information:

[0420] The smart glasses' cameras and microphones are used to capture customers' facial expressions and voices, which allows for the collection of information such as how often they visit a store, how long they stay, and what products they purchase.

[0421] 2. Send data to cloud:

[0422] The collected customer information is sent to a cloud server, which receives and stores the data.

[0423] 3. Diagnosis and analysis:

[0424] An analysis system on the cloud server analyzes collected customer information and diagnoses customer behavior and trends. In addition, an emotion recognition engine analyzes the customer's facial expressions and tone of voice to identify their emotions.

[0425] 4. Decision on the proposal:

[0426] Based on the results of diagnosis and analysis and emotion recognition, the most appropriate product and service suggestions are determined.

[0427] 5. Display of proposals:

[0428] The determined proposal content is displayed on the smart glasses' display, allowing store staff to make optimal proposals to customers in real time.

[0429] Examples

[0430] For example, when a customer visits a store and the camera in the smart glasses captures the customer's smile, the emotion recognition engine will recognize it as "happiness." Based on the data sent to the cloud server, the system will suggest products that the customer has previously purchased or related products. The smart glasses' display will say, "We also recommend this product!"

[0431] Examples of prompt statements

[0432] Please create a program that captures customer images, recognizes their emotions, and sends them to a cloud server. Please also include a function to get suggestions from the cloud server and display them on the smart glasses display.

[0433] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0434] Step 1:

[0435] Collecting customer information

[0436] The device (smart glasses) uses a camera and microphone to capture the customer's facial expressions and voice. As input, the customer's facial image and voice data are obtained. These data are collected together with information such as the customer's frequency of visits to the store, the length of stay, and the products purchased. As output, the customer's facial image data and voice data are generated.

[0437] Step 2:

[0438] Send data to the cloud

[0439] The terminal transmits the collected customer information to a cloud server. As input, the customer's facial image data and voice data are used. The data is transmitted to the cloud server via an internet connection. As output, the customer information stored in the cloud server is obtained.

[0440] Step 3:

[0441] Diagnosis and Analysis

[0442] The server analyzes the collected customer information using an analysis system on the cloud server. As input, the customer's facial image data and voice data stored on the cloud server are used. For data processing, face recognition algorithms and voice analysis algorithms are used to diagnose the customer's behavior and tendencies. As output, diagnosis results regarding the customer's behavior patterns and tendencies are obtained.

[0443] Step 4:

[0444] emotion recognition

[0445] The server uses an emotion recognition engine to analyze the customer's facial expression and tone of voice to identify the emotion. As input, the customer's face image data and voice data are used. As data calculation, the emotion of the customer is recognized using an expression analysis algorithm and a tone of voice analysis algorithm. As output, the emotion data of the customer is obtained.

[0446] Step 5:

[0447] Deciding on the proposal content

[0448] The server determines the optimal product or service proposal content based on the diagnosis and analysis results and emotion recognition results. As input, diagnosis results and emotion data on customer behavior patterns and tendencies are used. As data calculation, a proposal algorithm is used to generate optimal proposal content. As output, the proposal content is obtained.

[0449] Step 6:

[0450] View Proposals

[0451] The terminal (smart glasses) displays the determined proposal on the display. The proposal sent from the server is used as input. In concrete terms, a message such as "We also recommend this product!" is displayed on the smart glasses display. As output, the proposal is visually provided to the customer.

[0452] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing.

[0453] The microphone 38B acquires a voice indicating a user input for a result of the specific processing. The control unit 46A transmits the voice data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0454] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0455] Another example of generative AI is Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) are mentioned.

[0456] In the above embodiment, an example has been given in which the specific process is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0457] [Second embodiment]

[0458] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0459] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0460] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0461] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0462] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[0463] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of a typical healthy person).

[0464] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[0465] Fig. 4 shows an example of main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0466] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0467] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0468] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0469] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0470] "Example 1"

[0471] The system of the present invention uses sensors and cameras installed in stores, or applications used by customers, as a means for collecting customer usage information. Through these means, information such as the frequency of customers' visits, the duration of their stay, and the products purchased is collected. In addition, as a means for data linkage, the collected customer usage information is sent to a cloud server, etc., and linked to an analysis system. The analysis system diagnoses and analyzes customers based on the collected customer usage information and data linkage, and determines the products and talks to be proposed based on the diagnosis and analysis results. The determined proposal content is then displayed on the store's display or the crew's tablet, etc. This allows proposals to be made at the same level by new and veteran crew members. As a specific example, when a new crew member proposes the most suitable product based on information such as the purchasing tendency and frequency of visits of a customer they are serving for the first time, by using this system, the new crew member can make proposals equivalent to those of a veteran crew member.

[0472] "Example 2"

[0473] The system of the present invention uses sensors and cameras installed in stores, or applications used by customers, as a means for collecting customer usage information. Through these means, information such as the frequency of customers' visits, the duration of their stay, and the products purchased is collected. In addition, as a means for data linkage, the collected customer usage information is sent to a cloud server, etc., and linked to an analysis system. The analysis system diagnoses and analyzes customers based on the collected customer usage information and data linkage, and determines the products and talks to be proposed based on the diagnosis and analysis results. The determined proposal content is then displayed on the store's display or the crew's tablet, etc. This allows proposals to be made at the same level by new and veteran crew members. As a specific example, when a new crew member proposes the most suitable product based on information such as the purchasing tendency and frequency of visits of a customer they are serving for the first time, by using this system, the new crew member can make proposals equivalent to those of a veteran crew member.

[0474] The process flow of each embodiment will be described below.

[0475] "Example 1"

[0476] Step 1: Using sensors and cameras installed in the store, or applications used by customers, collect information such as how often customers visit the store, how long they stay, and what items they purchase.

[0477] Step 2: The collected customer usage information is sent to a cloud server, etc., and linked to the analysis system.

[0478] Step 3: The analysis system diagnoses and analyzes customers based on the collected customer usage information and data linkages.

[0479] Step 4: Determine the products and conversations to be proposed based on the results of the diagnosis and analysis.

[0480] Step 5: The finalized proposal content is displayed on the store's display and on the crew's tablets, etc. This allows new and experienced crew members to make proposals at the same level.

[0481] "Example 2"

[0482] Step 1: Using sensors and cameras installed in the store, or applications used by customers, collect information such as how often customers visit the store, how long they stay, and what items they purchase.

[0483] Step 2: The collected customer usage information is sent to a cloud server, etc., and linked to the analysis system.

[0484] Step 3: The analysis system diagnoses and analyzes customers based on the collected customer usage information and data linkages.

[0485] Step 4: Determine the products and conversations to be proposed based on the results of the diagnosis and analysis.

[0486] Step 5: The finalized proposal content is displayed on the store's display and on the crew's tablets, etc. This allows new and experienced crew members to make proposals at the same level.

[0487] Example 1

[0488] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0489] With conventional customer support systems, it was difficult to effectively collect and analyze customer usage information and make appropriate product suggestions. In addition, there was often a difference in the quality of suggestions between new and experienced crew members, making it difficult to improve customer satisfaction.

[0490] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0491] In this invention, the server includes a means for collecting customer usage information, a means for transmitting the collected customer usage information to the cloud server, a means for preprocessing the customer usage information received from the cloud server, a means for analyzing the preprocessed customer usage information, a means for generating products and talks to be proposed based on the analysis results, and a means for displaying the generated proposal contents. This makes it possible to effectively collect and analyze customer usage information and to make appropriate product proposals at the same level to new and experienced crew members alike.

[0492] "Customer usage information" is information generated when a customer uses a store, such as the customer's frequency of visits to the store, length of stay, and purchased items.

[0493] "Means of collection" refers to devices and software for obtaining customer usage information using sensors, cameras, applications, etc.

[0494] A "cloud server" is a remote server for storing and managing data via the Internet.

[0495] A "preprocessing means" is a device or software that performs processing to convert the received data into a format suitable for analysis.

[0496] "Means of analysis" refers to devices or software that perform statistical analysis and machine learning on preprocessed data to derive customer behavior patterns and trends.

[0497] "Means for generating" refers to devices or software that automatically create products and conversations to be suggested based on the analysis results.

[0498] The "display means" refers to a device such as a display or tablet for visually displaying the generated proposal content.

[0499] The present invention is a system for collecting and analyzing customer usage information and making appropriate product suggestions. A specific embodiment of this system will be described below.

[0500] Collection of customer usage information

[0501] When a user visits a store, the terminal activates sensors and cameras installed in the store. Specific hardware used is network cameras and motion sensors. These devices collect information such as the frequency of the customer's visits to the store, the length of time they stay, and the products they purchase. Similar information is also collected from applications used by the user. The applications send the customer's purchase history and store visit history to a server.

[0502] Sending data to cloud server

[0503] Before sending the data collected by the terminal to the cloud server, the data is formatted. Specifically, the data is converted to JSON format. The formatted data is securely sent to the cloud server using the HTTPS protocol. A general cloud service is used as the cloud server.

[0504] Analyzing data

[0505] The server receives the data sent from the cloud server. The received data undergoes pre-processing, such as missing value completion and data normalization. The pre-processed data is analyzed using known data analysis tools. The analysis system derives customer behavior patterns and trends.

[0506] Proposal Generation

[0507] The server generates suggested products and chats based on the analysis results. For example, it identifies products that are likely to be purchased next based on the customer's purchase history. The generated suggestions are displayed on store displays and crew tablets.

[0508] View Proposals

[0509] The suggestions generated by the server are displayed on the store's display. Specific hardware used is digital signage and tablets. The suggestions are displayed on the display as "recommended products," and on the tablet it says, "Please suggest this product to this customer."

[0510] Examples

[0511] When a new crew member comes to the store to serve a customer for the first time, sensors and cameras detect the customer's arrival and collect their length of stay and purchase history. The device sends this data to a cloud server, which analyzes it. Based on the analysis results, the server generates a suggestion such as "Please suggest product A to this customer, as he or she has purchased it in the past," and displays it on the crew member's tablet. This allows new crew members to make suggestions on a par with those of veteran crew members.

[0512] Examples of prompt statements

[0513] "Based on how often customers visit the store and their purchase history, what products should we recommend to them next?"

[0514] The system enables stores to provide consistent, high-quality service to their customers.

[0515] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0516] Step 1:

[0517] When a user visits a store, the terminal activates sensors and cameras installed in the store. The sensors and cameras detect the customer's visit and collect information such as frequency of visits, length of stay, and purchased items. The input is the customer's actions and behavior, and the output is the collected customer usage information. In concrete terms, the camera recognizes the customer's face, and the sensor measures the length of stay.

[0518] Step 2:

[0519] Before sending the customer usage information collected by the terminal to the cloud server, the data is formatted. The input is the collected raw data, and the output is the formatted data. Specifically, the data is converted to JSON format and sent to the cloud server using the HTTPS protocol.

[0520] Step 3:

[0521] The server receives the data sent from the cloud server. The input is the data from the cloud server, and the output is the received data. In concrete terms, the server receives the data and stores it in a database.

[0522] Step 4:

[0523] The server preprocesses the data it receives. The input is the received data, and the output is the preprocessed data. Specific operations include filling in missing values ​​and normalizing the data.

[0524] Step 5:

[0525] The server analyzes the pre-processed data. The input is the pre-processed data, and the output is the analysis results. Specifically, the server analyzes the data using known techniques to derive customer behavior patterns and trends.

[0526] Step 6:

[0527] The server generates suggested products and tokens based on the analysis results. The input is the analysis results, and the output is the generated suggestions. Specifically, the server identifies the products that are likely to be purchased next based on the customer's purchase history.

[0528] Step 7:

[0529] The proposal content generated by the server is displayed on the store's display and on the crew's tablet. The input is the generated proposal content, and the output is the displayed proposal content. The specific operation is to display it as a "recommended product" on the display and to display "Please suggest this product to this customer" on the tablet.

[0530] (Application example 1)

[0531] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0532] In conventional customer service at stores, it was difficult for new staff to make optimal product recommendations based on the customer's purchasing habits and frequency of visits, resulting in a difference in the quality of service between new and experienced staff. In addition, there was a lack of tools to grasp customer information in real time and make appropriate recommendations.

[0533] The identification process by the identification processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means. In this invention, the server includes a means for collecting customer usage information, a means for data linking, a means for diagnosing and analyzing visiting customers based on the collected customer usage information and the data linking, a means for displaying products and talks to be proposed based on the diagnosis and analysis results, a means for displaying on the smart glasses, and a means for generating proposal contents using a generative AI model. This enables even a new crew member to make optimal product proposals to customers at the same level as a veteran crew member.

[0534] "Customer usage information" refers to information about a customer's behavior and preferences when using a store, such as the frequency of the customer's visit, the length of time they stay, and the products they purchase.

[0535] "Data integration" means sending collected customer usage information to an external system such as a cloud server and sharing the information with other systems.

[0536] "Diagnosis and analysis" refers to the process of evaluating customer behavior and preferences based on collected customer usage information and determining appropriate product suggestions and services.

[0537] "Products and conversations to be proposed" refers to products recommended to customers based on the results of diagnosis and analysis, as well as topics and explanations to be used in communication with customers.

[0538] "Smart glasses" are wearable devices that have a built-in display and provide users with visual information when worn.

[0539] A "generative AI model" is an algorithm or system that uses artificial intelligence technology to generate optimal suggestions based on input data.

[0540] The system for implementing this invention performs a series of processes to collect customer usage information, link the data, and display products and talks to be proposed based on the diagnosis and analysis results. Specifically, the following hardware and software are used.

[0541] Hardware

[0542] 1. Sensors and cameras: Installed in stores, they collect information such as how often customers visit, how long they stay, and what items they purchase.

[0543] 2. Cloud server: Stores collected customer usage information and connects to the analysis system.

[0544] 3. Smart glasses: Worn by crew members, they display real-time suggestions based on diagnostic and analysis results.

[0545] software

[0546] 1. Data collection module: Collects customer usage information from sensors and cameras and transmits it to the cloud server.

[0547] 2. Data linking module: Links data between the cloud server and the analysis system.

[0548] 3. Analysis system: Diagnosis and analysis are performed based on collected customer usage information, and proposals are generated using a generative AI model.

[0549] 4. Display module: displays the suggestions on the smart glasses.

[0550] Processing flow

[0551] 1. Data collection: Sensors and cameras collect information such as how often customers visit the store, how long they stay, and what items they purchase.

[0552] 2. Data transmission: The collected data is transmitted to the cloud server through the data collection module.

[0553] 3. Data linkage: The cloud server shares data with the analysis system via the data linkage module.

[0554] 4. Diagnosis and analysis: The analysis system performs diagnosis and analysis based on the collected data, and generates optimal proposals using a generative AI model.

[0555] 5. Displaying the Suggestions: The generated suggestions are displayed on the smart glasses through the display module.

[0556] Examples

[0557] For example, if the customer ID is "12345", the customer visits the store twice a week and the products they have purchased in the past are "coffee" and "sandwiches". Based on this information, the following prompt sentence is input to the generative AI model.

[0558] Example prompt:

[0559] Customer visit frequency: twice a week, purchases: coffee, sandwiches

[0560] Based on this prompt, the generative AI model generates suggestions such as the following:

[0561] Examples of suggestions:

[0562] "This customer comes in twice a week and buys a coffee and a sandwich. The next time they come, you could suggest a new seasonal drink or sandwich. You could also encourage them to use their loyalty card to encourage them to visit more frequently."

[0563] These suggestions are then displayed on the smart glasses' display, allowing the crew to make optimal product suggestions to the customer based on that information.

[0564] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0565] Step 1:

[0566] Sensors and cameras collect information such as how often customers visit the store, how long they stay, what products they purchase, etc. These devices monitor customers' behavior from the moment they enter the store and collect data in real time. The input is customer behavior data, and the output is collected customer usage information.

[0567] Step 2:

[0568] The data collection module sends the collected customer usage information to the cloud server. The data collection module organizes the data obtained from the sensors and cameras and uploads it to the cloud server. The input is the collected customer usage information, and the output is the data stored in the cloud server.

[0569] Step 3:

[0570] The cloud server shares data with the analysis system via the data linking module. The cloud server transmits collected customer usage information to the analysis system and links the data. The input is the data stored in the cloud server, and the output is the data transmitted to the analysis system.

[0571] Step 4:

[0572] The analysis system performs diagnosis and analysis based on the data collected. The analysis system analyzes information such as the frequency of customer visits to the store and the products purchased, and generates optimal proposals using a generative AI model. The input is the data sent to the analysis system, and the output is the generated proposals.

[0573] Step 5:

[0574] The generative AI model generates suggestions based on the prompt text. For example, the generative AI model generates appropriate suggestions based on the prompt text "Customer visit frequency: twice a week, Purchased items: coffee, sandwiches." The input is the prompt text, and the output is the generated suggestions.

[0575] Step 6:

[0576] The display module displays the generated proposal on the smart glasses. The display module displays the proposal obtained from the generative AI model on the display of the smart glasses for the crew to review in real time. The input is the generated proposal, and the output is the information displayed on the smart glasses.

[0577] Step 7:

[0578] The crew makes optimal product proposals to the customer based on the proposals displayed on the smart glasses. The crew uses the information displayed on the smart glasses to propose appropriate products and services to the customer. The input is the information displayed on the smart glasses, and the output is the proposals to the customer.

[0579] Example 2

[0580] Next, a description will be given of Example 2 of the embodiment 2. In the following description, the data processing device 12 is referred to as a "server" and the smart glasses 214 are referred to as a "terminal".

[0581] With conventional customer support systems, it was difficult to effectively collect and analyze customer usage information and make optimal product proposals. It was also difficult for new crew members to make proposals to customers at the same level as veteran crew members. This resulted in issues such as insufficient improvement of customer satisfaction and sales promotion.

[0582] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0583] In this invention, the server includes a means for collecting customer usage information, a means for transmitting the collected customer usage information to a cloud server, a means for linking the customer usage information transmitted to the cloud server to an analysis system, a means for diagnosing and analyzing customers based on the collected customer usage information and data linkage, a means for determining products and conversations to be proposed based on the diagnosis and analysis results using a generative AI model, and a means for displaying the determined proposal content on a store display or a crew tablet. This makes it possible to effectively collect and analyze customer usage information and make optimal product proposals. In addition, even new crew members can make proposals to customers at the same level as veteran crew members, which is expected to improve customer satisfaction and promote sales.

[0584] "Customer usage information" refers to information such as the frequency of a customer's visits to a store, the length of their stay, and the items purchased.

[0585] A "cloud server" refers to a remote server that stores and processes data over the Internet.

[0586] An "analysis system" refers to a system that analyzes collected data and derives customer behavior patterns and trends.

[0587] A "generative AI model" refers to an algorithm that uses artificial intelligence to generate optimal suggestions or conversations from data.

[0588] "Display" refers to a display device for visually displaying information.

[0589] "Tablet" refers to a portable, touch-screen computing device.

[0590] "Proposal content" refers to the content of the products or talk to be proposed to the customer.

[0591] "Diagnosis and Analysis" refers to the process of evaluating customer behavior and trends based on collected Customer Usage Information.

[0592] The system of the present invention is for collecting and analyzing customer usage information and making optimal product proposals. A specific embodiment of this system will be described below.

[0593] Data collection

[0594] The terminals use sensors and cameras installed in the store, or applications used by customers, to collect information such as the frequency of customers' visits, length of stay, and purchased items. Specific hardware used is a "general-purpose sensor" for the sensor, a "general-purpose camera" for the camera, and a "store-specific application" for the application. These terminals record customer behavior in real time and collect data.

[0595] Data transmission

[0596] The terminal transmits the collected customer usage information to a cloud server. A "general-purpose cloud service" is used as the cloud server. The secure communication protocol "HTTPS" is used for data transmission. This allows the collected data to be safely stored on the cloud server.

[0597] Data analysis

[0598] The server receives the data sent to the cloud server and connects to the analysis system. The analysis system uses a "general-purpose AI platform." The server diagnoses and analyzes customers based on the collected customer usage information. Specifically, it analyzes data such as the frequency of customers' visits, length of stay, and purchased items to derive customers' purchasing patterns and store visit tendencies.

[0599] Deciding on the proposal content

[0600] The server decides which products and conversations to suggest based on the analysis results. As for the specific software, a "general-purpose generative AI model" is used as the generative AI model. The server inputs a prompt text into the generative AI model and generates optimal product suggestions and conversations. For example, the following is used as a prompt text:

[0601] "Customer ID: 12345 visits the store twice a week, stays for an average of 30 minutes, and recently purchased coffee and sandwiches. Please generate a conversation that will provide optimal product suggestions to this customer."

[0602] View Proposals

[0603] The server displays the determined proposals on the store's displays and on the crew's tablets. A "general-purpose display" is used for the display and a "general-purpose tablet" is used for the tablet. This allows the crew to make optimal product proposals to customers.

[0604] Examples

[0605] By using this system, new crew members can make suggestions on par with those made by veteran crew members when suggesting optimal products to customers they are serving for the first time, based on their purchasing habits and frequency of visits. For example, if a customer frequently purchases coffee, the system will generate a suggestion such as "Would you like to try a new coffee flavor?" and display it on the crew member's tablet.

[0606] The above is a specific embodiment for implementing the system of the present invention. This system makes it possible to effectively collect and analyze customer usage information and make optimal product proposals. In addition, even new crew members can make proposals to customers at the same level as veteran crew members, which is expected to improve customer satisfaction and promote sales.

[0607] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0608] Step 1: Data collection

[0609] The terminal uses sensors and cameras installed in the store, or an application used by the customer, to collect information such as the frequency of customer visits, length of stay, and purchased items. In concrete terms, the sensors detect when customers enter and leave the store, and the cameras record the length of time the customer stays. The application saves the customer's purchase history. The input is customer behavior data, and the output is collected customer usage information.

[0610] Step 2: Send data

[0611] The terminal transmits the collected customer usage information to the cloud server. Specifically, the terminal transmits the collected data to the cloud server using the HTTPS protocol. The input is the collected customer usage information, and the output is the data stored in the cloud server.

[0612] Step 3: Data analysis

[0613] The server receives the data sent to the cloud server and connects it to the analysis system. Specifically, the server uses the AI ​​platform to analyze data such as the frequency of customer visits, length of stay, and purchased items. The input is customer usage information stored in the cloud server, and the output is the analysis results.

[0614] Step 4: Decide on the proposal

[0615] The server decides which products and conversations to suggest based on the analysis results. Specifically, the server inputs a prompt text into the generative AI model to generate optimal product suggestions and conversations. The inputs are the analysis results and the prompt text, and the output is the generated suggestions. For example, the prompt text could be "Customer ID: 12345 visits the store twice a week, spends an average of 30 minutes, and most recently purchased coffee and sandwiches. Please generate a conversation to suggest optimal products to this customer."

[0616] Step 5: View the proposal

[0617] The server displays the determined proposal content on the store's display or on the crew's tablet. In concrete terms, the server sends the generated proposal content to the display or tablet and displays it in real time. The input is the generated proposal content, and the output is the proposal content displayed on the display or tablet. This enables the crew to make optimal product proposals to the customer.

[0618] (Application example 2)

[0619] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the smart glasses 214 will be referred to as a "terminal".

[0620] Customer service in conventional stores was heavily dependent on the experience and skills of staff, making it difficult for new staff to provide the same level of service as veteran staff. In addition, there was a lack of effective means to utilize information such as customers' purchase history and frequency of visits to the store, making it difficult to propose optimal products and services to customers. Furthermore, the inability to analyze and propose customer information in real time made improving customer satisfaction an issue.

[0621] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0622] In this invention, the server includes a means for collecting customer usage information, a means for data linking, a means for diagnosing and analyzing customers visiting the store based on the collected customer usage information and the data linking, a means for generating proposal contents using a generative AI model, and a means for displaying the generated proposal contents on the mobile terminal of the store staff. This allows even new staff to propose optimal products and services to customers at the same level as veteran staff. In addition, since customer information can be analyzed and proposed in real time, it is expected that customer satisfaction will improve.

[0623] "Customer usage information" refers to information such as the frequency of a customer's visit to a store, the length of time they stay, and the products they purchase.

[0624] "Data integration" means sending collected customer usage information to a cloud server or the like and sharing the information with other systems.

[0625] "Diagnosis and analysis" refers to the process of evaluating customer behavior and trends based on collected customer usage information and making appropriate suggestions.

[0626] The "products and conversations to be proposed" refer to the products and services recommended to the customer, as well as the related explanations and conversation content.

[0627] A "generative AI model" is an algorithm or system that uses artificial intelligence to analyze data and generate optimal suggestions.

[0628] A "mobile terminal" is a device that can be carried by store staff, including a smartphone or tablet.

[0629] "Real-time" refers to data collection, analysis, and recommendations being made immediately.

[0630] "Store staff" refers to employees who deal with customers in stores.

[0631] "New staff" refers to employees who have little experience working in a store.

[0632] "Veteran staff" refers to employees with extensive experience working in stores.

[0633] The system for implementing this invention collects customer usage information, links the data, performs diagnosis and analysis based on the collected information, generates proposal content using a generative AI model, and displays the proposal content on the mobile devices of store staff.

[0634] To collect customer usage information, the server uses sensors and cameras installed in the store, or applications used by customers, to collect information such as the frequency of customers' visits, length of stay, and purchased items. The collected information is sent to a cloud server, where it is shared with other systems.

[0635] The server performs diagnosis and analysis based on the collected customer usage information. For this analysis, for example, an AI model of TensorFlow is used. The AI ​​model analyzes data such as the customer's purchase history and frequency of visits to the store, and predicts the product that the customer is likely to purchase next.

[0636] The generative AI model generates proposals based on the analysis results. The generated proposals are displayed as prompts on the store staff's mobile devices. This enables even new staff to propose optimal products and services to customers at the same level as experienced staff.

[0637] As a concrete example, the following prompt sentence is input to the generative AI model:

[0638] "Customer ID: 12345

[0639] Visit frequency: Once a week

[0640] Purchase history: [Product A, Product B, Product C]

[0641] Duration: Average 30 minutes

[0642] Based on this prompt, the AI ​​model generates the following suggestions:

[0643] "The product we recommend to you is Product D. Since you have previously purchased Products A, B, and C, we think Product D will also suit your taste."

[0644] In this way, store staff can recommend the most suitable products to customers.

[0645] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0646] Step 1:

[0647] The server collects customer usage information from sensors and cameras installed in the store or through applications used by customers. Specifically, it obtains data such as the frequency of customers' visits to the store, the length of their stay, and the products they purchase. The input is real-time data from the sensors and cameras, and the output is the collected customer usage information.

[0648] Step 2:

[0649] The server sends the collected customer usage information to a cloud server and shares data with other systems. Specifically, the collected data is stored in a database on the cloud and shared with other analysis systems as necessary. The input is the collected customer usage information, and the output is the data stored in the cloud server.

[0650] Step 3:

[0651] The server performs diagnosis and analysis based on the customer usage information stored in the cloud server. Specifically, it uses an AI model (using TensorFlow, for example) to analyze customer behavior patterns and purchasing trends. The input is the customer usage information stored in the cloud server, and the output is the diagnosis and analysis results.

[0652] Step 4:

[0653] The server uses a generative AI model based on the diagnosis and analysis results to generate proposals. Specifically, the AI ​​model generates prompts that recommend optimal products and services to customers based on the analysis results. The input is the diagnosis and analysis results, and the output is the generated prompts.

[0654] Step 5:

[0655] The server sends the generated prompt text to the mobile device of the store staff. Specifically, the generated suggestion content is displayed in real time on the staff's smartphone or tablet. The input is the generated prompt text, and the output is the suggestion content displayed on the mobile device.

[0656] Step 6:

[0657] The terminal proposes products and services to the customer based on the received proposal content. Specifically, the staff member refers to the prompt displayed on the mobile terminal and introduces appropriate products and services to the customer. The input is the proposal content displayed on the mobile terminal, and the output is the proposal to the customer.

[0658] In addition, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0659] "Example 1"

[0660] In one embodiment of the present invention, an emotion engine is incorporated. The emotion engine recognizes emotions from the user's facial expressions, tone of voice, choice of words, and the like. Specifically, when a user visits a store, the user's facial expressions and voice are captured through cameras and microphones installed in the store, and the emotion engine analyzes this information. The emotion engine can recognize the user's emotions, such as joy, anger, surprise, and sadness, and provide services according to those emotions. For example, if it is recognized that the user is happy to see a product, it will suggest products and services related to that product. Also, if it is recognized that the user is in trouble, it will provide appropriate assistance. This makes it possible to further improve the user's satisfaction.

[0661] "Example 2"

[0662] In one embodiment of the present invention, an emotion engine is incorporated. The emotion engine recognizes emotions from the user's facial expressions, tone of voice, choice of words, and the like. Specifically, when a user visits a store, the user's facial expressions and voice are captured through cameras and microphones installed in the store, and the emotion engine analyzes this information. The emotion engine can recognize the user's emotions, such as joy, anger, surprise, and sadness, and provide services according to those emotions. For example, if it is recognized that the user is happy to see a product, it will suggest products and services related to that product. Also, if it is recognized that the user is in trouble, it will provide appropriate assistance. This makes it possible to further improve the user's satisfaction.

[0663] The process flow of each embodiment will be described below.

[0664] "Example 1"

[0665] Step 1: The user visits the store. Cameras and microphones installed in the store capture the user's facial expressions and voice.

[0666] Step 2: The captured information is sent to the emotion engine, which recognizes emotions from the user's facial expressions, tone of voice, choice of words, etc.

[0667] Step 3: The emotion engine decides which products and services to suggest based on the recognized emotion. For example, if it recognizes that the user is happy to see a product, it will suggest products and services related to that product. Also, if it recognizes that the user is in trouble, it will provide appropriate assistance.

[0668] Step 4: Information about the proposed products and services is communicated to the store staff, who then make the proposal to the user. This makes it possible to further improve user satisfaction.

[0669] "Example 2"

[0670] Step 1: The user visits the store. Cameras and microphones installed in the store capture the user's facial expressions and voice.

[0671] Step 2: The captured information is sent to the emotion engine, which recognizes emotions from the user's facial expressions, tone of voice, choice of words, etc.

[0672] Step 3: The emotion engine decides which products and services to suggest based on the recognized emotion. For example, if it recognizes that the user is happy to see a product, it will suggest products and services related to that product. Also, if it recognizes that the user is in trouble, it will provide appropriate assistance.

[0673] Step 4: Information about the proposed products and services is communicated to the store staff, who then make the proposal to the user. This makes it possible to further improve user satisfaction.

[0674] Example 1

[0675] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0676] With conventional customer service systems, it was difficult to effectively collect and analyze customer usage information and make appropriate suggestions. In addition, because it depended on the experience and skills of the crew, it was difficult for a new crew member to make the same suggestions as a veteran crew member. Furthermore, it was not possible to provide services that took into account the customer's feelings, making it difficult to improve customer satisfaction.

[0677] The identification process by the identification processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means. In this invention, the server includes a means for collecting customer usage information, a means for transmitting the collected customer usage information to a cloud server, a means for analyzing the customer usage information in cooperation with the cloud server, a means for determining products and conversations to be proposed based on the analysis results, a means for displaying the determined proposal contents, a means for recognizing the user's facial expression and voice and analyzing emotions, and a means for providing appropriate services based on the emotion analysis results. This makes it possible to effectively collect and analyze customer usage information and make appropriate suggestions. In addition, even a new crew member can make suggestions equivalent to those of a veteran crew member, and it becomes possible to provide services that take into account the emotions of the customer.

[0678] "Customer usage information" is data generated when a customer uses a store, such as the customer's frequency of visits, length of stay, and purchased items.

[0679] "Means of collection" refers to devices and software for obtaining customer usage information using sensors, cameras, applications, etc.

[0680] A "cloud server" is a remote server that stores and manages data via the Internet and connects with other systems.

[0681] "Analytical tools" are software and algorithms used to process and analyze collected data and understand customer behavior and trends.

[0682] The "means of determining the products and conversations to be proposed" refers to a system for selecting the most suitable products and services for each customer based on the analysis results and generating proposal content.

[0683] The "display means" refers to a device such as a display or tablet for visually displaying the determined proposal content.

[0684] "Means for analyzing emotions" refers to software or algorithms that analyze a user's facial expressions and tone of voice to recognize emotions.

[0685] The "means for providing appropriate services" is a system for providing optimal responses and services to users based on the results of sentiment analysis.

[0686] The present invention is a system for effectively collecting and analyzing customer usage information and making appropriate proposals. A specific embodiment of this system will be described below.

[0687] The server collects customer usage information, such as the frequency of visits, length of stay, and purchased items, from sensors and cameras installed in the store, or through an application used by the customer. The sensors detect the moment a customer enters the store, and the cameras track the customer's movements in real time. The application automatically records data on the items purchased by the customer.

[0688] The collected customer usage information is sent from the server to a cloud server. The cloud server securely stores this data and connects it to the analysis system. Specifically, the data is sent and stored using the Amazon Web Services (AWS) cloud infrastructure.

[0689] The server diagnoses and analyzes customers based on the customer usage information collected using the analysis system. It uses known techniques to analyze the frequency of customers' visits and purchasing trends. For example, it formats and cleanses the data and models the frequency of customers' visits and purchasing trends.

[0690] Based on the analysis results, the server decides which products and conversations should be proposed. The proposed content is then displayed on the store's display or on the crew's tablet. Specifically, the proposed content is displayed using an iPad or Android tablet. The server sends the proposed content generated by the server to the store's display or the crew's tablet, and the device displays the proposed content.

[0691] In addition, the server uses an emotion engine to recognize emotions from the user's facial expressions, tone of voice, choice of words, etc. The emotion engine analyzes the user's facial expressions and voice captured by cameras and microphones installed in the store. Specifically, it uses an emotion recognition API to recognize the user's emotions such as joy, anger, surprise, and sadness.

[0692] The server provides appropriate services depending on the user's emotions recognized by the emotion engine. For example, if it recognizes that the user is happy when looking at a product, it will suggest products or services related to that product. If it recognizes that the user is in trouble, it will provide appropriate assistance. The server determines the appropriate service based on the emotion recognition results and sends it to the store's display or the crew's tablet. The terminal displays the service details, and the crew responds to the user.

[0693] As a concrete example, when a new staff member is serving a customer for the first time, and is trying to recommend the best product based on the customer's purchasing habits and frequency of visits, this system will enable the new staff member to make the same suggestions as a veteran staff member. For example, the system can suggest related products based on the customer's past purchase data.

[0694] An example of a prompt sentence is, "Please tell us how to suggest the best products to customers based on their frequency of visits to the store and their purchasing tendencies."

[0695] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0696] Step 1: Collect customer usage information

[0697] The server collects information such as the frequency of customers' visits, length of stay, and purchased items from sensors and cameras installed in the store, or from applications used by customers. The sensors detect the moment a customer enters the store, and the cameras track the customer's movements in real time. The application automatically records data on the items purchased by the customer. The input is data from the sensors, cameras, and applications, and the output is the collected customer usage information.

[0698] Specific behavior:

[0699] The sensor detects when a customer enters the store and notifies the server.

[0700] The cameras track customer movements and transmit video data to a server.

[0701] The application records the purchase data and sends it to the server.

[0702] Step 2: Send and link data

[0703] The server transmits the collected customer usage information to the cloud server. The cloud server securely stores the data and connects it to the analysis system. The input is the collected customer usage information, and the output is the data stored in the cloud server.

[0704] Specific behavior:

[0705] The server transmits the collected data to a cloud server.

[0706] The cloud server receives and stores the data.

[0707] Step 3: Analyze customer information

[0708] The server diagnoses and analyzes customers based on the customer usage information collected using the analysis system. It uses known technology to analyze the frequency of customers' visits and their purchasing trends. The input is the data stored in the cloud server, and the output is the analysis results.

[0709] Specific behavior:

[0710] The server formats and cleanses the data.

[0711] The server models customers' frequency of visits and purchasing habits.

[0712] Step 4: Decide and present your proposal

[0713] The server decides what products and conversations to suggest based on the diagnosis and analysis results obtained from the analysis system. The proposed content is displayed on the store's display and on the crew's tablets. The input is the analysis results, and the output is the proposed content.

[0714] Specific behavior:

[0715] The server generates suggestions based on the analysis results.

[0716] The suggestions generated by the server are sent to store displays and crew tablets.

[0717] The device will display the suggestions.

[0718] Step 5: Use the Emotion Engine

[0719] The server uses an emotion engine to recognize emotions from the user's facial expressions, tone of voice, choice of words, etc. The emotion engine analyzes the user's facial expressions and voice captured by cameras and microphones installed in the store. The input is data from the cameras and microphones, and the output is the emotion analysis results.

[0720] Specific behavior:

[0721] The camera captures the user's facial expressions and transmits them to the server.

[0722] The microphone records the user's voice and transmits it to the server.

[0723] The server calls the emotion recognition API and analyzes the user's emotions.

[0724] Step 6: Deliver emotional service

[0725] The server provides appropriate services according to the user's emotions recognized by the emotion engine. For example, if it recognizes that the user is happy when looking at a product, it will suggest products and services related to that product. If it recognizes that the user is in trouble, it will provide appropriate assistance. The input is the emotion analysis results, and the output is the service content to be provided.

[0726] Specific behavior:

[0727] The server determines the appropriate service based on the emotion recognition results.

[0728] The server transmits the determined service content to the store's display and crew members' tablets.

[0729] The terminal displays the service details, and the crew responds to the user.

[0730] (Application example 1)

[0731] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0732] Conventional customer service systems can collect and analyze customer usage information, but it is difficult to recognize customer emotions in real time and provide optimal product suggestions and services based on that. In addition, it is difficult for new crew members to provide customer service at the same level as veteran crew members, making it difficult to improve customer satisfaction.

[0733] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0734] In this invention, the server includes a means for collecting customer usage information, a means for linking data, a means for diagnosing and analyzing customers based on the collected customer usage information and the data linking, a means for displaying products and conversations to be proposed based on the results of the diagnosis and analysis, a means for recognizing the customer's facial expression and tone of voice and analyzing their emotions, and a means for determining and displaying the contents of proposals based on the results of the emotion analysis. This makes it possible to provide optimal product proposals and services in real time according to the customer's emotions, and enables even new crew members to handle customers at the same level as veteran crew members.

[0735] "Customer usage information" refers to information such as the frequency of a customer's visit to a store, the length of time they stay, and the products they purchase.

[0736] "Data integration" means sending collected customer usage information to a cloud server or the like and sharing the information with other systems.

[0737] "Diagnosis and analysis" refers to the process of analyzing the behavior and trends of customers visiting the store based on collected customer usage information and proposing appropriate products and services.

[0738] "Means of display" refers to devices or methods for displaying products and conversations to be suggested based on the results of diagnosis and analysis on store displays or staff tablets, etc.

[0739] The "means for recognizing facial expressions and tone of voice" refers to a device or method for capturing a customer's facial expressions and tone of voice using a camera or microphone and analyzing them.

[0740] The "means for analyzing emotions" refers to a device or method for analyzing the emotions of a customer from recognized facial expressions and tone of voice, and identifying emotions such as joy, anger, surprise, sadness, etc.

[0741] The "means for determining the content of proposals based on the results of emotion analysis" refers to a device or method for determining the optimal product or service proposal content according to the analyzed emotion.

[0742] The system for implementing this invention collects customer usage information, links the data, and diagnoses and analyzes customers based on the collected information.Furthermore, it displays products and conversations to be suggested based on the results of the diagnosis and analysis, recognizes the customer's facial expression and tone of voice to analyze their emotions, and determines the content of the suggestions based on those emotions.

[0743] Hardware and Software Used

[0744] Hardware:

[0745] Smart glasses (e.g. Google Glass)

[0746] camera

[0747] microphone

[0748] software:

[0749] Image Processing Library

[0750] Emotion Recognition Library

[0751] Customer Data Management Library

[0752] Cloud server integration library

[0753] System processing overview

[0754] The server first collects customer usage information, including the frequency of customers' visits, the length of their stay, and the products they purchase. This information is collected from sensors and cameras installed in the store, or through applications used by customers. The collected information is sent to a cloud server, where it is shared with other systems.

[0755] Next, the server diagnoses and analyzes the customer based on the collected customer usage information. Based on the results of the diagnosis and analysis, it decides which products and conversations to recommend, and displays them on the store's displays and on staff tablets. This allows even new staff members to serve customers at the same level as veteran staff members.

[0756] In addition, the server recognizes the customer's facial expressions and tone of voice and analyzes their emotions. To do this, the server uses a camera and microphone to capture the customer's facial expressions and voice, and analyzes them using an emotion recognition engine. Based on the analyzed emotions, the server determines and displays the most suitable product suggestions and services.

[0757] Examples

[0758] For example, when a customer visits a store and a staff member wearing smart glasses captures the customer's facial expression, if the staff member recognizes that the customer looks "happy," the store will suggest new products related to those that the customer previously purchased.

[0759] Examples of prompt statements

[0760] Customer ID: 12345

[0761] Customer facial expressions: Delighted

[0762] Recommendation: We also recommend this product!

[0763] In this way, it becomes possible to provide optimal product suggestions and services in real time according to the customer's emotions, thereby improving customer satisfaction.

[0764] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0765] Step 1:

[0766] The server collects customer usage information, such as the frequency of customer visits, length of stay, and purchased items, from sensors and cameras installed in the store or through applications used by customers. The input is data from the sensors and cameras, and the output is the collected customer usage information.

[0767] Step 2:

[0768] The server sends the collected customer usage information to the cloud server and shares data with other systems. The input is the collected customer usage information, and the output is the data stored in the cloud server.

[0769] Step 3:

[0770] The server diagnoses and analyzes customers who visit the store based on the customer usage information stored in the cloud server. The input is the customer usage information obtained from the cloud server, and the output is the diagnosis and analysis results. Specifically, the server uses a data analysis algorithm to analyze customer behavior and trends.

[0771] Step 4:

[0772] The server determines the products and conversations to be proposed based on the diagnosis and analysis results, and displays them on the store's displays and staff tablets. The input is the diagnosis and analysis results, and the output is the displayed proposal content. In concrete terms, the proposal content is generated using a generative AI model and sent to the display or tablet.

[0773] Step 5:

[0774] The terminal (a staff member wearing smart glasses) captures the customer's facial expressions and voice using a camera and microphone. The input is the data from the camera and microphone, and the output is the captured facial expression and voice data.

[0775] Step 6:

[0776] The server uses the captured facial and voice data to analyze the customer's emotions with an emotion recognition engine. The input is the captured facial and voice data, and the output is the analyzed emotion. The specific operation is to analyze facial expressions and tone of voice using an emotion recognition algorithm.

[0777] Step 7:

[0778] The server determines optimal product suggestions and services based on the analyzed emotions and displays them on the smart glasses. The input is the analyzed emotions, and the output is the displayed suggestions. Specifically, the server uses a generative AI model to generate suggestions based on the emotions and sends them to the smart glasses.

[0779] Step 8:

[0780] The terminal (a staff member wearing smart glasses) responds to the customer based on the displayed proposal. The input is the proposal displayed on the smart glasses, and the output is the response to the customer. In concrete terms, the staff member conveys the proposal to the customer.

[0781] Example 2

[0782] Next, a description will be given of Example 2 of the embodiment 2. In the following description, the data processing device 12 is referred to as a "server" and the smart glasses 214 are referred to as a "terminal".

[0783] Conventional customer service systems could collect and analyze customer usage information, but they could not recognize customer emotions and provide services based on them. This made it difficult to respond appropriately to improve customer satisfaction. It was also difficult for new crew members to make suggestions to customers at the same level as veteran crew members.

[0784] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0785] In this invention, the server includes a means for collecting customer usage information, a means for linking data, a means for diagnosing and analyzing customers based on the collected customer usage information and the data linking, a means for displaying products and conversations to be proposed based on the results of the diagnosis and analysis, a means for recognizing the user's facial expressions and voice and analyzing emotions, and a means for providing services based on emotions. This makes it possible to provide services according to the customer's emotions, improving customer satisfaction and enabling new crew members to make suggestions equivalent to those of veteran crew members.

[0786] "Customer usage information" is data generated when a customer uses a store, such as the customer's frequency of visits, length of stay, and purchased items.

[0787] "Data integration" is the process of sending collected customer usage information to an external system such as a cloud server and sharing the information with other systems.

[0788] "Diagnosis and analysis" is the process of analyzing customer behavior and trends based on collected customer usage information and deriving appropriate proposals and services.

[0789] "Products and conversations to be proposed" refer to products recommended to customers based on the results of diagnosis and analysis, as well as topics and phrases to be used in communication with customers.

[0790] "Means of display" refers to devices and software for showing suggested products and conversations on store displays, crew tablets, etc.

[0791] "Means for recognizing a user's facial expressions and voice and analyzing emotions" refers to technology that uses a camera and microphone to capture a user's facial expressions and voice, and analyzes them to identify the user's emotions.

[0792] A "means for providing services based on emotions" is a process or system for providing appropriate product suggestions and assistance based on the results of user emotion analysis.

[0793] The present invention is a system that collects and analyzes customer usage information and makes appropriate suggestions. This system collects information such as the frequency of customers' visits to a store, the length of their stay, and the products they purchase, and uses this information to analyze customer behavior and trends. It can also provide services based on emotions by recognizing the user's facial expressions and voice and analyzing their emotions.

[0794] Collection of customer usage information

[0795] The terminals collect information such as the frequency of customers' visits, the length of their stay, and the products they purchase through sensors and cameras installed in the store, or through an application used by the customer. Specifically, the sensors detect when customers enter and leave the store, and the cameras track their movements. The application records data on the products customers purchase.

[0796] Sending data to cloud server

[0797] The terminal transmits the collected customer usage information to a cloud server. The cloud server receives the data and links it to an analysis system. Specifically, it uses cloud services.

[0798] Analysis of customer usage information

[0799] The server diagnoses and analyzes customers based on the collected customer usage information. The analysis system uses machine learning algorithms to analyze customers' purchasing trends and frequency of visits to the store. Specifically, it uses publicly known libraries.

[0800] Deciding and displaying the proposal contents

[0801] The server decides which products and conversations should be suggested based on the results of the diagnosis and analysis. The decided suggestions are displayed on the store's displays and on the crew's tablets. This allows suggestions to be made at the same level to all crew members, from newbies to veterans. Specifically, iPads and Android tablets are used.

[0802] Emotion recognition by emotion engine

[0803] The server uses an emotion engine to recognize emotions from the user's facial expressions, tone of voice, choice of words, etc. The emotion engine analyzes the user's facial expressions and voice captured by cameras and microphones installed in the store. Specifically, it uses an emotion recognition API.

[0804] Emotionally responsive service

[0805] The server provides services according to the user's emotions. For example, if it recognizes that the user is happy when looking at a product, it will suggest products and services related to that product. Also, if it recognizes that the user is in trouble, it will provide appropriate assistance.

[0806] Examples

[0807] By using this system, new staff members can suggest the most suitable products to customers they are serving for the first time, based on information such as their purchasing habits and frequency of visits to the store, and can make suggestions on the same level as experienced staff members.

[0808] Examples of prompt statements

[0809] "How can we suggest the best products to customers based on their frequency of visits to the store and their purchasing habits?"

[0810] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0811] Step 1:

[0812] The terminal collects information such as the frequency of customers' visits, the length of their stay, and the products they purchase through sensors and cameras installed in the store, or through an application used by the customer. The sensors detect when customers enter and leave the store, and the cameras track their movements. The application records data on the products the customer purchases. This data is then entered into the terminal.

[0813] Step 2:

[0814] The terminal transmits the collected customer usage information to the cloud server. Specifically, the data is uploaded using known techniques. The cloud server receives the data and stores it in a database. The input is the customer usage information from the terminal, and the output is the data stored in the cloud server.

[0815] Step 3:

[0816] The server performs preprocessing of the data based on the customer usage information stored in the cloud server. Specifically, it performs completion of missing values ​​and normalization of the data. It formats the data using a publicly known library. The input is the raw data stored in the cloud server, and the output is the preprocessed data.

[0817] Step 4:

[0818] The server runs a machine learning algorithm using the preprocessed data. Specifically, it analyzes customer purchasing habits and store visit frequency using known techniques. The input is the preprocessed data, and the output is the analysis result.

[0819] Step 5:

[0820] The server decides which products and phrases to suggest based on the analysis results. Specifically, it selects the most suitable products by taking into account the customer's past purchase history and frequency of visits to the store. The input is the analysis results, and the output is the suggestions.

[0821] Step 6:

[0822] The server displays the determined proposal on the store's display or on the crew's tablet. Specifically, the information is displayed using an iPad or an Android tablet. The input is the proposal, and the output is the displayed information.

[0823] Step 7:

[0824] The terminal collects the user's facial expressions and voice using a camera and microphone installed in the store. The camera captures the user's face, and the microphone records the tone of the voice. These data are input into the terminal.

[0825] Step 8:

[0826] The server inputs the collected data into the emotion engine to analyze the user's emotions. Specifically, it uses an emotion recognition API to recognize the user's emotions such as joy, anger, surprise, and sadness. The input is data from the camera and microphone, and the output is the emotion analysis result.

[0827] Step 9:

[0828] The server provides services according to the user's emotions. For example, if it recognizes that the user is happy when looking at a product, it will suggest products or services related to that product. Also, if it recognizes that the user is in trouble, it will provide appropriate assistance. The input is the emotion analysis result, and the output is the service to be provided.

[0829] (Application example 2)

[0830] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the smart glasses 214 will be referred to as a "terminal".

[0831] Conventional customer service systems in stores were able to collect and analyze information such as the frequency of customer visits and purchase history, but it was difficult to recognize customers' emotions in real time and provide optimal product suggestions and conversations based on that. In addition, there was often a difference in the quality of suggestions between new and experienced crew members, making it difficult to improve customer satisfaction. Furthermore, there were limited ways to visually display the content of suggestions, making it difficult to provide efficient customer service.

[0832] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0833] In this invention, the server includes a means for collecting customer usage information, a means for data linking, a means for diagnosing and analyzing customers based on the collected customer usage information and the data linking, a means for displaying products and phrases to be proposed based on the diagnosis and analysis results, a means for recognizing emotions from the customer's facial expressions and tone of voice, a means for determining the content of the proposal based on the recognized emotion, and a means for displaying the determined content of the proposal. This enables optimal product proposals and talks according to the customer's emotions, and enables even new crew members to make proposals equivalent to those of veteran crew members, thereby improving customer satisfaction. In addition, efficient customer service can be achieved by displaying the content of the proposal on the display of the smart glasses.

[0834] "Customer usage information" is information generated when a customer uses a store, such as the customer's frequency of visits to the store, length of stay, and purchased items.

[0835] "Data integration" means sending collected customer usage information to an external system such as a cloud server and sharing the information with other systems.

[0836] "Diagnosis and analysis" refers to analyzing customer behavior and trends based on collected customer usage information, and generating information for appropriate product proposals and service provision.

[0837] "Products and conversations to be proposed" refers to products recommended to customers based on the results of diagnosis and analysis, as well as topics and explanations to be used in communication with customers.

[0838] "Means of recognizing emotions from facial expressions and tone of voice" refers to technology that uses a camera and microphone to capture a customer's facial expressions and voice, and analyzes this data to identify the customer's emotions.

[0839] The "means for determining the proposal content" refers to an algorithm or system for determining the optimal product or service proposal content based on recognized customer emotions and collected customer usage information.

[0840] The "display means" refers to a device or interface for visually displaying the determined proposal content, and specifically includes the display of smart glasses.

[0841] The system for carrying out the present invention includes a series of processes for collecting customer usage information, linking data, performing diagnosis and analysis, and displaying proposal contents. Specific embodiments are described below.

[0842] System Configuration

[0843] The system consists of the following main components:

[0844] 1. How we collect customer usage information:

[0845] Smart glasses equipped with a camera and microphone

[0846] Sensors that collect information such as customer visit frequency, length of stay, and purchased items

[0847] 2. Means of data integration:

[0848] Cloud Server

[0849] A communication module that transmits data over an Internet connection

[0850] 3. Diagnostic and analytical methods:

[0851] Analysis system on a cloud server

[0852] Algorithms for analyzing collected customer usage information

[0853] 4. To view the suggestions:

[0854] Smart Glasses Display

[0855] Tablets and store displays

[0856] 5. Ways to recognize emotions:

[0857] Emotion recognition engine that analyzes customer facial expressions and tone of voice

[0858] 6. How the proposal will be determined:

[0859] An algorithm that generates optimal proposals based on emotion recognition results and customer usage information

[0860] Program Processing

[0861] The server does the following:

[0862] 1. Collection of Customer Information:

[0863] The smart glasses' cameras and microphones are used to capture customers' facial expressions and voices, which allows for the collection of information such as how often they visit a store, how long they stay, and what products they purchase.

[0864] 2. Send data to cloud:

[0865] The collected customer information is sent to a cloud server, which receives and stores the data.

[0866] 3. Diagnosis and analysis:

[0867] An analysis system on the cloud server analyzes collected customer information and diagnoses customer behavior and trends. In addition, an emotion recognition engine analyzes the customer's facial expressions and tone of voice to identify their emotions.

[0868] 4. Decision on the proposal:

[0869] Based on the results of diagnosis and analysis and emotion recognition, the most appropriate product and service suggestions are determined.

[0870] 5. Display of proposals:

[0871] The determined proposal content is displayed on the smart glasses' display, allowing store staff to make optimal proposals to customers in real time.

[0872] Examples

[0873] For example, when a customer visits a store and the camera in the smart glasses captures the customer's smile, the emotion recognition engine will recognize it as "happiness." Based on the data sent to the cloud server, the system will suggest products that the customer has previously purchased or related products. The smart glasses' display will say, "We also recommend this product!"

[0874] Examples of prompt statements

[0875] Please create a program that captures customer images, recognizes their emotions, and sends them to a cloud server. Please also include a function to get suggestions from the cloud server and display them on the smart glasses display.

[0876] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0877] Step 1:

[0878] Collecting customer information

[0879] The device (smart glasses) uses a camera and microphone to capture the customer's facial expressions and voice. As input, the customer's facial image and voice data are obtained. These data are collected together with information such as the customer's frequency of visits to the store, the length of stay, and the products purchased. As output, the customer's facial image data and voice data are generated.

[0880] Step 2:

[0881] Send data to the cloud

[0882] The terminal transmits the collected customer information to a cloud server. As input, the customer's facial image data and voice data are used. The data is transmitted to the cloud server via an internet connection. As output, the customer information stored in the cloud server is obtained.

[0883] Step 3:

[0884] Diagnosis and Analysis

[0885] The server analyzes the collected customer information using an analysis system on the cloud server. As input, the customer's facial image data and voice data stored on the cloud server are used. For data processing, face recognition algorithms and voice analysis algorithms are used to diagnose the customer's behavior and tendencies. As output, diagnosis results regarding the customer's behavior patterns and tendencies are obtained.

[0886] Step 4:

[0887] emotion recognition

[0888] The server uses an emotion recognition engine to analyze the customer's facial expression and tone of voice to identify the emotion. As input, the customer's face image data and voice data are used. As data calculation, the emotion of the customer is recognized using an expression analysis algorithm and a tone of voice analysis algorithm. As output, the emotion data of the customer is obtained.

[0889] Step 5:

[0890] Deciding on the proposal content

[0891] The server determines the optimal product or service proposal content based on the diagnosis and analysis results and emotion recognition results. As input, diagnosis results and emotion data on customer behavior patterns and tendencies are used. As data calculation, a proposal algorithm is used to generate optimal proposal content. As output, the proposal content is obtained.

[0892] Step 6:

[0893] View Proposals

[0894] The terminal (smart glasses) displays the determined proposal on the display. The proposal sent from the server is used as input. In concrete terms, a message such as "We also recommend this product!" is displayed on the smart glasses display. As output, the proposal is visually provided to the customer.

[0895] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits the voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0896] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0897] Another example of generative AI is Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) are mentioned.

[0898] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0899] [Third embodiment]

[0900] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0901] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0902] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0903] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0904] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[0905] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of a typical healthy person).

[0906] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[0907] Fig. 6 shows an example of main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0908] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0909] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0910] In the headset type terminal 314, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0911] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0912] "Example 1"

[0913] The system of the present invention uses sensors and cameras installed in stores, or applications used by customers, as a means for collecting customer usage information. Through these means, information such as the frequency of customers' visits, the duration of their stay, and the products purchased is collected. In addition, as a means for data linkage, the collected customer usage information is sent to a cloud server, etc., and linked to an analysis system. The analysis system diagnoses and analyzes customers based on the collected customer usage information and data linkage, and determines the products and talks to be proposed based on the diagnosis and analysis results. The determined proposal content is then displayed on the store's display or the crew's tablet, etc. This allows proposals to be made at the same level by new and veteran crew members. As a specific example, when a new crew member proposes the most suitable product based on information such as the purchasing tendency and frequency of visits of a customer they are serving for the first time, by using this system, the new crew member can make proposals equivalent to those of a veteran crew member.

[0914] "Example 2"

[0915] The system of the present invention uses sensors and cameras installed in stores, or applications used by customers, as a means for collecting customer usage information. Through these means, information such as the frequency of customers' visits, the duration of their stay, and the products purchased is collected. In addition, as a means for data linkage, the collected customer usage information is sent to a cloud server, etc., and linked to an analysis system. The analysis system diagnoses and analyzes customers based on the collected customer usage information and data linkage, and determines the products and talks to be proposed based on the diagnosis and analysis results. The determined proposal content is then displayed on the store's display or the crew's tablet, etc. This allows proposals to be made at the same level by new and veteran crew members. As a specific example, when a new crew member proposes the most suitable product based on information such as the purchasing tendency and frequency of visits of a customer they are serving for the first time, by using this system, the new crew member can make proposals equivalent to those of a veteran crew member.

[0916] The process flow of each embodiment will be described below.

[0917] "Example 1"

[0918] Step 1: Using sensors and cameras installed in the store, or applications used by customers, collect information such as how often customers visit the store, how long they stay, and what items they purchase.

[0919] Step 2: The collected customer usage information is sent to a cloud server, etc., and linked to the analysis system.

[0920] Step 3: The analysis system diagnoses and analyzes customers based on the collected customer usage information and data linkages.

[0921] Step 4: Determine the products and conversations to be proposed based on the results of the diagnosis and analysis.

[0922] Step 5: The finalized proposal content is displayed on the store's display and on the crew's tablets, etc. This allows new and experienced crew members to make proposals at the same level.

[0923] "Example 2"

[0924] Step 1: Using sensors and cameras installed in the store, or applications used by customers, collect information such as how often customers visit the store, how long they stay, and what items they purchase.

[0925] Step 2: The collected customer usage information is sent to a cloud server, etc., and linked to the analysis system.

[0926] Step 3: The analysis system diagnoses and analyzes customers based on the collected customer usage information and data linkages.

[0927] Step 4: Determine the products and conversations to be proposed based on the results of the diagnosis and analysis.

[0928] Step 5: The finalized proposal content is displayed on the store's display and on the crew's tablets, etc. This allows new and experienced crew members to make proposals at the same level.

[0929] Example 1

[0930] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[0931] With conventional customer support systems, it was difficult to effectively collect and analyze customer usage information and make appropriate product suggestions. In addition, there was often a difference in the quality of suggestions between new and experienced crew members, making it difficult to improve customer satisfaction.

[0932] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0933] In this invention, the server includes a means for collecting customer usage information, a means for transmitting the collected customer usage information to the cloud server, a means for preprocessing the customer usage information received from the cloud server, a means for analyzing the preprocessed customer usage information, a means for generating products and talks to be proposed based on the analysis results, and a means for displaying the generated proposal contents. This makes it possible to effectively collect and analyze customer usage information and to make appropriate product proposals at the same level to new and experienced crew members alike.

[0934] "Customer usage information" is information generated when a customer uses a store, such as the customer's frequency of visits to the store, length of stay, and purchased items.

[0935] "Means of collection" refers to devices and software for obtaining customer usage information using sensors, cameras, applications, etc.

[0936] A "cloud server" is a remote server for storing and managing data via the Internet.

[0937] A "preprocessing means" is a device or software that performs processing to convert the received data into a format suitable for analysis.

[0938] "Means of analysis" refers to devices or software that perform statistical analysis and machine learning on preprocessed data to derive customer behavior patterns and trends.

[0939] "Means for generating" refers to devices or software that automatically create products and conversations to be suggested based on the analysis results.

[0940] The "display means" refers to a device such as a display or tablet for visually displaying the generated proposal content.

[0941] The present invention is a system for collecting and analyzing customer usage information and making appropriate product suggestions. A specific embodiment of this system will be described below.

[0942] Collection of customer usage information

[0943] When a user visits a store, the terminal activates sensors and cameras installed in the store. Specific hardware used is network cameras and motion sensors. These devices collect information such as the frequency of the customer's visits to the store, the length of time they stay, and the products they purchase. Similar information is also collected from applications used by the user. The applications send the customer's purchase history and store visit history to a server.

[0944] Sending data to cloud server

[0945] Before sending the data collected by the terminal to the cloud server, the data is formatted. Specifically, the data is converted to JSON format. The formatted data is securely sent to the cloud server using the HTTPS protocol. A general cloud service is used as the cloud server.

[0946] Analyzing data

[0947] The server receives the data sent from the cloud server. The received data undergoes pre-processing, such as missing value completion and data normalization. The pre-processed data is analyzed using known data analysis tools. The analysis system derives customer behavior patterns and trends.

[0948] Proposal Generation

[0949] The server generates suggested products and chats based on the analysis results. For example, it identifies products that are likely to be purchased next based on the customer's purchase history. The generated suggestions are displayed on store displays and crew tablets.

[0950] View Proposals

[0951] The suggestions generated by the server are displayed on the store's display. Specific hardware used is digital signage and tablets. The suggestions are displayed on the display as "recommended products," and on the tablet it says, "Please suggest this product to this customer."

[0952] Examples

[0953] When a new crew member comes to the store to serve a customer for the first time, sensors and cameras detect the customer's arrival and collect their length of stay and purchase history. The device sends this data to a cloud server, which analyzes it. Based on the analysis results, the server generates a suggestion such as "Please suggest product A to this customer, as he or she has purchased it in the past," and displays it on the crew member's tablet. This allows new crew members to make suggestions on a par with those of veteran crew members.

[0954] Examples of prompt statements

[0955] "Based on how often customers visit the store and their purchase history, what products should we recommend to them next?"

[0956] The system enables stores to provide consistent, high-quality service to their customers.

[0957] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0958] Step 1:

[0959] When a user visits a store, the terminal activates sensors and cameras installed in the store. The sensors and cameras detect the customer's visit and collect information such as frequency of visits, length of stay, and purchased items. The input is the customer's actions and behavior, and the output is the collected customer usage information. In concrete terms, the camera recognizes the customer's face, and the sensor measures the length of stay.

[0960] Step 2:

[0961] Before sending the customer usage information collected by the terminal to the cloud server, the data is formatted. The input is the collected raw data, and the output is the formatted data. Specifically, the data is converted to JSON format and sent to the cloud server using the HTTPS protocol.

[0962] Step 3:

[0963] The server receives the data sent from the cloud server. The input is the data from the cloud server, and the output is the received data. In concrete terms, the server receives the data and stores it in a database.

[0964] Step 4:

[0965] The server preprocesses the data it receives. The input is the received data, and the output is the preprocessed data. Specific operations include filling in missing values ​​and normalizing the data.

[0966] Step 5:

[0967] The server analyzes the pre-processed data. The input is the pre-processed data, and the output is the analysis results. Specifically, the server analyzes the data using known techniques to derive customer behavior patterns and trends.

[0968] Step 6:

[0969] The server generates suggested products and tokens based on the analysis results. The input is the analysis results, and the output is the generated suggestions. Specifically, the server identifies the products that are likely to be purchased next based on the customer's purchase history.

[0970] Step 7:

[0971] The proposal content generated by the server is displayed on the store's display and on the crew's tablet. The input is the generated proposal content, and the output is the displayed proposal content. The specific operation is to display it as a "recommended product" on the display and to display "Please suggest this product to this customer" on the tablet.

[0972] (Application example 1)

[0973] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[0974] In conventional customer service at stores, it was difficult for new staff to make optimal product recommendations based on the customer's purchasing habits and frequency of visits, resulting in a difference in the quality of service between new and experienced staff. In addition, there was a lack of tools to grasp customer information in real time and make appropriate recommendations.

[0975] The identification process by the identification processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means. In this invention, the server includes a means for collecting customer usage information, a means for data linking, a means for diagnosing and analyzing visiting customers based on the collected customer usage information and the data linking, a means for displaying products and talks to be proposed based on the diagnosis and analysis results, a means for displaying on the smart glasses, and a means for generating proposal contents using a generative AI model. This enables even a new crew member to make optimal product proposals to customers at the same level as a veteran crew member.

[0976] "Customer usage information" refers to information about a customer's behavior and preferences when using a store, such as the frequency of the customer's visit, the length of time they stay, and the products they purchase.

[0977] "Data integration" means sending collected customer usage information to an external system such as a cloud server and sharing the information with other systems.

[0978] "Diagnosis and analysis" refers to the process of evaluating customer behavior and preferences based on collected customer usage information and determining appropriate product suggestions and services.

[0979] "Products and conversations to be proposed" refers to products recommended to customers based on the results of diagnosis and analysis, as well as topics and explanations to be used in communication with customers.

[0980] "Smart glasses" are wearable devices that have a built-in display and provide users with visual information when worn.

[0981] A "generative AI model" is an algorithm or system that uses artificial intelligence technology to generate optimal suggestions based on input data.

[0982] The system for implementing this invention performs a series of processes to collect customer usage information, link the data, and display products and talks to be proposed based on the diagnosis and analysis results. Specifically, the following hardware and software are used.

[0983] Hardware

[0984] 1. Sensors and cameras: Installed in stores, they collect information such as how often customers visit, how long they stay, and what items they purchase.

[0985] 2. Cloud server: Stores collected customer usage information and connects to the analysis system.

[0986] 3. Smart glasses: Worn by crew members, they display real-time suggestions based on diagnostic and analysis results.

[0987] software

[0988] 1. Data collection module: Collects customer usage information from sensors and cameras and transmits it to the cloud server.

[0989] 2. Data linking module: Links data between the cloud server and the analysis system.

[0990] 3. Analysis system: Diagnosis and analysis are performed based on collected customer usage information, and proposals are generated using a generative AI model.

[0991] 4. Display module: displays the suggestions on the smart glasses.

[0992] Processing flow

[0993] 1. Data collection: Sensors and cameras collect information such as how often customers visit the store, how long they stay, and what items they purchase.

[0994] 2. Data transmission: The collected data is transmitted to the cloud server through the data collection module.

[0995] 3. Data linkage: The cloud server shares data with the analysis system via the data linkage module.

[0996] 4. Diagnosis and analysis: The analysis system performs diagnosis and analysis based on the collected data, and generates optimal proposals using a generative AI model.

[0997] 5. Displaying the Suggestions: The generated suggestions are displayed on the smart glasses through the display module.

[0998] Examples

[0999] For example, if the customer ID is "12345", the customer visits the store twice a week and the products they have purchased in the past are "coffee" and "sandwiches". Based on this information, the following prompt sentence is input to the generative AI model.

[1000] Example prompt:

[1001] Customer visit frequency: twice a week, purchases: coffee, sandwiches

[1002] Based on this prompt, the generative AI model generates suggestions such as the following:

[1003] Examples of suggestions:

[1004] "This customer comes in twice a week and buys a coffee and a sandwich. The next time they come, you could suggest a new seasonal drink or sandwich. You could also encourage them to use their loyalty card to encourage them to visit more frequently."

[1005] These suggestions are then displayed on the smart glasses' display, allowing the crew to make optimal product suggestions to the customer based on that information.

[1006] The flow of the specific process in the application example 1 will be described with reference to FIG.

[1007] Step 1:

[1008] Sensors and cameras collect information such as how often customers visit the store, how long they stay, what products they purchase, etc. These devices monitor customers' behavior from the moment they enter the store and collect data in real time. The input is customer behavior data, and the output is collected customer usage information.

[1009] Step 2:

[1010] The data collection module sends the collected customer usage information to the cloud server. The data collection module organizes the data obtained from the sensors and cameras and uploads it to the cloud server. The input is the collected customer usage information, and the output is the data stored in the cloud server.

[1011] Step 3:

[1012] The cloud server shares data with the analysis system via the data linking module. The cloud server transmits collected customer usage information to the analysis system and links the data. The input is the data stored in the cloud server, and the output is the data transmitted to the analysis system.

[1013] Step 4:

[1014] The analysis system performs diagnosis and analysis based on the data collected. The analysis system analyzes information such as the frequency of customer visits to the store and the products purchased, and generates optimal proposals using a generative AI model. The input is the data sent to the analysis system, and the output is the generated proposals.

[1015] Step 5:

[1016] The generative AI model generates suggestions based on the prompt text. For example, the generative AI model generates appropriate suggestions based on the prompt text "Customer visit frequency: twice a week, Purchased items: coffee, sandwiches." The input is the prompt text, and the output is the generated suggestions.

[1017] Step 6:

[1018] The display module displays the generated proposal on the smart glasses. The display module displays the proposal obtained from the generative AI model on the display of the smart glasses for the crew to review in real time. The input is the generated proposal, and the output is the information displayed on the smart glasses.

[1019] Step 7:

[1020] The crew makes optimal product proposals to the customer based on the proposals displayed on the smart glasses. The crew uses the information displayed on the smart glasses to propose appropriate products and services to the customer. The input is the information displayed on the smart glasses, and the output is the proposals to the customer.

[1021] Example 2

[1022] Next, a second embodiment of the second form will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1023] With conventional customer support systems, it was difficult to effectively collect and analyze customer usage information and make optimal product proposals. It was also difficult for new crew members to make proposals to customers at the same level as veteran crew members. This resulted in issues such as insufficient improvement of customer satisfaction and sales promotion.

[1024] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1025] In this invention, the server includes a means for collecting customer usage information, a means for transmitting the collected customer usage information to a cloud server, a means for linking the customer usage information transmitted to the cloud server to an analysis system, a means for diagnosing and analyzing customers based on the collected customer usage information and data linkage, a means for determining products and conversations to be proposed based on the diagnosis and analysis results using a generative AI model, and a means for displaying the determined proposal content on a store display or a crew tablet. This makes it possible to effectively collect and analyze customer usage information and make optimal product proposals. In addition, even new crew members can make proposals to customers at the same level as veteran crew members, which is expected to improve customer satisfaction and promote sales.

[1026] "Customer usage information" refers to information such as the frequency of a customer's visits to a store, the length of their stay, and the items purchased.

[1027] A "cloud server" refers to a remote server that stores and processes data over the Internet.

[1028] An "analysis system" refers to a system that analyzes collected data and derives customer behavior patterns and trends.

[1029] A "generative AI model" refers to an algorithm that uses artificial intelligence to generate optimal suggestions or conversations from data.

[1030] "Display" refers to a display device for visually displaying information.

[1031] "Tablet" refers to a portable, touch-screen computing device.

[1032] "Proposal content" refers to the content of the products or talk to be proposed to the customer.

[1033] "Diagnosis and Analysis" refers to the process of evaluating customer behavior and trends based on collected Customer Usage Information.

[1034] The system of the present invention is for collecting and analyzing customer usage information and making optimal product proposals. A specific embodiment of this system will be described below.

[1035] Data collection

[1036] The terminals use sensors and cameras installed in the store, or applications used by customers, to collect information such as the frequency of customers' visits, length of stay, and purchased items. Specific hardware used is a "general-purpose sensor" for the sensor, a "general-purpose camera" for the camera, and a "store-specific application" for the application. These terminals record customer behavior in real time and collect data.

[1037] Data transmission

[1038] The terminal transmits the collected customer usage information to a cloud server. A "general-purpose cloud service" is used as the cloud server. The secure communication protocol "HTTPS" is used for data transmission. This allows the collected data to be safely stored on the cloud server.

[1039] Data analysis

[1040] The server receives the data sent to the cloud server and connects to the analysis system. The analysis system uses a "general-purpose AI platform." The server diagnoses and analyzes customers based on the collected customer usage information. Specifically, it analyzes data such as the frequency of customers' visits, length of stay, and purchased items to derive customers' purchasing patterns and store visit tendencies.

[1041] Deciding on the proposal content

[1042] The server decides which products and conversations to suggest based on the analysis results. As for the specific software, a "general-purpose generative AI model" is used as the generative AI model. The server inputs a prompt text into the generative AI model and generates optimal product suggestions and conversations. For example, the following is used as a prompt text:

[1043] "Customer ID: 12345 visits the store twice a week, stays for an average of 30 minutes, and recently purchased coffee and sandwiches. Please generate a conversation that will provide optimal product suggestions to this customer."

[1044] View Proposals

[1045] The server displays the determined proposals on the store's displays and on the crew's tablets. A "general-purpose display" is used for the display and a "general-purpose tablet" is used for the tablet. This allows the crew to make optimal product proposals to customers.

[1046] Examples

[1047] By using this system, new crew members can make suggestions on par with those made by veteran crew members when suggesting optimal products to customers they are serving for the first time, based on their purchasing habits and frequency of visits. For example, if a customer frequently purchases coffee, the system will generate a suggestion such as "Would you like to try a new coffee flavor?" and display it on the crew member's tablet.

[1048] The above is a specific embodiment for implementing the system of the present invention. This system makes it possible to effectively collect and analyze customer usage information and make optimal product proposals. In addition, even new crew members can make proposals to customers at the same level as veteran crew members, which is expected to improve customer satisfaction and promote sales.

[1049] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1050] Step 1: Data collection

[1051] The terminal uses sensors and cameras installed in the store, or an application used by the customer, to collect information such as the frequency of customer visits, length of stay, and purchased items. In concrete terms, the sensors detect when customers enter and leave the store, and the cameras record the length of time the customer stays. The application saves the customer's purchase history. The input is customer behavior data, and the output is collected customer usage information.

[1052] Step 2: Send data

[1053] The terminal transmits the collected customer usage information to the cloud server. Specifically, the terminal transmits the collected data to the cloud server using the HTTPS protocol. The input is the collected customer usage information, and the output is the data stored in the cloud server.

[1054] Step 3: Data analysis

[1055] The server receives the data sent to the cloud server and connects it to the analysis system. Specifically, the server uses the AI ​​platform to analyze data such as the frequency of customer visits, length of stay, and purchased items. The input is customer usage information stored in the cloud server, and the output is the analysis results.

[1056] Step 4: Decide on the proposal

[1057] The server decides which products and conversations to suggest based on the analysis results. Specifically, the server inputs a prompt text into the generative AI model to generate optimal product suggestions and conversations. The inputs are the analysis results and the prompt text, and the output is the generated suggestions. For example, the prompt text could be "Customer ID: 12345 visits the store twice a week, spends an average of 30 minutes, and most recently purchased coffee and sandwiches. Please generate a conversation to suggest optimal products to this customer."

[1058] Step 5: View the proposal

[1059] The server displays the determined proposal content on the store's display or on the crew's tablet. In concrete terms, the server sends the generated proposal content to the display or tablet and displays it in real time. The input is the generated proposal content, and the output is the proposal content displayed on the display or tablet. This enables the crew to make optimal product proposals to the customer.

[1060] (Application example 2)

[1061] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1062] Customer service in conventional stores was heavily dependent on the experience and skills of staff, making it difficult for new staff to provide the same level of service as veteran staff. In addition, there was a lack of effective means to utilize information such as customers' purchase history and frequency of visits to the store, making it difficult to propose optimal products and services to customers. Furthermore, the inability to analyze and propose customer information in real time made improving customer satisfaction an issue.

[1063] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1064] In this invention, the server includes a means for collecting customer usage information, a means for data linking, a means for diagnosing and analyzing customers visiting the store based on the collected customer usage information and the data linking, a means for generating proposal contents using a generative AI model, and a means for displaying the generated proposal contents on the mobile terminal of the store staff. This allows even new staff to propose optimal products and services to customers at the same level as veteran staff. In addition, since customer information can be analyzed and proposed in real time, it is expected that customer satisfaction will improve.

[1065] "Customer usage information" refers to information such as the frequency of a customer's visit to a store, the length of time they stay, and the products they purchase.

[1066] "Data integration" means sending collected customer usage information to a cloud server or the like and sharing the information with other systems.

[1067] "Diagnosis and analysis" refers to the process of evaluating customer behavior and trends based on collected customer usage information and making appropriate suggestions.

[1068] The "products and conversations to be proposed" refer to the products and services recommended to the customer, as well as the related explanations and conversation content.

[1069] A "generative AI model" is an algorithm or system that uses artificial intelligence to analyze data and generate optimal suggestions.

[1070] A "mobile terminal" is a device that can be carried by store staff, including a smartphone or tablet.

[1071] "Real-time" refers to data collection, analysis, and recommendations being made immediately.

[1072] "Store staff" refers to employees who deal with customers in stores.

[1073] "New staff" refers to employees who have little experience working in a store.

[1074] "Veteran staff" refers to employees with extensive experience working in stores.

[1075] The system for implementing this invention collects customer usage information, links the data, performs diagnosis and analysis based on the collected information, generates proposal content using a generative AI model, and displays the proposal content on the mobile devices of store staff.

[1076] To collect customer usage information, the server uses sensors and cameras installed in the store, or applications used by customers, to collect information such as the frequency of customers' visits, length of stay, and purchased items. The collected information is sent to a cloud server, where it is shared with other systems.

[1077] The server performs diagnosis and analysis based on the collected customer usage information. For this analysis, for example, an AI model of TensorFlow is used. The AI ​​model analyzes data such as the customer's purchase history and frequency of visits to the store, and predicts the product that the customer is likely to purchase next.

[1078] The generative AI model generates proposals based on the analysis results. The generated proposals are displayed as prompts on the store staff's mobile devices. This enables even new staff to propose optimal products and services to customers at the same level as experienced staff.

[1079] As a concrete example, the following prompt sentence is input to the generative AI model:

[1080] "Customer ID: 12345

[1081] Visit frequency: Once a week

[1082] Purchase history: [Product A, Product B, Product C]

[1083] Duration: Average 30 minutes

[1084] Based on this prompt, the AI ​​model generates the following suggestions:

[1085] "The product we recommend to you is Product D. Since you have previously purchased Products A, B, and C, we think Product D will also suit your taste."

[1086] In this way, store staff can recommend the most suitable products to customers.

[1087] The flow of the specific process in the application example 2 will be described with reference to FIG.

[1088] Step 1:

[1089] The server collects customer usage information from sensors and cameras installed in the store or through applications used by customers. Specifically, it obtains data such as the frequency of customers' visits to the store, the length of their stay, and the products they purchase. The input is real-time data from the sensors and cameras, and the output is the collected customer usage information.

[1090] Step 2:

[1091] The server sends the collected customer usage information to a cloud server and shares data with other systems. Specifically, the collected data is stored in a database on the cloud and shared with other analysis systems as necessary. The input is the collected customer usage information, and the output is the data stored in the cloud server.

[1092] Step 3:

[1093] The server performs diagnosis and analysis based on the customer usage information stored in the cloud server. Specifically, it uses an AI model (using TensorFlow, for example) to analyze customer behavior patterns and purchasing trends. The input is the customer usage information stored in the cloud server, and the output is the diagnosis and analysis results.

[1094] Step 4:

[1095] The server uses a generative AI model based on the diagnosis and analysis results to generate proposals. Specifically, the AI ​​model generates prompts that recommend optimal products and services to customers based on the analysis results. The input is the diagnosis and analysis results, and the output is the generated prompts.

[1096] Step 5:

[1097] The server sends the generated prompt text to the mobile device of the store staff. Specifically, the generated suggestion content is displayed in real time on the staff's smartphone or tablet. The input is the generated prompt text, and the output is the suggestion content displayed on the mobile device.

[1098] Step 6:

[1099] The terminal proposes products and services to the customer based on the received proposal content. Specifically, the staff member refers to the prompt displayed on the mobile terminal and introduces appropriate products and services to the customer. The input is the proposal content displayed on the mobile terminal, and the output is the proposal to the customer.

[1100] In addition, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1101] "Example 1"

[1102] In one embodiment of the present invention, an emotion engine is incorporated. The emotion engine recognizes emotions from the user's facial expressions, tone of voice, choice of words, and the like. Specifically, when a user visits a store, the user's facial expressions and voice are captured through cameras and microphones installed in the store, and the emotion engine analyzes this information. The emotion engine can recognize the user's emotions, such as joy, anger, surprise, and sadness, and provide services according to those emotions. For example, if it is recognized that the user is happy to see a product, it will suggest products and services related to that product. Also, if it is recognized that the user is in trouble, it will provide appropriate assistance. This makes it possible to further improve the user's satisfaction.

[1103] "Example 2"

[1104] In one embodiment of the present invention, an emotion engine is incorporated. The emotion engine recognizes emotions from the user's facial expressions, tone of voice, choice of words, and the like. Specifically, when a user visits a store, the user's facial expressions and voice are captured through cameras and microphones installed in the store, and the emotion engine analyzes this information. The emotion engine can recognize the user's emotions, such as joy, anger, surprise, and sadness, and provide services according to those emotions. For example, if it is recognized that the user is happy to see a product, it will suggest products and services related to that product. Also, if it is recognized that the user is in trouble, it will provide appropriate assistance. This makes it possible to further improve the user's satisfaction.

[1105] The process flow of each embodiment will be described below.

[1106] "Example 1"

[1107] Step 1: The user visits the store. Cameras and microphones installed in the store capture the user's facial expressions and voice.

[1108] Step 2: The captured information is sent to the emotion engine, which recognizes emotions from the user's facial expressions, tone of voice, choice of words, etc.

[1109] Step 3: The emotion engine decides which products and services to suggest based on the recognized emotion. For example, if it recognizes that the user is happy to see a product, it will suggest products and services related to that product. Also, if it recognizes that the user is in trouble, it will provide appropriate assistance.

[1110] Step 4: Information about the proposed products and services is communicated to the store staff, who then make the proposal to the user. This makes it possible to further improve user satisfaction.

[1111] "Example 2"

[1112] Step 1: The user visits the store. Cameras and microphones installed in the store capture the user's facial expressions and voice.

[1113] Step 2: The captured information is sent to the emotion engine, which recognizes emotions from the user's facial expressions, tone of voice, choice of words, etc.

[1114] Step 3: The emotion engine decides which products and services to suggest based on the recognized emotion. For example, if it recognizes that the user is happy to see a product, it will suggest products and services related to that product. Also, if it recognizes that the user is in trouble, it will provide appropriate assistance.

[1115] Step 4: Information about the proposed products and services is communicated to the store staff, who then make the proposal to the user. This makes it possible to further improve user satisfaction.

[1116] Example 1

[1117] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1118] With conventional customer service systems, it was difficult to effectively collect and analyze customer usage information and make appropriate suggestions. In addition, because it depended on the experience and skills of the crew, it was difficult for a new crew member to make the same suggestions as a veteran crew member. Furthermore, it was not possible to provide services that took into account the customer's feelings, making it difficult to improve customer satisfaction.

[1119] The identification process by the identification processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means. In this invention, the server includes a means for collecting customer usage information, a means for transmitting the collected customer usage information to a cloud server, a means for analyzing the customer usage information in cooperation with the cloud server, a means for determining products and conversations to be proposed based on the analysis results, a means for displaying the determined proposal contents, a means for recognizing the user's facial expression and voice and analyzing emotions, and a means for providing appropriate services based on the emotion analysis results. This makes it possible to effectively collect and analyze customer usage information and make appropriate suggestions. In addition, even a new crew member can make suggestions equivalent to those of a veteran crew member, and it becomes possible to provide services that take into account the emotions of the customer.

[1120] "Customer usage information" is data generated when a customer uses a store, such as the customer's frequency of visits, length of stay, and purchased items.

[1121] "Means of collection" refers to devices and software for obtaining customer usage information using sensors, cameras, applications, etc.

[1122] A "cloud server" is a remote server that stores and manages data via the Internet and connects with other systems.

[1123] "Analytical tools" are software and algorithms used to process and analyze collected data and understand customer behavior and trends.

[1124] The "means of determining the products and conversations to be proposed" refers to a system for selecting the most suitable products and services for each customer based on the analysis results and generating proposal content.

[1125] The "display means" refers to a device such as a display or tablet for visually displaying the determined proposal content.

[1126] "Means for analyzing emotions" refers to software or algorithms that analyze a user's facial expressions and tone of voice to recognize emotions.

[1127] The "means for providing appropriate services" is a system for providing optimal responses and services to users based on the results of sentiment analysis.

[1128] The present invention is a system for effectively collecting and analyzing customer usage information and making appropriate proposals. A specific embodiment of this system will be described below.

[1129] The server collects customer usage information, such as the frequency of visits, length of stay, and purchased items, from sensors and cameras installed in the store, or through an application used by the customer. The sensors detect the moment a customer enters the store, and the cameras track the customer's movements in real time. The application automatically records data on the items purchased by the customer.

[1130] The collected customer usage information is sent from the server to a cloud server. The cloud server securely stores this data and connects it to the analysis system. Specifically, the data is sent and stored using the Amazon Web Services (AWS) cloud infrastructure.

[1131] The server diagnoses and analyzes customers based on the customer usage information collected using the analysis system. It uses known techniques to analyze the frequency of customers' visits and purchasing trends. For example, it formats and cleanses the data and models the frequency of customers' visits and purchasing trends.

[1132] Based on the analysis results, the server decides which products and conversations should be proposed. The proposed content is then displayed on the store's display or on the crew's tablet. Specifically, the proposed content is displayed using an iPad or Android tablet. The server sends the proposed content generated by the server to the store's display or the crew's tablet, and the device displays the proposed content.

[1133] In addition, the server uses an emotion engine to recognize emotions from the user's facial expressions, tone of voice, choice of words, etc. The emotion engine analyzes the user's facial expressions and voice captured by cameras and microphones installed in the store. Specifically, it uses an emotion recognition API to recognize the user's emotions such as joy, anger, surprise, and sadness.

[1134] The server provides appropriate services depending on the user's emotions recognized by the emotion engine. For example, if it recognizes that the user is happy when looking at a product, it will suggest products or services related to that product. If it recognizes that the user is in trouble, it will provide appropriate assistance. The server determines the appropriate service based on the emotion recognition results and sends it to the store's display or the crew's tablet. The terminal displays the service details, and the crew responds to the user.

[1135] As a concrete example, when a new staff member is serving a customer for the first time, and is trying to recommend the best product based on the customer's purchasing habits and frequency of visits, this system will enable the new staff member to make the same suggestions as a veteran staff member. For example, the system can suggest related products based on the customer's past purchase data.

[1136] An example of a prompt sentence is, "Please tell us how to suggest the best products to customers based on their frequency of visits to the store and their purchasing tendencies."

[1137] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1138] Step 1: Collect customer usage information

[1139] The server collects information such as the frequency of customers' visits, length of stay, and purchased items from sensors and cameras installed in the store, or from applications used by customers. The sensors detect the moment a customer enters the store, and the cameras track the customer's movements in real time. The application automatically records data on the items purchased by the customer. The input is data from the sensors, cameras, and applications, and the output is the collected customer usage information.

[1140] Specific behavior:

[1141] The sensor detects when a customer enters the store and notifies the server.

[1142] The cameras track customer movements and transmit video data to a server.

[1143] The application records the purchase data and sends it to the server.

[1144] Step 2: Send and link data

[1145] The server transmits the collected customer usage information to the cloud server. The cloud server securely stores the data and connects it to the analysis system. The input is the collected customer usage information, and the output is the data stored in the cloud server.

[1146] Specific behavior:

[1147] The server transmits the collected data to a cloud server.

[1148] The cloud server receives and stores the data.

[1149] Step 3: Analyze customer information

[1150] The server diagnoses and analyzes customers based on the customer usage information collected using the analysis system. It uses known technology to analyze the frequency of customers' visits and their purchasing trends. The input is the data stored in the cloud server, and the output is the analysis results.

[1151] Specific behavior:

[1152] The server formats and cleanses the data.

[1153] The server models customers' frequency of visits and purchasing habits.

[1154] Step 4: Decide and present your proposal

[1155] The server decides what products and conversations to suggest based on the diagnosis and analysis results obtained from the analysis system. The proposed content is displayed on the store's display and on the crew's tablets. The input is the analysis results, and the output is the proposed content.

[1156] Specific behavior:

[1157] The server generates suggestions based on the analysis results.

[1158] The suggestions generated by the server are sent to store displays and crew tablets.

[1159] The device will display the suggestions.

[1160] Step 5: Use the Emotion Engine

[1161] The server uses an emotion engine to recognize emotions from the user's facial expressions, tone of voice, choice of words, etc. The emotion engine analyzes the user's facial expressions and voice captured by cameras and microphones installed in the store. The input is data from the cameras and microphones, and the output is the emotion analysis results.

[1162] Specific behavior:

[1163] The camera captures the user's facial expressions and transmits them to the server.

[1164] The microphone records the user's voice and transmits it to the server.

[1165] The server calls the emotion recognition API and analyzes the user's emotions.

[1166] Step 6: Deliver emotional service

[1167] The server provides appropriate services according to the user's emotions recognized by the emotion engine. For example, if it recognizes that the user is happy when looking at a product, it will suggest products and services related to that product. If it recognizes that the user is in trouble, it will provide appropriate assistance. The input is the emotion analysis results, and the output is the service content to be provided.

[1168] Specific behavior:

[1169] The server determines the appropriate service based on the emotion recognition results.

[1170] The server transmits the determined service content to the store's display and crew members' tablets.

[1171] The terminal displays the service details, and the crew responds to the user.

[1172] (Application example 1)

[1173] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1174] Conventional customer service systems can collect and analyze customer usage information, but it is difficult to recognize customer emotions in real time and provide optimal product suggestions and services based on that. In addition, it is difficult for new crew members to provide customer service at the same level as veteran crew members, making it difficult to improve customer satisfaction.

[1175] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1176] In this invention, the server includes a means for collecting customer usage information, a means for linking data, a means for diagnosing and analyzing customers based on the collected customer usage information and the data linking, a means for displaying products and conversations to be proposed based on the results of the diagnosis and analysis, a means for recognizing the customer's facial expression and tone of voice and analyzing their emotions, and a means for determining and displaying the contents of proposals based on the results of the emotion analysis. This makes it possible to provide optimal product proposals and services in real time according to the customer's emotions, and enables even new crew members to handle customers at the same level as veteran crew members.

[1177] "Customer usage information" refers to information such as the frequency of a customer's visit to a store, the length of time they stay, and the products they purchase.

[1178] "Data integration" means sending collected customer usage information to a cloud server or the like and sharing the information with other systems.

[1179] "Diagnosis and analysis" refers to the process of analyzing the behavior and trends of customers visiting the store based on collected customer usage information and proposing appropriate products and services.

[1180] "Means of display" refers to devices or methods for displaying products and conversations to be suggested based on the results of diagnosis and analysis on store displays or staff tablets, etc.

[1181] The "means for recognizing facial expressions and tone of voice" refers to a device or method for capturing a customer's facial expressions and tone of voice using a camera or microphone and analyzing them.

[1182] The "means for analyzing emotions" refers to a device or method for analyzing the emotions of a customer from recognized facial expressions and tone of voice, and identifying emotions such as joy, anger, surprise, sadness, etc.

[1183] The "means for determining the content of proposals based on the results of emotion analysis" refers to a device or method for determining the optimal product or service proposal content according to the analyzed emotion.

[1184] The system for implementing this invention collects customer usage information, links the data, and diagnoses and analyzes customers based on the collected information.Furthermore, it displays products and conversations to be suggested based on the results of the diagnosis and analysis, recognizes the customer's facial expression and tone of voice to analyze their emotions, and determines the content of the suggestions based on those emotions.

[1185] Hardware and Software Used

[1186] Hardware:

[1187] Smart glasses (e.g. Google Glass)

[1188] camera

[1189] microphone

[1190] software:

[1191] Image Processing Library

[1192] Emotion Recognition Library

[1193] Customer Data Management Library

[1194] Cloud server integration library

[1195] System processing overview

[1196] The server first collects customer usage information, including the frequency of customers' visits, the length of their stay, and the products they purchase. This information is collected from sensors and cameras installed in the store, or through applications used by customers. The collected information is sent to a cloud server, where it is shared with other systems.

[1197] Next, the server diagnoses and analyzes the customer based on the collected customer usage information. Based on the results of the diagnosis and analysis, it decides which products and conversations to recommend, and displays them on the store's displays and on staff tablets. This allows even new staff members to serve customers at the same level as veteran staff members.

[1198] In addition, the server recognizes the customer's facial expressions and tone of voice and analyzes their emotions. To do this, the server uses a camera and microphone to capture the customer's facial expressions and voice, and analyzes them using an emotion recognition engine. Based on the analyzed emotions, the server determines and displays the most suitable product suggestions and services.

[1199] Examples

[1200] For example, when a customer visits a store and a staff member wearing smart glasses captures the customer's facial expression, if the staff member recognizes that the customer looks "happy," the store will suggest new products related to those that the customer previously purchased.

[1201] Examples of prompt statements

[1202] Customer ID: 12345

[1203] Customer facial expressions: Delighted

[1204] Recommendation: We also recommend this product!

[1205] In this way, it becomes possible to provide optimal product suggestions and services in real time according to the customer's emotions, thereby improving customer satisfaction.

[1206] The flow of the specific process in the application example 1 will be described with reference to FIG.

[1207] Step 1:

[1208] The server collects customer usage information, such as the frequency of customer visits, length of stay, and purchased items, from sensors and cameras installed in the store or through applications used by customers. The input is data from the sensors and cameras, and the output is the collected customer usage information.

[1209] Step 2:

[1210] The server sends the collected customer usage information to the cloud server and shares data with other systems. The input is the collected customer usage information, and the output is the data stored in the cloud server.

[1211] Step 3:

[1212] The server diagnoses and analyzes customers who visit the store based on the customer usage information stored in the cloud server. The input is the customer usage information obtained from the cloud server, and the output is the diagnosis and analysis results. Specifically, the server uses a data analysis algorithm to analyze customer behavior and trends.

[1213] Step 4:

[1214] The server determines the products and conversations to be proposed based on the diagnosis and analysis results, and displays them on the store's displays and staff tablets. The input is the diagnosis and analysis results, and the output is the displayed proposal content. In concrete terms, the proposal content is generated using a generative AI model and sent to the display or tablet.

[1215] Step 5:

[1216] The terminal (a staff member wearing smart glasses) captures the customer's facial expressions and voice using a camera and microphone. The input is the data from the camera and microphone, and the output is the captured facial expression and voice data.

[1217] Step 6:

[1218] The server uses the captured facial and voice data to analyze the customer's emotions with an emotion recognition engine. The input is the captured facial and voice data, and the output is the analyzed emotion. The specific operation is to analyze facial expressions and tone of voice using an emotion recognition algorithm.

[1219] Step 7:

[1220] The server determines optimal product suggestions and services based on the analyzed emotions and displays them on the smart glasses. The input is the analyzed emotions, and the output is the displayed suggestions. Specifically, the server uses a generative AI model to generate suggestions based on the emotions and sends them to the smart glasses.

[1221] Step 8:

[1222] The terminal (a staff member wearing smart glasses) responds to the customer based on the displayed proposal. The input is the proposal displayed on the smart glasses, and the output is the response to the customer. In concrete terms, the staff member conveys the proposal to the customer.

[1223] Example 2

[1224] Next, a second embodiment of the second form will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1225] Conventional customer service systems could collect and analyze customer usage information, but they could not recognize customer emotions and provide services based on them. This made it difficult to respond appropriately to improve customer satisfaction. It was also difficult for new crew members to make suggestions to customers at the same level as veteran crew members.

[1226] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1227] In this invention, the server includes a means for collecting customer usage information, a means for linking data, a means for diagnosing and analyzing customers based on the collected customer usage information and the data linking, a means for displaying products and conversations to be proposed based on the results of the diagnosis and analysis, a means for recognizing the user's facial expressions and voice and analyzing emotions, and a means for providing services based on emotions. This makes it possible to provide services according to the customer's emotions, improving customer satisfaction and enabling new crew members to make suggestions equivalent to those of veteran crew members.

[1228] "Customer usage information" is data generated when a customer uses a store, such as the customer's frequency of visits, length of stay, and purchased items.

[1229] "Data integration" is the process of sending collected customer usage information to an external system such as a cloud server and sharing the information with other systems.

[1230] "Diagnosis and analysis" is the process of analyzing customer behavior and trends based on collected customer usage information and deriving appropriate proposals and services.

[1231] "Products and conversations to be proposed" refer to products recommended to customers based on the results of diagnosis and analysis, as well as topics and phrases to be used in communication with customers.

[1232] "Means of display" refers to devices and software for showing suggested products and conversations on store displays, crew tablets, etc.

[1233] "Means for recognizing a user's facial expressions and voice and analyzing emotions" refers to technology that uses a camera and microphone to capture a user's facial expressions and voice, and analyzes them to identify the user's emotions.

[1234] A "means for providing services based on emotions" is a process or system for providing appropriate product suggestions and assistance based on the results of user emotion analysis.

[1235] The present invention is a system that collects and analyzes customer usage information and makes appropriate suggestions. This system collects information such as the frequency of customers' visits to a store, the length of their stay, and the products they purchase, and uses this information to analyze customer behavior and trends. It can also provide services based on emotions by recognizing the user's facial expressions and voice and analyzing their emotions.

[1236] Collection of customer usage information

[1237] The terminals collect information such as the frequency of customers' visits, the length of their stay, and the products they purchase through sensors and cameras installed in the store, or through an application used by the customer. Specifically, the sensors detect when customers enter and leave the store, and the cameras track their movements. The application records data on the products customers purchase.

[1238] Sending data to cloud server

[1239] The terminal transmits the collected customer usage information to a cloud server. The cloud server receives the data and links it to an analysis system. Specifically, it uses cloud services.

[1240] Analysis of customer usage information

[1241] The server diagnoses and analyzes customers based on the collected customer usage information. The analysis system uses machine learning algorithms to analyze customers' purchasing trends and frequency of visits to the store. Specifically, it uses publicly known libraries.

[1242] Deciding and displaying the proposal contents

[1243] The server decides which products and conversations should be suggested based on the results of the diagnosis and analysis. The decided suggestions are displayed on the store's displays and on the crew's tablets. This allows suggestions to be made at the same level to all crew members, from newbies to veterans. Specifically, iPads and Android tablets are used.

[1244] Emotion recognition by emotion engine

[1245] The server uses an emotion engine to recognize emotions from the user's facial expressions, tone of voice, choice of words, etc. The emotion engine analyzes the user's facial expressions and voice captured by cameras and microphones installed in the store. Specifically, it uses an emotion recognition API.

[1246] Emotionally responsive service

[1247] The server provides services according to the user's emotions. For example, if it recognizes that the user is happy when looking at a product, it will suggest products and services related to that product. Also, if it recognizes that the user is in trouble, it will provide appropriate assistance.

[1248] Examples

[1249] By using this system, new staff members can suggest the most suitable products to customers they are serving for the first time, based on information such as their purchasing habits and frequency of visits to the store, and can make suggestions on the same level as experienced staff members.

[1250] Examples of prompt statements

[1251] "How can we suggest the best products to customers based on their frequency of visits to the store and their purchasing habits?"

[1252] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1253] Step 1:

[1254] The terminal collects information such as the frequency of customers' visits, the length of their stay, and the products they purchase through sensors and cameras installed in the store, or through an application used by the customer. The sensors detect when customers enter and leave the store, and the cameras track their movements. The application records data on the products the customer purchases. This data is then entered into the terminal.

[1255] Step 2:

[1256] The terminal transmits the collected customer usage information to the cloud server. Specifically, the data is uploaded using known techniques. The cloud server receives the data and stores it in a database. The input is the customer usage information from the terminal, and the output is the data stored in the cloud server.

[1257] Step 3:

[1258] The server performs preprocessing of the data based on the customer usage information stored in the cloud server. Specifically, it performs completion of missing values ​​and normalization of the data. It formats the data using a publicly known library. The input is the raw data stored in the cloud server, and the output is the preprocessed data.

[1259] Step 4:

[1260] The server runs a machine learning algorithm using the preprocessed data. Specifically, it analyzes customer purchasing habits and store visit frequency using known techniques. The input is the preprocessed data, and the output is the analysis result.

[1261] Step 5:

[1262] The server decides which products and phrases to suggest based on the analysis results. Specifically, it selects the most suitable products by taking into account the customer's past purchase history and frequency of visits to the store. The input is the analysis results, and the output is the suggestions.

[1263] Step 6:

[1264] The server displays the determined proposal on the store's display or on the crew's tablet. Specifically, the information is displayed using an iPad or an Android tablet. The input is the proposal, and the output is the displayed information.

[1265] Step 7:

[1266] The terminal collects the user's facial expressions and voice using a camera and microphone installed in the store. The camera captures the user's face, and the microphone records the tone of the voice. These data are input into the terminal.

[1267] Step 8:

[1268] The server inputs the collected data into the emotion engine to analyze the user's emotions. Specifically, it uses an emotion recognition API to recognize the user's emotions such as joy, anger, surprise, and sadness. The input is data from the camera and microphone, and the output is the emotion analysis result.

[1269] Step 9:

[1270] The server provides services according to the user's emotions. For example, if it recognizes that the user is happy when looking at a product, it will suggest products or services related to that product. Also, if it recognizes that the user is in trouble, it will provide appropriate assistance. The input is the emotion analysis result, and the output is the service to be provided.

[1271] (Application example 2)

[1272] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1273] Conventional customer service systems in stores were able to collect and analyze information such as the frequency of customer visits and purchase history, but it was difficult to recognize customers' emotions in real time and provide optimal product suggestions and conversations based on that. In addition, there was often a difference in the quality of suggestions between new and experienced crew members, making it difficult to improve customer satisfaction. Furthermore, there were limited ways to visually display the content of suggestions, making it difficult to provide efficient customer service.

[1274] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1275] In this invention, the server includes a means for collecting customer usage information, a means for data linking, a means for diagnosing and analyzing customers based on the collected customer usage information and the data linking, a means for displaying products and phrases to be proposed based on the diagnosis and analysis results, a means for recognizing emotions from the customer's facial expressions and tone of voice, a means for determining the content of the proposal based on the recognized emotion, and a means for displaying the determined content of the proposal. This enables optimal product proposals and talks according to the customer's emotions, and enables even new crew members to make proposals equivalent to those of veteran crew members, thereby improving customer satisfaction. In addition, efficient customer service can be achieved by displaying the content of the proposal on the display of the smart glasses.

[1276] "Customer usage information" is information generated when a customer uses a store, such as the customer's frequency of visits to the store, length of stay, and purchased items.

[1277] "Data integration" means sending collected customer usage information to an external system such as a cloud server and sharing the information with other systems.

[1278] "Diagnosis and analysis" refers to analyzing customer behavior and trends based on collected customer usage information, and generating information for appropriate product proposals and service provision.

[1279] "Products and conversations to be proposed" refers to products recommended to customers based on the results of diagnosis and analysis, as well as topics and explanations to be used in communication with customers.

[1280] "Means of recognizing emotions from facial expressions and tone of voice" refers to technology that uses a camera and microphone to capture a customer's facial expressions and voice, and analyzes this data to identify the customer's emotions.

[1281] The "means for determining the proposal content" refers to an algorithm or system for determining the optimal product or service proposal content based on recognized customer emotions and collected customer usage information.

[1282] The "display means" refers to a device or interface for visually displaying the determined proposal content, and specifically includes the display of smart glasses.

[1283] The system for carrying out the present invention includes a series of processes for collecting customer usage information, linking data, performing diagnosis and analysis, and displaying proposal contents. Specific embodiments are described below.

[1284] System Configuration

[1285] The system consists of the following main components:

[1286] 1. How we collect customer usage information:

[1287] Smart glasses equipped with a camera and microphone

[1288] Sensors that collect information such as customer visit frequency, length of stay, and purchased items

[1289] 2. Means of data integration:

[1290] Cloud Server

[1291] A communication module that transmits data over an Internet connection

[1292] 3. Diagnostic and analytical methods:

[1293] Analysis system on a cloud server

[1294] Algorithms for analyzing collected customer usage information

[1295] 4. To view the suggestions:

[1296] Smart Glasses Display

[1297] Tablets and store displays

[1298] 5. Ways to recognize emotions:

[1299] Emotion recognition engine that analyzes customer facial expressions and tone of voice

[1300] 6. How the proposal will be determined:

[1301] An algorithm that generates optimal proposals based on emotion recognition results and customer usage information

[1302] Program Processing

[1303] The server does the following:

[1304] 1. Collection of Customer Information:

[1305] The smart glasses' cameras and microphones are used to capture customers' facial expressions and voices, which allows for the collection of information such as how often they visit a store, how long they stay, and what products they purchase.

[1306] 2. Send data to cloud:

[1307] The collected customer information is sent to a cloud server, which receives and stores the data.

[1308] 3. Diagnosis and analysis:

[1309] An analysis system on the cloud server analyzes collected customer information and diagnoses customer behavior and trends. In addition, an emotion recognition engine analyzes the customer's facial expressions and tone of voice to identify their emotions.

[1310] 4. Decision on the proposal:

[1311] Based on the results of diagnosis and analysis and emotion recognition, the most appropriate product and service suggestions are determined.

[1312] 5. Display of proposals:

[1313] The determined proposal content is displayed on the smart glasses' display, allowing store staff to make optimal proposals to customers in real time.

[1314] Examples

[1315] For example, when a customer visits a store and the camera in the smart glasses captures the customer's smile, the emotion recognition engine will recognize it as "happiness." Based on the data sent to the cloud server, the system will suggest products that the customer has previously purchased or related products. The smart glasses' display will say, "We also recommend this product!"

[1316] Examples of prompt statements

[1317] Please create a program that captures customer images, recognizes their emotions, and sends them to a cloud server. Please also include a function to get suggestions from the cloud server and display them on the smart glasses display.

[1318] The flow of the specific process in the application example 2 will be described with reference to FIG.

[1319] Step 1:

[1320] Collecting customer information

[1321] The device (smart glasses) uses a camera and microphone to capture the customer's facial expressions and voice. As input, the customer's facial image and voice data are obtained. These data are collected together with information such as the customer's frequency of visits to the store, the length of stay, and the products purchased. As output, the customer's facial image data and voice data are generated.

[1322] Step 2:

[1323] Send data to the cloud

[1324] The terminal transmits the collected customer information to a cloud server. As input, the customer's facial image data and voice data are used. The data is transmitted to the cloud server via an internet connection. As output, the customer information stored in the cloud server is obtained.

[1325] Step 3:

[1326] Diagnosis and Analysis

[1327] The server analyzes the collected customer information using an analysis system on the cloud server. As input, the customer's facial image data and voice data stored on the cloud server are used. For data processing, face recognition algorithms and voice analysis algorithms are used to diagnose the customer's behavior and tendencies. As output, diagnosis results regarding the customer's behavior patterns and tendencies are obtained.

[1328] Step 4:

[1329] emotion recognition

[1330] The server uses an emotion recognition engine to analyze the customer's facial expression and tone of voice to identify the emotion. As input, the customer's face image data and voice data are used. As data calculation, the emotion of the customer is recognized using an expression analysis algorithm and a tone of voice analysis algorithm. As output, the emotion data of the customer is obtained.

[1331] Step 5:

[1332] Deciding on the proposal content

[1333] The server determines the optimal product or service proposal content based on the diagnosis and analysis results and emotion recognition results. As input, diagnosis results and emotion data on customer behavior patterns and tendencies are used. As data calculation, a proposal algorithm is used to generate optimal proposal content. As output, the proposal content is obtained.

[1334] Step 6:

[1335] View Proposals

[1336] The terminal (smart glasses) displays the determined proposal on the display. The proposal sent from the server is used as input. In concrete terms, a message such as "We also recommend this product!" is displayed on the smart glasses display. As output, the proposal is visually provided to the customer.

[1337] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input for the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1338] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1339] Another example of generative AI is Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) are mentioned.

[1340] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1341] [Fourth embodiment]

[1342] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1343] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1344] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[1345] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. In addition, the microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1346] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[1347] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of a typical healthy person).

[1348] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[1349] The control target 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, legs, etc. The posture and behavior of the robot 414 are controlled by controlling the motors of the arms, hands, legs, etc. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1350] Fig. 8 shows an example of main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1351] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1352] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1353] In the robot 414, the reception and output process is performed by the processor 46. A reception and output program 60 is stored in the storage 50. The processor 46 reads the reception and output program 60 from the storage 50, and executes the read reception and output program 60 on the RAM 48. The reception and output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception and output program 60 executed on the RAM 48.

[1354] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1355] "Example 1"

[1356] The system of the present invention uses sensors and cameras installed in stores, or applications used by customers, as a means for collecting customer usage information. Through these means, information such as the frequency of customers' visits, the duration of their stay, and the products purchased is collected. In addition, as a means for data linkage, the collected customer usage information is sent to a cloud server, etc., and linked to an analysis system. The analysis system diagnoses and analyzes customers based on the collected customer usage information and data linkage, and determines the products and talks to be proposed based on the diagnosis and analysis results. The determined proposal content is then displayed on the store's display or the crew's tablet, etc. This allows proposals to be made at the same level by new and veteran crew members. As a specific example, when a new crew member proposes the most suitable product based on information such as the purchasing tendency and frequency of visits of a customer they are serving for the first time, by using this system, the new crew member can make proposals equivalent to those of a veteran crew member.

[1357] "Example 2"

[1358] The system of the present invention uses sensors and cameras installed in stores, or applications used by customers, as a means for collecting customer usage information. Through these means, information such as the frequency of customers' visits, the duration of their stay, and the products purchased is collected. In addition, as a means for data linkage, the collected customer usage information is sent to a cloud server, etc., and linked to an analysis system. The analysis system diagnoses and analyzes customers based on the collected customer usage information and data linkage, and determines the products and talks to be proposed based on the diagnosis and analysis results. The determined proposal content is then displayed on the store's display or the crew's tablet, etc. This allows proposals to be made at the same level by new and veteran crew members. As a specific example, when a new crew member proposes the most suitable product based on information such as the purchasing tendency and frequency of visits of a customer they are serving for the first time, by using this system, the new crew member can make proposals equivalent to those of a veteran crew member.

[1359] The process flow of each embodiment will be described below.

[1360] "Example 1"

[1361] Step 1: Using sensors and cameras installed in the store, or applications used by customers, collect information such as how often customers visit the store, how long they stay, and what items they purchase.

[1362] Step 2: The collected customer usage information is sent to a cloud server, etc., and linked to the analysis system.

[1363] Step 3: The analysis system diagnoses and analyzes customers based on the collected customer usage information and data linkages.

[1364] Step 4: Determine the products and conversations to be proposed based on the results of the diagnosis and analysis.

[1365] Step 5: The finalized proposal content is displayed on the store's display and on the crew's tablets, etc. This allows new and experienced crew members to make proposals at the same level.

[1366] "Example 2"

[1367] Step 1: Using sensors and cameras installed in the store, or applications used by customers, collect information such as how often customers visit the store, how long they stay, and what items they purchase.

[1368] Step 2: The collected customer usage information is sent to a cloud server, etc., and linked to the analysis system.

[1369] Step 3: The analysis system diagnoses and analyzes customers based on the collected customer usage information and data linkages.

[1370] Step 4: Determine the products and conversations to be proposed based on the results of the diagnosis and analysis.

[1371] Step 5: The finalized proposal content is displayed on the store's display and on the crew's tablets, etc. This allows new and experienced crew members to make proposals at the same level.

[1372] Example 1

[1373] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1374] With conventional customer support systems, it was difficult to effectively collect and analyze customer usage information and make appropriate product suggestions. In addition, there was often a difference in the quality of suggestions between new and experienced crew members, making it difficult to improve customer satisfaction.

[1375] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1376] In this invention, the server includes a means for collecting customer usage information, a means for transmitting the collected customer usage information to the cloud server, a means for preprocessing the customer usage information received from the cloud server, a means for analyzing the preprocessed customer usage information, a means for generating products and talks to be proposed based on the analysis results, and a means for displaying the generated proposal contents. This makes it possible to effectively collect and analyze customer usage information and to make appropriate product proposals at the same level to new and experienced crew members alike.

[1377] "Customer usage information" is information generated when a customer uses a store, such as the customer's frequency of visits to the store, length of stay, and purchased items.

[1378] "Means of collection" refers to devices and software for obtaining customer usage information using sensors, cameras, applications, etc.

[1379] A "cloud server" is a remote server for storing and managing data via the Internet.

[1380] A "preprocessing means" is a device or software that performs processing to convert the received data into a format suitable for analysis.

[1381] "Means of analysis" refers to devices or software that perform statistical analysis and machine learning on preprocessed data to derive customer behavior patterns and trends.

[1382] "Means for generating" refers to devices or software that automatically create products and conversations to be suggested based on the analysis results.

[1383] The "display means" refers to a device such as a display or tablet for visually displaying the generated proposal content.

[1384] The present invention is a system for collecting and analyzing customer usage information and making appropriate product suggestions. A specific embodiment of this system will be described below.

[1385] Collection of customer usage information

[1386] When a user visits a store, the terminal activates sensors and cameras installed in the store. Specific hardware used is network cameras and motion sensors. These devices collect information such as the frequency of the customer's visits to the store, the length of time they stay, and the products they purchase. Similar information is also collected from applications used by the user. The applications send the customer's purchase history and store visit history to a server.

[1387] Sending data to cloud server

[1388] Before sending the data collected by the terminal to the cloud server, the data is formatted. Specifically, the data is converted to JSON format. The formatted data is securely sent to the cloud server using the HTTPS protocol. A general cloud service is used as the cloud server.

[1389] Analyzing data

[1390] The server receives the data sent from the cloud server. The received data undergoes pre-processing, such as missing value completion and data normalization. The pre-processed data is analyzed using known data analysis tools. The analysis system derives customer behavior patterns and trends.

[1391] Proposal Generation

[1392] The server generates suggested products and chats based on the analysis results. For example, it identifies products that are likely to be purchased next based on the customer's purchase history. The generated suggestions are displayed on store displays and crew tablets.

[1393] View Proposals

[1394] The suggestions generated by the server are displayed on the store's display. Specific hardware used is digital signage and tablets. The suggestions are displayed on the display as "recommended products," and on the tablet it says, "Please suggest this product to this customer."

[1395] Examples

[1396] When a new crew member comes to the store to serve a customer for the first time, sensors and cameras detect the customer's arrival and collect their length of stay and purchase history. The device sends this data to a cloud server, which analyzes it. Based on the analysis results, the server generates a suggestion such as "Please suggest product A to this customer, as he or she has purchased it in the past," and displays it on the crew member's tablet. This allows new crew members to make suggestions on a par with those of veteran crew members.

[1397] Examples of prompt statements

[1398] "Based on how often customers visit the store and their purchase history, what products should we recommend to them next?"

[1399] The system enables stores to provide consistent, high-quality service to their customers.

[1400] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1401] Step 1:

[1402] When a user visits a store, the terminal activates sensors and cameras installed in the store. The sensors and cameras detect the customer's visit and collect information such as frequency of visits, length of stay, and purchased items. The input is the customer's actions and behavior, and the output is the collected customer usage information. In concrete terms, the camera recognizes the customer's face, and the sensor measures the length of stay.

[1403] Step 2:

[1404] Before sending the customer usage information collected by the terminal to the cloud server, the data is formatted. The input is the collected raw data, and the output is the formatted data. Specifically, the data is converted to JSON format and sent to the cloud server using the HTTPS protocol.

[1405] Step 3:

[1406] The server receives the data sent from the cloud server. The input is the data from the cloud server, and the output is the received data. In concrete terms, the server receives the data and stores it in a database.

[1407] Step 4:

[1408] The server preprocesses the data it receives. The input is the received data, and the output is the preprocessed data. Specific operations include filling in missing values ​​and normalizing the data.

[1409] Step 5:

[1410] The server analyzes the pre-processed data. The input is the pre-processed data, and the output is the analysis results. Specifically, the server analyzes the data using known techniques to derive customer behavior patterns and trends.

[1411] Step 6:

[1412] The server generates suggested products and tokens based on the analysis results. The input is the analysis results, and the output is the generated suggestions. Specifically, the server identifies the products that are likely to be purchased next based on the customer's purchase history.

[1413] Step 7:

[1414] The proposal content generated by the server is displayed on the store's display and on the crew's tablet. The input is the generated proposal content, and the output is the displayed proposal content. The specific operation is to display it as a "recommended product" on the display and to display "Please suggest this product to this customer" on the tablet.

[1415] (Application example 1)

[1416] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1417] In conventional customer service at stores, it was difficult for new staff to make optimal product recommendations based on the customer's purchasing habits and frequency of visits, resulting in a difference in the quality of service between new and experienced staff. In addition, there was a lack of tools to grasp customer information in real time and make appropriate recommendations.

[1418] The identification process by the identification processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means. In this invention, the server includes a means for collecting customer usage information, a means for data linking, a means for diagnosing and analyzing visiting customers based on the collected customer usage information and the data linking, a means for displaying products and talks to be proposed based on the diagnosis and analysis results, a means for displaying on the smart glasses, and a means for generating proposal contents using a generative AI model. This enables even a new crew member to make optimal product proposals to customers at the same level as a veteran crew member.

[1419] "Customer usage information" refers to information about a customer's behavior and preferences when using a store, such as the frequency of the customer's visit, the length of time they stay, and the products they purchase.

[1420] "Data integration" means sending collected customer usage information to an external system such as a cloud server and sharing the information with other systems.

[1421] "Diagnosis and analysis" refers to the process of evaluating customer behavior and preferences based on collected customer usage information and determining appropriate product suggestions and services.

[1422] "Products and conversations to be proposed" refers to products recommended to customers based on the results of diagnosis and analysis, as well as topics and explanations to be used in communication with customers.

[1423] "Smart glasses" are wearable devices that have a built-in display and provide users with visual information when worn.

[1424] A "generative AI model" is an algorithm or system that uses artificial intelligence technology to generate optimal suggestions based on input data.

[1425] The system for implementing this invention performs a series of processes to collect customer usage information, link the data, and display products and talks to be proposed based on the diagnosis and analysis results. Specifically, the following hardware and software are used.

[1426] Hardware

[1427] 1. Sensors and cameras: Installed in stores, they collect information such as how often customers visit, how long they stay, and what items they purchase.

[1428] 2. Cloud server: Stores collected customer usage information and connects to the analysis system.

[1429] 3. Smart glasses: Worn by crew members, they display real-time suggestions based on diagnostic and analysis results.

[1430] software

[1431] 1. Data collection module: Collects customer usage information from sensors and cameras and transmits it to the cloud server.

[1432] 2. Data linking module: Links data between the cloud server and the analysis system.

[1433] 3. Analysis system: Diagnosis and analysis are performed based on collected customer usage information, and proposals are generated using a generative AI model.

[1434] 4. Display module: displays the suggestions on the smart glasses.

[1435] Processing flow

[1436] 1. Data collection: Sensors and cameras collect information such as how often customers visit the store, how long they stay, and what items they purchase.

[1437] 2. Data transmission: The collected data is transmitted to the cloud server through the data collection module.

[1438] 3. Data linkage: The cloud server shares data with the analysis system via the data linkage module.

[1439] 4. Diagnosis and analysis: The analysis system performs diagnosis and analysis based on the collected data, and generates optimal proposals using a generative AI model.

[1440] 5. Displaying the Suggestions: The generated suggestions are displayed on the smart glasses through the display module.

[1441] Examples

[1442] For example, if the customer ID is "12345", the customer visits the store twice a week and the products they have purchased in the past are "coffee" and "sandwiches". Based on this information, the following prompt sentence is input to the generative AI model.

[1443] Example prompt:

[1444] Customer visit frequency: twice a week, purchases: coffee, sandwiches

[1445] Based on this prompt, the generative AI model generates suggestions such as the following:

[1446] Examples of suggestions:

[1447] "This customer comes in twice a week and buys a coffee and a sandwich. The next time they come, you could suggest a new seasonal drink or sandwich. You could also encourage them to use their loyalty card to encourage them to visit more frequently."

[1448] These suggestions are then displayed on the smart glasses' display, allowing the crew to make optimal product suggestions to the customer based on that information.

[1449] The flow of the specific process in the application example 1 will be described with reference to FIG.

[1450] Step 1:

[1451] Sensors and cameras collect information such as how often customers visit the store, how long they stay, what products they purchase, etc. These devices monitor customers' behavior from the moment they enter the store and collect data in real time. The input is customer behavior data, and the output is collected customer usage information.

[1452] Step 2:

[1453] The data collection module sends the collected customer usage information to the cloud server. The data collection module organizes the data obtained from the sensors and cameras and uploads it to the cloud server. The input is the collected customer usage information, and the output is the data stored in the cloud server.

[1454] Step 3:

[1455] The cloud server shares data with the analysis system via the data linking module. The cloud server transmits collected customer usage information to the analysis system and links the data. The input is the data stored in the cloud server, and the output is the data transmitted to the analysis system.

[1456] Step 4:

[1457] The analysis system performs diagnosis and analysis based on the data collected. The analysis system analyzes information such as the frequency of customer visits to the store and the products purchased, and generates optimal proposals using a generative AI model. The input is the data sent to the analysis system, and the output is the generated proposals.

[1458] Step 5:

[1459] The generative AI model generates suggestions based on the prompt text. For example, the generative AI model generates appropriate suggestions based on the prompt text "Customer visit frequency: twice a week, Purchased items: coffee, sandwiches." The input is the prompt text, and the output is the generated suggestions.

[1460] Step 6:

[1461] The display module displays the generated proposal on the smart glasses. The display module displays the proposal obtained from the generative AI model on the display of the smart glasses for the crew to review in real time. The input is the generated proposal, and the output is the information displayed on the smart glasses.

[1462] Step 7:

[1463] The crew makes optimal product proposals to the customer based on the proposals displayed on the smart glasses. The crew uses the information displayed on the smart glasses to propose appropriate products and services to the customer. The input is the information displayed on the smart glasses, and the output is the proposals to the customer.

[1464] Example 2

[1465] Next, a second embodiment of the second embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1466] With conventional customer support systems, it was difficult to effectively collect and analyze customer usage information and make optimal product proposals. It was also difficult for new crew members to make proposals to customers at the same level as veteran crew members. This resulted in issues such as insufficient improvement of customer satisfaction and sales promotion.

[1467] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1468] In this invention, the server includes a means for collecting customer usage information, a means for transmitting the collected customer usage information to a cloud server, a means for linking the customer usage information transmitted to the cloud server to an analysis system, a means for diagnosing and analyzing customers based on the collected customer usage information and data linkage, a means for determining products and conversations to be proposed based on the diagnosis and analysis results using a generative AI model, and a means for displaying the determined proposal content on a store display or a crew tablet. This makes it possible to effectively collect and analyze customer usage information and make optimal product proposals. In addition, even new crew members can make proposals to customers at the same level as veteran crew members, which is expected to improve customer satisfaction and promote sales.

[1469] "Customer usage information" refers to information such as the frequency of a customer's visits to a store, the length of their stay, and the items purchased.

[1470] A "cloud server" refers to a remote server that stores and processes data over the Internet.

[1471] An "analysis system" refers to a system that analyzes collected data and derives customer behavior patterns and trends.

[1472] A "generative AI model" refers to an algorithm that uses artificial intelligence to generate optimal suggestions or conversations from data.

[1473] "Display" refers to a display device for visually displaying information.

[1474] "Tablet" refers to a portable, touch-screen computing device.

[1475] "Proposal content" refers to the content of the products or talk to be proposed to the customer.

[1476] "Diagnosis and Analysis" refers to the process of evaluating customer behavior and trends based on collected Customer Usage Information.

[1477] The system of the present invention is for collecting and analyzing customer usage information and making optimal product proposals. A specific embodiment of this system will be described below.

[1478] Data collection

[1479] The terminals use sensors and cameras installed in the store, or applications used by customers, to collect information such as the frequency of customers' visits, length of stay, and purchased items. Specific hardware used is a "general-purpose sensor" for the sensor, a "general-purpose camera" for the camera, and a "store-specific application" for the application. These terminals record customer behavior in real time and collect data.

[1480] Data transmission

[1481] The terminal transmits the collected customer usage information to a cloud server. A "general-purpose cloud service" is used as the cloud server. The secure communication protocol "HTTPS" is used for data transmission. This allows the collected data to be safely stored on the cloud server.

[1482] Data analysis

[1483] The server receives the data sent to the cloud server and connects to the analysis system. The analysis system uses a "general-purpose AI platform." The server diagnoses and analyzes customers based on the collected customer usage information. Specifically, it analyzes data such as the frequency of customers' visits, length of stay, and purchased items to derive customers' purchasing patterns and store visit tendencies.

[1484] Deciding on the proposal content

[1485] The server decides which products and conversations to suggest based on the analysis results. As for the specific software, a "general-purpose generative AI model" is used as the generative AI model. The server inputs a prompt text into the generative AI model and generates optimal product suggestions and conversations. For example, the following is used as a prompt text:

[1486] "Customer ID: 12345 visits the store twice a week, stays for an average of 30 minutes, and recently purchased coffee and sandwiches. Please generate a conversation that will provide optimal product suggestions to this customer."

[1487] View Proposals

[1488] The server displays the determined proposals on the store's displays and on the crew's tablets. A "general-purpose display" is used for the display and a "general-purpose tablet" is used for the tablet. This allows the crew to make optimal product proposals to customers.

[1489] Examples

[1490] By using this system, new crew members can make suggestions on par with those made by veteran crew members when suggesting optimal products to customers they are serving for the first time, based on their purchasing habits and frequency of visits. For example, if a customer frequently purchases coffee, the system will generate a suggestion such as "Would you like to try a new coffee flavor?" and display it on the crew member's tablet.

[1491] The above is a specific embodiment for implementing the system of the present invention. This system makes it possible to effectively collect and analyze customer usage information and make optimal product proposals. In addition, even new crew members can make proposals to customers at the same level as veteran crew members, which is expected to improve customer satisfaction and promote sales.

[1492] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1493] Step 1: Data collection

[1494] The terminal uses sensors and cameras installed in the store, or an application used by the customer, to collect information such as the frequency of customer visits, length of stay, and purchased items. In concrete terms, the sensors detect when customers enter and leave the store, and the cameras record the length of time the customer stays. The application saves the customer's purchase history. The input is customer behavior data, and the output is collected customer usage information.

[1495] Step 2: Send data

[1496] The terminal transmits the collected customer usage information to the cloud server. Specifically, the terminal transmits the collected data to the cloud server using the HTTPS protocol. The input is the collected customer usage information, and the output is the data stored in the cloud server.

[1497] Step 3: Data analysis

[1498] The server receives the data sent to the cloud server and connects it to the analysis system. Specifically, the server uses the AI ​​platform to analyze data such as the frequency of customer visits, length of stay, and purchased items. The input is customer usage information stored in the cloud server, and the output is the analysis results.

[1499] Step 4: Decide on the proposal

[1500] The server decides which products and conversations to suggest based on the analysis results. Specifically, the server inputs a prompt text into the generative AI model to generate optimal product suggestions and conversations. The inputs are the analysis results and the prompt text, and the output is the generated suggestions. For example, the prompt text could be "Customer ID: 12345 visits the store twice a week, spends an average of 30 minutes, and most recently purchased coffee and sandwiches. Please generate a conversation to suggest optimal products to this customer."

[1501] Step 5: View the proposal

[1502] The server displays the determined proposal content on the store's display or on the crew's tablet. In concrete terms, the server sends the generated proposal content to the display or tablet and displays it in real time. The input is the generated proposal content, and the output is the proposal content displayed on the display or tablet. This enables the crew to make optimal product proposals to the customer.

[1503] (Application example 2)

[1504] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1505] Customer service in conventional stores was heavily dependent on the experience and skills of staff, making it difficult for new staff to provide the same level of service as veteran staff. In addition, there was a lack of effective means to utilize information such as customers' purchase history and frequency of visits to the store, making it difficult to propose optimal products and services to customers. Furthermore, the inability to analyze and propose customer information in real time made improving customer satisfaction an issue.

[1506] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1507] In this invention, the server includes a means for collecting customer usage information, a means for data linking, a means for diagnosing and analyzing customers visiting the store based on the collected customer usage information and the data linking, a means for generating proposal contents using a generative AI model, and a means for displaying the generated proposal contents on the mobile terminal of the store staff. This allows even new staff to propose optimal products and services to customers at the same level as veteran staff. In addition, since customer information can be analyzed and proposed in real time, it is expected that customer satisfaction will improve.

[1508] "Customer usage information" refers to information such as the frequency of a customer's visit to a store, the length of time they stay, and the products they purchase.

[1509] "Data integration" means sending collected customer usage information to a cloud server or the like and sharing the information with other systems.

[1510] "Diagnosis and analysis" refers to the process of evaluating customer behavior and trends based on collected customer usage information and making appropriate suggestions.

[1511] The "products and conversations to be proposed" refer to the products and services recommended to the customer, as well as the related explanations and conversation content.

[1512] A "generative AI model" is an algorithm or system that uses artificial intelligence to analyze data and generate optimal suggestions.

[1513] A "mobile terminal" is a device that can be carried by store staff, including a smartphone or tablet.

[1514] "Real-time" refers to data collection, analysis, and recommendations being made immediately.

[1515] "Store staff" refers to employees who deal with customers in stores.

[1516] "New staff" refers to employees who have little experience working in a store.

[1517] "Veteran staff" refers to employees with extensive experience working in stores.

[1518] The system for implementing this invention collects customer usage information, links the data, performs diagnosis and analysis based on the collected information, generates proposal content using a generative AI model, and displays the proposal content on the mobile devices of store staff.

[1519] To collect customer usage information, the server uses sensors and cameras installed in the store, or applications used by customers, to collect information such as the frequency of customers' visits, length of stay, and purchased items. The collected information is sent to a cloud server, where it is shared with other systems.

[1520] The server performs diagnosis and analysis based on the collected customer usage information. For...

Claims

1. A first means for collecting customer usage information including a customer's frequency of visits to a store, a duration of stay, and information on purchased items; a second means for generating a first prompt sentence for outputting a proposal content to be proposed to the customer based on the collected customer usage information; a third means for inputting the generated first prompt sentence into a generative AI model to generate the suggestion content; A fourth means for displaying the generated proposal content on a tablet of a crew member who provides a service to the customer; A system including:

2. a fifth means for analyzing emotions of the customer using an emotion recognition engine by using facial expression and voice data of the customer; The second means generates a second prompt sentence for outputting the proposal content according to the customer's emotion based on the customer usage information and the customer's emotion, The third means inputs the generated second prompt sentence into a generative AI model to generate the suggestion content. The system of claim 1 .

3. The third means generates the proposal content for proposing a product or service related to the product when the emotion of the customer who has viewed the product indicates joy. The system of claim 2.

Images