system

Abstract

A system is provided.SOLUTION: A system comprising: means for collecting voice data; means for converting the collected voice data into text data; means for collecting search data; means for analyzing a reaction or concern of a customer using the voice data and the search data; means for generating a virtual FAQ based on an analysis result; means for generating learning content and a promotion tool based on the analysis result; and means for displaying the generated virtual FAQ, learning content, and promotion tool.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] When introducing a new service, the success of the service can be jeopardized because customer reactions and concerns are unknown in advance. Furthermore, store staff are often unable to make the necessary preparations or take the necessary measures based on this unclear information, which can lead to a decline in customer satisfaction. Furthermore, there is a risk of losing customer trust due to an inability to respond quickly and accurately to customer questions. To solve these problems, a system is needed that can predict customer reactions and concerns in advance and provide solutions. [Means for solving the problem]

[0005] The present invention solves these problems by providing a system that includes a means for collecting voice data, a means for converting the collected voice data into text data, a means for collecting search data, a means for analyzing customer responses and concerns using the voice data and the search data, a means for generating a virtual FAQ based on the analysis results, a means for generating learning content and promotional tools based on the analysis results, and a means for displaying the generated virtual FAQ, learning content, and promotional tools.

[0006] Specifically, the system collects voice data from voice recorders used in stores, converts it into text data, and also collects user search ranking data. Using this data, AI analyzes customer reactions and concerns, and runs simulations using virtual avatars. Based on the results, virtual FAQs, necessary learning content, and sales promotion tools are automatically generated and displayed for easy use by store staff so that they can respond in advance. This makes it possible to understand customer reactions and concerns before introducing new services, and to prepare appropriate countermeasures.

[0007] "Voice data" refers to audio information recorded from conversations or dialogues with customers.

[0008] "Text data" is character information obtained by analyzing and converting voice data.

[0009] "Search data" refers to keywords entered by users through online search systems and ranking information for search results.

[0010] An "AI model" is an algorithm and its implementation that uses artificial intelligence technology to analyze and learn from data and mimic human behavior and reactions.

[0011] A "virtual FAQ" is a collection of frequently asked questions from customers and their answers, generated based on the analysis results of an AI model.

[0012] "Learning content" refers to teaching materials and resources that store staff can use to learn about new services.

[0013] "Promotional tools" are materials (POP, pamphlets, posters, etc.) used to advertise new services and promote sales.

[0014] A "voice recorder" is a device for recording audio.

[0015] A "customer avatar" is a digital simulation model generated by an AI model to mimic a real-life customer.

[0016] An "interface module" is a software component that displays generated FAQs, learning content, and promotional tools to users. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] The present invention is a system that uses voice data and search data to analyze customer reactions and concerns, and generates and provides virtual FAQs, learning content, and sales promotion tools based on the results. This system is implemented as follows.

[0039] System Overview

[0040] Information gathering phase

[0041] The terminal records conversations with customers using a voice recorder used in the store. This voice data is sent to multiple servers via the network. At the same time, the servers obtain user search ranking data from an online search system.

[0042] Data analysis phase

[0043] The server first converts the received voice data into text data using speech recognition. It then trains an AI model based on the voice data and search data. This AI model uses natural language processing (NLP) algorithms to automatically analyze customer responses and concerns. The server then generates and simulates an AI customer avatar based on multiple attributes.

[0044] FAQ and tool generation phase

[0045] Based on the simulation results, the server generates a virtual FAQ, which includes frequently asked questions and their answers. It also generates educational content and promotional tools (POPs, pamphlets, etc.) for store staff to learn about new services.

[0046] Results delivery phase

[0047] The generated virtual FAQs, learning content, and sales promotion tools are provided to the store staff via an interface module in an accessible state, allowing users to understand customer reactions and concerns before implementing the service and prepare appropriate countermeasures in advance.

[0048] Specific examples

[0049] Collecting and analyzing customer interaction data

[0050] Users (store staff) use voice recorders to record their daily conversations with customers. For example, they can record conversations while explaining a new service. The data is sent to a server in real time and converted into text data using voice recognition technology.

[0051] Search data collection and analysis

[0052] At the same time, the server obtains search rankings for keywords related to the new service from the search system, including information on what questions users frequently search.

[0053] Training and simulating AI models

[0054] The server uses the collected voice and search data to train an AI model that analyzes likely customer reactions and concerns, such as the likelihood that many customers will have questions about the pricing structure of a particular service.

[0055] Virtual FAQ and tool generation

[0056] Based on the analysis results, the server automatically generates virtual FAQs, learning content, and sales promotion tools, allowing users to check specific questions and answers and what they need to learn in advance, while also providing marketing materials to encourage purchases.

[0057] In this way, the system provides a concrete means for anticipating customer reactions and concerns when introducing new services and providing appropriate responses.

[0058] The processing flow will be explained below.

[0059] Specific flow of program processing

[0060] Information gathering phase

[0061] Step 1:

[0062] The terminal uses a voice recorder used in the store to record conversations with customers, including questions and feedback from the customers.

[0063] Step 2:

[0064] The server collects the audio data from the voice recorder at regular intervals, which involves the process of transferring the data over the network.

[0065] Step 3:

[0066] The server sends the voice data to a speech recognition engine and converts it into text data, which is then stored in a database.

[0067] Step 4:

[0068] The server retrieves user search ranking data, including search keywords and frequently asked questions related to the new service, and stores this information in a database.

[0069] Data analysis phase

[0070] Step 5:

[0071] The server takes in the voice data and search ranking data and begins training the AI ​​model, which uses natural language processing (NLP) algorithms.

[0072] Step 6:

[0073] The server uses an AI model to analyze customer reactions and concerns, for example, determining which parts of the service customers are likely to have questions about and what their concerns are.

[0074] Step 7:

[0075] The server generates AI customer avatars based on multiple attributes, including age, gender, and region, and simulates how each avatar would react to new services.

[0076] FAQ and tool generation phase

[0077] Step 8:

[0078] The server generates a virtual FAQ based on the simulation results, which includes frequently asked questions and answers derived from the reactions and concerns of each customer avatar.

[0079] Step 9:

[0080] The server generates learning content based on the simulation results, similar to the virtual FAQ, including manuals and online learning materials to help store staff deepen their understanding of the new service.

[0081] Step 10:

[0082] The server automatically generates promotional tools, such as POP displays and pamphlets, to effectively introduce and advertise new services.

[0083] Results delivery phase

[0084] Step 11:

[0085] Users (store staff) can access the generated virtual FAQs, learning content, and sales promotion tools through the interface module, allowing users to make the necessary preparations before introducing new services.

[0086] Step 12:

[0087] When users actually implement new services, they can use the generated tools and content to respond to customers, thereby increasing customer satisfaction.

[0088] Example 1

[0089] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0090] Today's consumers have diverse and complex needs, and it is essential to accurately understand those needs and provide them with appropriate information. However, traditional methods make it difficult to fully grasp customers' specific reactions and concerns, making it impossible to generate effective FAQs, learning content, and sales promotion tools. Furthermore, manually analyzing voice data and search data is laborious and time-consuming, placing a significant burden on on-site staff. A system that can solve these problems is needed.

[0091] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0092] In this invention, the server includes means for collecting voice data, means for converting the collected voice data into text data, means for collecting search data, means for training an AI model using a natural language processing algorithm to analyze customer reactions and concerns using the voice data and the search data, means for generating a virtual FAQ based on the analysis results, means for generating learning content and promotional tools based on the analysis results, means for generating an AI customer avatar based on multiple attributes and performing a simulation, and means for displaying the generated virtual FAQ, learning content, and promotional tools. This makes it possible to quickly and accurately grasp specific customer reactions and concerns and effectively provide the information and tools needed by on-site staff.

[0093] 1. "Voice data" refers to data that represents acoustic signals in digital format, and is a record of conversations with customers, etc.

[0094] 2. "Text data" means data that has been converted from audio data into textual information and is expressed in a format that can be analyzed using natural language processing.

[0095] 3. "Search Data" means data including ranking information for keywords and search queries that users search on the Internet or internal systems.

[0096] 4. "Natural language processing algorithms" refers to the algorithms used by AI models to analyze text data, including technologies such as speech recognition and semantic analysis.

[0097] 5. "AI Model" means a model of artificial intelligence trained using machine learning techniques and designed to perform specific tasks automatically.

[0098] 6. "Virtual FAQ" refers to automatically generated questions and answers frequently asked by customers, including information to resolve customer concerns.

[0099] 7. "Learning Content" refers to educational materials that store staff and customers can use to learn about new services and products, including text, images, videos, etc.

[0100] 8. "Promotional tools" are materials and advertising materials used to effectively introduce new services and products to customers, including posters, pamphlets, etc.

[0101] 9. An "AI customer avatar" is a virtual customer character generated based on various attribute information and is a model for conducting simulations.

[0102] 10. "Simulation" is a methodology for using AI customer avatars to recreate and analyze anticipated customer reactions and concerns.

[0103] This invention is a system that uses voice data and search data to analyze customer reactions and concerns, and generates and provides virtual FAQs, learning content, and sales promotion tools based on the results. This system is implemented as follows.

[0104] System Overview

[0105] The system includes means for collecting voice data and converting it into text data, means for collecting search data, means for analyzing customer responses and concerns, means for generating virtual FAQs, learning content, and promotional tools, and means for displaying them.

[0106] Hardware and software used

[0107] Terminal: Voice recorder (to record conversations with customers), network connection device (to transmit voice data)

[0108] Server: High-performance computer (data processing and storage), software (data conversion and analysis)

[0109] Google Cloud Speech-to-Text API (converts voice data into text data)

[0110] Machine learning libraries (e.g., TensorFlow) (used to train AI models)

[0111] Natural language processing algorithms (analyzing customer reactions and concerns)

[0112] CAIL (Cognitive AI Library) (AI customer avatar generation and simulation)

[0113] Python's Flask framework (generating virtual FAQs)

[0114] e-learning platform (e.g. Moodle) (learning content generation)

[0115] Adobe InDesign script (generate promotional tools)

[0116] Example of a system

[0117] Information gathering phase

[0118] The device uses a voice recorder to record conversations with customers. The recorded audio data (e.g., audio123.wav) is sent to a server via the network. At the same time, the server uses the Google Search API to collect user search ranking data (e.g., "new product features," "new product price," etc.).

[0119] Data analysis phase

[0120] The server uses the Google Cloud Speech-to-Text API to convert the received voice data into text data (e.g., "What are the features of the new product?"). It then uses TensorFlow to train an AI model using the collected voice and search data. It then uses CAIL to generate AI customer avatars based on multiple attributes and runs simulations. This simulation allows it to analyze what questions and complaints a particular customer avatar might have about the new service.

[0121] FAQ and tool generation phase

[0122] Based on the analysis results, the server generates a virtual FAQ using Python's Flask framework. For example, it creates an FAQ page containing answers to questions such as, "What are the features of the new product?". It also uses the same results to create learning content for e-learning platforms such as Moodle. This includes materials and videos with detailed explanations of the new product's features and sales strategies. It also uses Adobe InDesign scripts to automatically design promotional brochures and posters.

[0123] Results delivery phase

[0124] The server provides the generated virtual FAQs, learning content, and promotional tools to users via an interface module (e.g., a web portal). For example, a login page for the web portal can be created so that store staff can access the FAQs and learning content. It can also send a QR code or link to users via email, providing them in an easily accessible format.

[0125] Prompt Sentence Examples

[0126] "Please tell me more about the pricing for this service."

[0127] This system makes it possible to quickly and accurately grasp specific customer reactions and concerns, and effectively provide on-site staff with the information and tools they need.

[0128] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0129] Step 1:

[0130] The terminal uses a voice recorder to collect voice data. Specifically, the voice recorder records the conversation between the store staff and the customer while they are explaining about new products. The input is the conversation with the customer, and the output is the voice data (e.g., audio123.wav). After the voice data is collected, it is sent to the server via the network.

[0131] Step 2:

[0132] The server converts the received voice data into text data. Specifically, it uses the Google Cloud Speech-to-Text API to convert the voice data (input: audio123.wav) into text data (output: text data "What are the features of the new product?"). The converted text data is stored in the server's database.

[0133] Step 3:

[0134] The server collects search data from online search systems. Specifically, it uses the Google Search API to obtain search ranking data for keywords related to new products (input: API request). The output is the search keywords and their ranking data (e.g., "new product features," "new product price," etc.). This data is also stored in the server's database.

[0135] Step 4:

[0136] The server uses the collected voice data and search data to train an AI model. Specifically, it uses a machine learning library (e.g., TensorFlow) to train the AI ​​model based on natural language processing algorithms (input: text data and search data). The output is a trained AI model. This AI model has the ability to analyze customer responses and concerns.

[0137] Step 5:

[0138] The server uses the trained AI model to generate AI customer avatars based on multiple attributes and run simulations. Specifically, it uses CAIL (Cognitive AI Library) to generate multiple customer avatars based on attribute information such as age, gender, and purchase history (input: attribute information). The output is the generated AI customer avatar, which is then used to run simulations and predict customer reactions and concerns.

[0139] Step 6:

[0140] The server generates a virtual FAQ based on the simulation results. Specifically, it uses the Python Flask framework to create an FAQ page containing frequently asked questions and their answers (input: simulation results). The output is a virtual FAQ page (e.g., "What are the features of the new product?" and its answer).

[0141] Step 7:

[0142] The server generates learning content and promotional tools based on the analysis results. Specifically, it creates learning modules to be installed on e-learning platforms (e.g., Moodle) (input: simulation results), and automatically generates promotional brochures and posters using Adobe InDesign scripts (input: simulation results). The output is learning content and promotional tools (e.g., brochures and posters).

[0143] Step 8:

[0144] The server provides the generated virtual FAQs, learning content, and promotional tools to users via the interface module. Specifically, it creates a login page for a web portal from which store staff can access the FAQs and learning content (input: generated content and tools). The output is the content and tools provided in a form that users can use (e.g., web links, QR codes).

[0145] (Application example 1)

[0146] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0147] To improve the efficiency and accuracy of customer service, it is necessary to analyze customer reactions and concerns in real time and enable store staff to quickly respond appropriately based on that information. However, traditional systems struggled to efficiently integrate these functions, and lacked support tools to improve the quality of customer service. Furthermore, there were insufficient means to identify customer questions and concerns in advance when introducing new services, and to provide effective learning content and promotional tools. As a result, customer satisfaction declined, hindering sales growth.

[0148] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0149] In this invention, the server includes means for collecting voice data, means for converting the collected voice data into text data, means for collecting search data, means for analyzing customer responses and concerns using the voice data and the search data, means for generating a virtual FAQ based on the analysis results, means for generating learning content and promotional tools based on the analysis results, means for displaying the generated virtual FAQ, learning content, and promotional tools, and means for providing a smartphone application that supports customer service using the virtual FAQ, learning content, and promotional tools generated based on the analysis results. This enables real-time understanding of customer responses and concerns and prompt appropriate responses. Furthermore, by identifying customer questions and concerns in advance when introducing a new service and providing effective learning content and promotional tools, the burden on store staff can be reduced, leading to improved customer satisfaction and increased sales.

[0150] "Voice data" refers to acoustic signal data that records conversations between customers and store staff.

[0151] "Text data" refers to data obtained by converting voice data into text using voice recognition technology.

[0152] "Search Data" refers to data that includes customer online search behavior and ranking information.

[0153] "Customer reactions and concerns" refers to feedback and reactions such as interest, questions, and concerns that customers show in-store.

[0154] "Analysis" refers to the act of using AI models and natural language processing algorithms to analyze customer responses and concerns from voice and search data.

[0155] A "virtual FAQ" is a virtual FAQ (Frequently Asked Questions) that is automatically generated based on the analysis results and contains questions that customers frequently ask and their answers.

[0156] "Learning content" refers to educational materials and content provided to store staff to help them learn about new services.

[0157] "Sales promotional tools" are promotional materials and tools such as POPs and pamphlets used to promote the sale of products and services.

[0158] "Display" refers to the act of visually providing the generated virtual FAQs, learning content, and promotional tools to users.

[0159] A "smartphone application" is a software program that is installed on a smartphone device to assist with customer service.

[0160] Information gathering phase

[0161] To collect voice data, the server records conversations between customers and store staff using a voice recorder used in the store. The recorded voice data is sent to the server in real time. At the same time, the server obtains user search ranking data from an online search system, thereby collecting information related to the customer's interests and questions.

[0162] Data analysis phase

[0163] The server converts the received voice data into text data using speech recognition technology. Services such as Google Cloud Speech-to-Text and IBM Watson Speech to Text are used for speech recognition. Next, an AI model (e.g., BERT or GPT-4) is trained based on the voice data and search data. This AI model uses natural language processing algorithms to automatically analyze customer responses and concerns. The server also generates an AI model based on multiple attributes and performs simulations. Google Cloud Natural Language and Amazon Comprehend are commonly used NLP services.

[0164] FAQ and tool generation phase

[0165] The server generates a virtual FAQ based on the analysis results. This virtual FAQ contains frequently asked questions and their answers, and aggregates information useful for responding to customers. Based on the same results, the server also automatically generates learning content and promotional tools (e.g., POPs, pamphlets, etc.) for store staff to learn about new services. The generated virtual FAQ, learning content, and promotional tools are stored on the server.

[0166] Results delivery phase

[0167] The generated virtual FAQs, learning content, and promotional tools are provided to store staff via a smartphone application. Staff can view these through the application interface. This application enables staff to respond to customer questions quickly and accurately. Staff can also use the learning content to efficiently acquire knowledge about new services.

[0168] Specific examples

[0169] For example, when a physical store introduces a new electronic payment service, if a customer asks staff questions about how to use it, fees, or security, the store can use the app to get real-time answers.

[0170] Prompt Sentence Examples

[0171] Customer: "Can you show me how to use the new electronic payment service?"

[0172] Application: "The basic steps for using the new electronic payment service are to first download the app and then create an account. Detailed instructions can be found in the following FAQ."

[0173] In this way, the system of the present invention can grasp and analyze customer reactions and concerns in real time, creating an environment in which they can be responded to efficiently, thereby improving customer satisfaction and increasing sales.

[0174] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0175] Step 1: Collecting audio data

[0176] The user (store staff) uses a voice recorder to record conversations with customers to collect voice data. After the voice data is collected, it is sent to the server in real time. The input is the voice data, and the output is the voice data sent to the server.

[0177] Step 2: Convert audio data to text

[0178] The server converts the received voice data into text data using speech recognition technology. This process uses services such as Google Cloud Speech-to-Text and IBM Watson Speech to Text. The input is voice data and the output is text data. Specific operations include the process of converting voice data into a string of characters.

[0179] Step 3: Collect search data

[0180] The server retrieves user search ranking data from the online search system. This data indicates what questions and keywords customers frequently search. The input is the search data retrieved from the online search system, and the output is the search data used for analysis.

[0181] Step 4: Analyze customer reactions and concerns

[0182] The server trains an AI model (e.g., BERT or GPT-4) based on the received text data and search data to analyze customer reactions and concerns. It uses an NLP algorithm, such as Google Cloud Natural Language or Amazon Comprehend. The input is the text data and search data, and the output is the analysis results of customer reactions and concerns. Specific operations include inputting data, training the model, and analyzing reactions and concerns.

[0183] Step 5: Generate a virtual FAQ

[0184] The server generates a virtual FAQ based on the analysis results. This virtual FAQ contains frequently asked questions and their answers from customers. The input is the analysis results on customer reactions and concerns, and the output is the virtual FAQ. Specific operations include generating patterns of questions and answers.

[0185] Step 6: Generate learning content and collateral

[0186] The server automatically generates learning content and promotional tools based on the same analysis results. The generated learning content includes educational materials about new services, and the promotional tools include POP displays and pamphlets. The input is the analysis results regarding customer reactions and concerns, and the output is learning content and promotional tools. Specific operations include generating educational materials and promotional materials.

[0187] Step 7: Delivering results

[0188] The generated virtual FAQs, learning content, and promotional tools are provided to store staff via a smartphone application. Users (store staff) can use this application to access the generated information and quickly and accurately answer customer questions. The input is the virtual FAQs, learning content, and promotional tools, and the output is customer support based on them. Specific operations include displaying and using the information via the smartphone application.

[0189] Through these steps, the inventive system is able to grasp customer reactions and concerns in real time, enabling appropriate responses to be taken quickly.

[0190] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0191] The present invention combines a system that uses voice data and search data to analyze customer reactions and concerns, and generates and provides virtual FAQs, learning content, and sales promotion tools based on the results, with an emotion engine that recognizes user emotions. Specific embodiments of the present invention will be described below.

[0192] System Overview

[0193] Information gathering phase

[0194] The terminal uses a voice recorder used in the store to record conversations with customers. This conversation includes questions and feedback from customers. The server collects voice data from the voice recorder at regular intervals and receives it via the network.

[0195] The server sends the voice data to the emotion engine for emotion recognition. The voice data is then sent to the speech recognition engine for conversion into text data. The converted text data and the recognized emotion data are stored in a database. At the same time, the server obtains the user's search ranking data and stores this data in the database.

[0196] Data analysis phase

[0197] The server uses voice data, search ranking data, and sentiment data to begin training the AI ​​model, which uses natural language processing (NLP) algorithms to better analyze customer reactions and concerns.

[0198] The server generates AI customer avatars based on multiple attributes and simulates how each avatar will react and feel toward new services.

[0199] FAQ and tool generation phase

[0200] The server generates a virtual FAQ based on the simulation results, which contains questions and answers based on each customer avatar's reactions, concerns, and emotional data.

[0201] The server generates learning content based on the simulation results. This includes manuals and online learning materials to help store staff deepen their understanding of the new service. It also takes into account emotional data to identify points requiring special attention and specific countermeasures.

[0202] The server automatically generates promotional tools, such as POP displays and pamphlets, to effectively introduce and advertise new services.

[0203] Results delivery phase

[0204] The generated virtual FAQs, learning content, and sales promotion tools are provided to the store staff (users) via an interface module in an accessible state, allowing users to make the necessary preparations before introducing the service.

[0205] For example, when users (store staff) prepare to serve customers before introducing a new service, they can check FAQs for specific questions and points to note based on emotional data, making it easier to elicit positive customer reactions to the new service.

[0206] In this way, the system predicts customer reactions and concerns when introducing new services and provides appropriate responses, thereby increasing customer satisfaction and supporting the success of the service. The addition of an emotion engine makes it possible to more accurately grasp the customer's emotional state and respond in a way that reflects that information, resulting in even more effective service provision.

[0207] The processing flow will be explained below.

[0208] Specific flow of program processing

[0209] Information gathering phase

[0210] Step 1:

[0211] The terminal uses a voice recorder used in the store to record conversations with customers, including questions and feedback from the customers.

[0212] Step 2:

[0213] The server collects the audio data from the voice recorder at regular intervals, which involves the process of transferring the data over the network.

[0214] Step 3:

[0215] The server sends the collected voice data to an emotion engine to recognize the customer's emotions, for example, whether the customer is expressing negative emotions or interest in the service.

[0216] Step 4:

[0217] The server sends the voice data to a speech recognition engine and converts it into text data. The converted text data and the recognized emotion data are stored in a database.

[0218] Step 5:

[0219] The server retrieves user search ranking data, including search keywords and frequently asked questions related to the new service, and stores this information in a database.

[0220] Data analysis phase

[0221] Step 6:

[0222] The server then uses the collected voice data, search ranking data, and sentiment data to begin training the AI ​​model, using natural language processing (NLP) algorithms.

[0223] Step 7:

[0224] The server uses an AI model to analyze customer reactions and concerns, such as which parts of the service customers are likely to have questions about and what emotions they are likely to experience.

[0225] Step 8:

[0226] The server generates AI customer avatars based on multiple attributes, including age, gender, and region, and simulates how each avatar would react to new services.

[0227] FAQ and tool generation phase

[0228] Step 9:

[0229] Based on the simulation results, the server generates a virtual FAQ, which includes questions and answers based on each customer avatar's reactions, concerns, and emotional data.

[0230] Step 10:

[0231] The server generates learning content based on the simulation results. This includes manuals and online learning materials to help store staff deepen their understanding of the new service. It also takes into account emotional data to identify points requiring special attention and specific countermeasures.

[0232] Step 11:

[0233] The server automatically generates promotional tools, such as POP displays and pamphlets, to effectively introduce and advertise new services.

[0234] Results delivery phase

[0235] Step 12:

[0236] Users (store staff) can access the generated virtual FAQs, learning content, and sales promotion tools through the interface module, allowing users to make the necessary preparations before introducing new services.

[0237] Step 13:

[0238] When users actually introduce new services, they can use the various tools and content generated to respond to customers. For example, they can increase customer satisfaction by quickly responding to customer questions based on FAQs and implementing methods to alleviate customer concerns based on emotional data.

[0239] Example 2

[0240] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0241] Conventional customer support systems have difficulty accurately grasping customer reactions and emotions, making it difficult to provide appropriate responses quickly. Furthermore, there is a lack of technology to effectively utilize customer feedback and search data to generate services and promotional tools. This makes it difficult to improve customer satisfaction and support the successful introduction of services.

[0242] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0243] In this invention, the server includes means for collecting voice data, means for transmitting the collected voice data to an emotion engine and recognizing customer emotions, means for converting the emotion-recognized voice data into text data, means for storing the converted text data and the recognized emotion data in a database, means for collecting search data, means for training an AI model using the voice data, the emotion data, and the search data to analyze customer reactions and concerns, means for generating an AI customer avatar based on the analysis results and simulating reactions and emotions to a new service, means for generating a virtual FAQ based on the simulation results, means for generating learning content and promotional tools based on the simulation results, and means for displaying the generated virtual FAQ, learning content, and promotional tools. This enables accurate understanding of customer emotions and prompt provision of appropriate countermeasures based on the understanding. Furthermore, customer feedback and search data can be utilized to generate effective virtual FAQs, learning content, and promotional tools, thereby improving customer satisfaction and supporting successful service introduction.

[0244] "Voice data" refers to information recorded in digital format from the conversations and voices of people such as customers and staff.

[0245] "Emotion engine" is a general term for algorithms and technologies that analyze voice data and identify the emotional state contained within it (e.g., joy, anger, sadness, etc.).

[0246] "Text data" is information in the form of a string of characters converted from voice data by a voice recognition engine.

[0247] A "database" is a system or device for systematically storing and managing collected and converted data.

[0248] "Search Data" means records and statistical information about users' search behavior on the Internet and internal systems.

[0249] "AI Model" means an artificial intelligence model trained using machine learning and natural language processing algorithms and used to analyze customer responses and concerns.

[0250] "Customer avatars" are virtual customer profiles generated based on AI models to simulate reactions and emotions toward new services.

[0251] "Simulation results" are the predicted results of virtual reactions and emotions by AI customer avatars.

[0252] A "virtual FAQ" is a virtual document that compiles frequently asked questions from customers and their answers, generated based on the simulation results.

[0253] "Learning content" refers to educational materials and online learning materials that help store staff understand new services and use them to assist them in dealing with customers.

[0254] "Promotional tools" are promotional materials (e.g., POPs, pamphlets, etc.) used to effectively introduce and advertise new services.

[0255] An "interface module" is a system component or software that provides the generated virtual FAQs, learning content, and promotional tools in an accessible form to users.

[0256] "Users" are store staff and other related parties who operate the system and use the generated virtual FAQs, learning content, and promotional tools.

[0257] The present invention combines a system that uses voice data and search data to analyze customer reactions and concerns, and generates and provides virtual FAQs, learning content, and sales promotion tools based on the results, with an emotion engine that recognizes user emotions. Specific embodiments of the present invention will be described below.

[0258] 1. Information gathering phase

[0259] The terminal uses a voice recorder used in the store to record conversations with customers. This conversation includes questions and feedback from customers. The terminal sends this voice data to a server at regular intervals. The voice data is recorded with a timestamp, making it possible to determine the exact point in time at which customer feedback was given.

[0260] 2. Emotion and voice recognition phase

[0261] The server sends the received voice data to an emotion engine to recognize the customer's emotions. The emotion engine uses a voice analysis algorithm to identify emotional states such as "joy," "anger," and "sadness." The server then sends the voice data to a speech recognition engine, which converts it into text data. The converted text data is a string of characters that directly represents the content of the conversation, including specific questions and feedback.

[0262] 3. Data storage phase

[0263] The server stores the converted text data and the recognized emotion data in a database. The database is used to identify changes and patterns over time. At the same time, the server obtains users' search ranking data and stores this data in the database. This search data includes the keywords and topics that users most frequently searched for over a certain period of time.

[0264] 4. AI model training phase

[0265] The server trains an AI model using voice data, search ranking data, and sentiment data. The AI ​​model uses natural language processing (NLP) algorithms to more accurately analyze customer reactions and concerns. The AI ​​model also generates an AI customer avatar based on multiple attributes. The customer avatar has a virtual profile and simulates reactions and emotions toward new services.

[0266] 5. FAQ and tool generation phase

[0267] The server generates a virtual FAQ based on the simulation results. This FAQ includes questions and answers based on the reactions, concerns, and emotional data of each customer avatar. The server also generates learning content and promotional tools based on the simulation results. The learning content includes manuals and online learning materials to help store staff deepen their understanding of the new service. Points that require particular attention and specific countermeasures are also added, improving the quality of customer service. The promotional tools include POP displays and pamphlets to effectively introduce and promote the new service.

[0268] 6. Results delivery phase

[0269] The generated virtual FAQs, learning content, and sales promotion tools are provided to the store staff (users) via the interface module. This allows the users to make the necessary preparations before introducing the service. For example, when users (store staff) prepare to deal with customers before introducing a new service, they can check FAQs for specific questions and points to note based on emotion data. This makes it easier to elicit a positive customer response to the new service.

[0270] Specific examples

[0271] For example, when a user is preparing to introduce a new service, they can input a prompt such as, "Please tell us the questions that customers are most likely to ask when the new service is introduced. Also, please simulate appropriate answers and customer reactions." Based on this prompt, the server uses an AI model to generate the most appropriate virtual FAQ and provides questions and answers based on emotional data. In this way, users can smoothly introduce new services and increase customer satisfaction.

[0272] This system efficiently utilizes voice and search data to accurately grasp customer emotions and reactions, enabling it to provide prompt and appropriate responses. Furthermore, by combining a generative AI model with an emotion engine, the system achieves more effective service delivery.

[0273] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0274] Step 1:

[0275] The device records conversations with customers using a voice recorder.

[0276] Specific operation: The terminal activates a voice recorder installed in the store and records the conversation between the customer and the staff. The voice data is recorded with a timestamp and saved on the terminal.

[0277] Step 2:

[0278] The server sends the voice data to an emotion engine to recognize the customer's emotion.

[0279] Input: Audio data received from the device

[0280] Output: Recognized emotion data

[0281] Specific operation: The server periodically transmits voice data acquired from the device to the emotion engine. The emotion engine uses a voice analysis algorithm to identify emotional states such as "joy," "anger," and "sadness." The recognized emotions are returned to the server in data format.

[0282] Step 3:

[0283] The server sends the voice data to a voice recognition engine and converts it into text data.

[0284] Input: Audio data received from the device

[0285] Output: Converted text data

[0286] Specific operation: The server sends the voice data to a speech recognition engine, which converts the voice into text. For example, the voice saying "I want to know more about this new service" is converted into text "I want to know more about this new service."

[0287] Step 4:

[0288] The server stores the text data and emotion data in a database.

[0289] Input: converted text data and recognized emotion data

[0290] Output: Data stored in the database

[0291] Specific operation: The server stores the converted text data and emotion data in a database, including timestamps and conversation context.

[0292] Step 5:

[0293] The server retrieves the user's search ranking data and stores it in a database.

[0294] Input: Search data from websites or internal systems

[0295] Output: Search ranking data stored in a database

[0296] What it does: The server retrieves user search ranking data from the internet or internal systems and stores it in a database, including the most searched keywords and topics for the past week.

[0297] Step 6:

[0298] The server trains the AI ​​model using voice data, search ranking data, and emotion data.

[0299] Input: Voice data, search ranking data, emotion data

[0300] Output: A trained AI model

[0301] How it works: The server feeds this data to an AI model, which trains it with natural language processing (NLP) algorithms. The AI ​​model is trained to analyze customer reactions and concerns.

[0302] Step 7:

[0303] The server generates AI customer avatars and simulates their reactions and emotions to new services.

[0304] Input: A trained AI model

[0305] Output: Simulated customer avatar reactions and emotions

[0306] How it works: The server uses the trained AI model to generate multiple customer avatars, each with a virtual profile that simulates their reactions and emotions to the new service.

[0307] Step 8:

[0308] The server generates a virtual FAQ based on the simulation results.

[0309] Input: Simulation results

[0310] Output: The generated virtual FAQ

[0311] Specific operation: Based on the simulation results of the AI ​​customer avatar, the server automatically generates a virtual FAQ that compiles frequently asked questions from customers and their answers.

[0312] Step 9:

[0313] The server generates learning content and promotional tools based on the simulation results.

[0314] Input: Simulation results

[0315] Output: Generated learning content and promotional tools

[0316] Specific operation: Based on the simulation results of the AI ​​customer avatar, the server generates learning content (e.g., manuals, online learning materials) and promotional tools (e.g., POPs, pamphlets) for store staff.

[0317] Step 10:

[0318] The server provides the generated virtual FAQs, learning content, and promotional tools to the user via an interface module.

[0319] Input: Generated virtual FAQs, learning content, promotional tools

[0320] Output: Information provided to the user

[0321] Specific operation: The server sends the generated virtual FAQ, learning content, and sales promotion tools to the interface module, making them accessible to users. Through this, users (store staff) can view and use the information they need.

[0322] (Application example 2)

[0323] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0324] In factory work environments, there is a need to reduce the confusion and mistakes that occur when workers introduce new robots and equipment, thereby improving production efficiency and safety. However, conventional systems generally use standalone FAQs and operation manuals, which do not fully consider workers' reactions and emotions. As a result, it is difficult to quickly and accurately respond to actual problems and concerns on the shop floor.

[0325] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting voice data, means for converting the collected voice data into text data, means for collecting search data, means for analyzing the worker's reactions and concerns using the voice data and the search data, means for generating a virtual FAQ based on the analysis results, means for recognizing the worker's emotional state based on the analysis results and generating an operation manual, means for displaying the generated virtual FAQ, operation manual, and warning system, means for training an AI model based on multiple task data, and means for presenting improvement suggestions and points to note based on simulation results generated by the AI ​​model. This makes it possible to grasp the worker's emotions and actual reactions in real time and provide prompt and accurate countermeasures.

[0326] "Voice data" refers to voice information collected from workers and equipment at the work site.

[0327] "Means for collecting voice data" refers to devices and methods for recording and collecting voice from workers, such as voice data acquisition devices and microphones within the factory.

[0328] "Text data" refers to character information obtained by analyzing voice data.

[0329] "Means for converting voice data into text data" refers to a device or method that converts collected voice data into text information using voice recognition software or the like.

[0330] "Search data" refers to information such as workers' search behavior and frequently used keywords.

[0331] "Means for collecting search data" refers to devices and methods for acquiring and recording the keywords searched by workers and their behavioral history.

[0332] "Means for analyzing reactions and concerns" refers to algorithms or software that analyze worker reactions and concerns based on collected data.

[0333] "Virtual FAQ" refers to a collection of hypothetical questions and answers based on each worker's reactions and concerns.

[0334] "Means for generating virtual FAQs" refers to a device or method that automatically generates answers to problems or questions that workers may encounter based on the results of data analysis.

[0335] "Means for recognizing emotional states" refers to algorithms or software that identify and analyze emotions from the tone and content of a worker's voice.

[0336] An "operation manual" refers to a document or guideline that explains how to operate a device or system.

[0337] "Means for generating an operating manual" refers to a device or method for creating an optimized operating procedure manual based on emotion data and reaction data.

[0338] "Warning system" refers to a notification system that alerts workers to important points and dangers.

[0339] "Means for displaying the generated virtual FAQ, operation manual, and warning system" refers to devices and methods for providing information visually or audibly to workers.

[0340] "Means for training AI models" refers to algorithms and software used to train machine learning models using collected data.

[0341] "Means for presenting improvement suggestions and points to note" refers to devices and methods that provide workers with suggestions and points to note for improving work efficiency and safety based on the results generated by a trained AI model.

[0342] System program generation

[0343] In the system that realizes this application example, the server performs processing in the following steps. First, to collect voice data, it collects voice data from voice data capture devices and microphones in the factory. Next, it converts the collected voice data into text data using voice recognition software (e.g., Google Cloud Speech-to-Text). Simultaneously with the converted text data, it extracts emotion data using an emotion recognition engine (e.g., EmotionAPI). This data is then stored in a database (e.g., MySQL).

[0344] Next, the server collects search data, including keywords used by the user and their search history. This information is also stored in a database.

[0345] The server analyzes the worker's reactions and concerns using the collected voice data, text data, search data, and emotion data. It uses NLP algorithms (e.g., spaCy) and machine learning models (e.g., TensorFlow) for the analysis. It generates a virtual FAQ based on the analysis results. It also recognizes the worker's emotional state based on the emotion data and generates an operation manual. An interface module (e.g., React.js) is used to present these results to the worker visually or audibly.

[0346] Furthermore, the AI ​​model is trained based on multiple task data sets, and based on the simulation results generated from the trained AI model, it presents improvement suggestions and points to be aware of to the worker.

[0347] Data Processing Description

[0348] The server converts the voice data collected from the voice data capture device and microphone into text data using Google Cloud Speech-to-Text. At the same time, it extracts emotion data using the Emotion API. This text data and emotion data are stored in a database (MySQL). Search data is tracked by the worker's search behavior and stored in the database.

[0349] The server analyzes the collected data using spaCy and TensorFlow to generate a virtual FAQ. It also recognizes the worker's emotional state based on the emotion data and generates an operation manual or warning system as needed. The generated virtual FAQ, operation manual, and warning system are presented to the worker using an interface module (React.js).

[0350] Examples of concrete examples and prompts

[0351] For example, if a new robot is being introduced at a factory and staff are confused, the system can extract concerns such as "The machine suddenly stopped working" or "I don't understand the error message" from the conversation data and automatically generate FAQs, which are then displayed on the smart glasses' display to assist with countermeasures in real time.

[0352] Example prompt sentence:

[0353] "Analyze staff conversation data regarding the introduction of a new factory robot to identify sources of confusion and bewilderment. Also, generate FAQs and operation manuals that take their emotional state into account."

[0354] This makes it possible to grasp the emotions and actual reactions of workers in real time and provide quick and appropriate countermeasures.

[0355] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0356] Step 1:

[0357] Audio data collection

[0358] Voice data acquisition devices (microphones, etc.) in the factory collect the voices of workers in real time. The collected voice data is sent to a server via a network. The input is the voice data of the workers, and the output is the voice data sent to the server.

[0359] Step 2:

[0360] Converting audio data to text

[0361] The server converts the received voice data into text data using speech recognition software (Google Cloud Speech-to-Text). The input is voice data and the output is text data. Specifically, the server sends the voice file to the API and receives the returned text.

[0362] Step 3:

[0363] Extracting Emotional Data

[0364] After obtaining text data from the voice data, the server uses an emotion recognition engine (Emotion API) to extract emotional data. The input is voice data and the output is emotional data. Specifically, the voice data is sent to the emotion recognition API, and emotion labels and numerical values ​​are received as analysis results.

[0365] Step 4:

[0366] Data storage

[0367] The server stores the converted text data and extracted emotion data in a database (MySQL). The input is text data and emotion data, and the output is the data stored in the database. Specifically, it executes SQL queries to insert each data into the appropriate table.

[0368] Step 5:

[0369] Search data collection

[0370] The server collects the keywords searched by the worker and their behavioral history and stores them in a database. The input is the search query, and the output is the search data stored in the database. Specifically, the server captures the search history and inserts it into the appropriate table.

[0371] Step 6:

[0372] Analysis of reactions and concerns

[0373] The server uses the collected voice, text, emotion, and search data to analyze the worker's reactions and concerns using an NLP algorithm (spaCy) and a machine learning model (TensorFlow). The input is the aforementioned data set, and the output is the analysis results. Specifically, the text data is tokenized and input into the model to obtain prediction results.

[0374] Step 7:

[0375] Virtual FAQ generation

[0376] The server generates a virtual FAQ based on the analysis results of reactions and concerns. The input is the analysis results, and the output is the virtual FAQ. Specifically, the analysis results are applied to a pre-prepared template to generate an FAQ document.

[0377] Step 8:

[0378] Generate operation manuals

[0379] The server recognizes the worker's emotional state based on the emotional data and generates an operation manual. The input is the emotional data and analysis results, and the output is the operation manual. Specifically, it generates a manual document with additional precautions and procedures according to the worker's emotional state.

[0380] Step 9:

[0381] Generate an alert system

[0382] The server generates a warning system based on the generated operation manual and emotion data. The input is the operation manual and emotion data, and the output is a warning message. Specifically, it sets up notifications to notify users of dangerous situations and important precautions in real time.

[0383] Step 10:

[0384] Presentation of information

[0385] Users use an interface module (React.js) to visually or audibly access and display the generated virtual FAQ, operation manual, and warning system. The input is the various documents generated, and the output is the information presented visually or audibly. Specifically, the information is displayed through a web application.

[0386] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0387] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0388] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0389] [Second embodiment]

[0390] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0391] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0392] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0393] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0394] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0395] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0396] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0397] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0398] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0399] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0400] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0401] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0402] The present invention is a system that uses voice data and search data to analyze customer reactions and concerns, and generates and provides virtual FAQs, learning content, and sales promotion tools based on the results. This system is implemented as follows.

[0403] System Overview

[0404] Information gathering phase

[0405] The terminal records conversations with customers using a voice recorder used in the store. This voice data is sent to multiple servers via the network. At the same time, the servers obtain user search ranking data from an online search system.

[0406] Data analysis phase

[0407] The server first converts the received voice data into text data using speech recognition. It then trains an AI model based on the voice data and search data. This AI model uses natural language processing (NLP) algorithms to automatically analyze customer responses and concerns. The server then generates and simulates an AI customer avatar based on multiple attributes.

[0408] FAQ and tool generation phase

[0409] Based on the simulation results, the server generates a virtual FAQ, which includes frequently asked questions and their answers. It also generates educational content and promotional tools (POPs, pamphlets, etc.) for store staff to learn about new services.

[0410] Results delivery phase

[0411] The generated virtual FAQs, learning content, and sales promotion tools are provided to the store staff via an interface module in an accessible state, allowing users to understand customer reactions and concerns before implementing the service and prepare appropriate countermeasures in advance.

[0412] Specific examples

[0413] Collecting and analyzing customer interaction data

[0414] Users (store staff) use voice recorders to record their daily conversations with customers. For example, they can record conversations while explaining a new service. The data is sent to a server in real time and converted into text data using voice recognition technology.

[0415] Search data collection and analysis

[0416] At the same time, the server obtains search rankings for keywords related to the new service from the search system, including information on what questions users frequently search.

[0417] Training and simulating AI models

[0418] The server uses the collected voice and search data to train an AI model that analyzes likely customer reactions and concerns, such as the likelihood that many customers will have questions about the pricing structure of a particular service.

[0419] Virtual FAQ and tool generation

[0420] Based on the analysis results, the server automatically generates virtual FAQs, learning content, and sales promotion tools, allowing users to check specific questions and answers and what they need to learn in advance, while also providing marketing materials to encourage purchases.

[0421] In this way, the system provides a concrete means for anticipating customer reactions and concerns when introducing new services and providing appropriate responses.

[0422] The processing flow will be explained below.

[0423] Specific flow of program processing

[0424] Information gathering phase

[0425] Step 1:

[0426] The terminal uses a voice recorder used in the store to record conversations with customers, including questions and feedback from the customers.

[0427] Step 2:

[0428] The server collects the audio data from the voice recorder at regular intervals, which involves the process of transferring the data over the network.

[0429] Step 3:

[0430] The server sends the voice data to a speech recognition engine and converts it into text data, which is then stored in a database.

[0431] Step 4:

[0432] The server retrieves user search ranking data, including search keywords and frequently asked questions related to the new service, and stores this information in a database.

[0433] Data analysis phase

[0434] Step 5:

[0435] The server takes in the voice data and search ranking data and begins training the AI ​​model, which uses natural language processing (NLP) algorithms.

[0436] Step 6:

[0437] The server uses an AI model to analyze customer reactions and concerns, for example, determining which parts of the service customers are likely to have questions about and what their concerns are.

[0438] Step 7:

[0439] The server generates AI customer avatars based on multiple attributes, including age, gender, and region, and simulates how each avatar would react to new services.

[0440] FAQ and tool generation phase

[0441] Step 8:

[0442] The server generates a virtual FAQ based on the simulation results, which includes frequently asked questions and answers derived from the reactions and concerns of each customer avatar.

[0443] Step 9:

[0444] The server generates learning content based on the simulation results, similar to the virtual FAQ, including manuals and online learning materials to help store staff deepen their understanding of the new service.

[0445] Step 10:

[0446] The server automatically generates promotional tools, such as POP displays and pamphlets, to effectively introduce and advertise new services.

[0447] Results delivery phase

[0448] Step 11:

[0449] Users (store staff) can access the generated virtual FAQs, learning content, and sales promotion tools through the interface module, allowing users to make the necessary preparations before introducing new services.

[0450] Step 12:

[0451] When users actually implement new services, they can use the generated tools and content to respond to customers, thereby increasing customer satisfaction.

[0452] Example 1

[0453] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0454] Today's consumers have diverse and complex needs, and it is essential to accurately understand those needs and provide them with appropriate information. However, traditional methods make it difficult to fully grasp customers' specific reactions and concerns, making it impossible to generate effective FAQs, learning content, and sales promotion tools. Furthermore, manually analyzing voice data and search data is laborious and time-consuming, placing a significant burden on on-site staff. A system that can solve these problems is needed.

[0455] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0456] In this invention, the server includes means for collecting voice data, means for converting the collected voice data into text data, means for collecting search data, means for training an AI model using a natural language processing algorithm to analyze customer reactions and concerns using the voice data and the search data, means for generating a virtual FAQ based on the analysis results, means for generating learning content and promotional tools based on the analysis results, means for generating an AI customer avatar based on multiple attributes and performing a simulation, and means for displaying the generated virtual FAQ, learning content, and promotional tools. This makes it possible to quickly and accurately grasp specific customer reactions and concerns and effectively provide the information and tools needed by on-site staff.

[0457] 1. "Voice data" refers to data that represents acoustic signals in digital format, and is a record of conversations with customers, etc.

[0458] 2. "Text data" means data that has been converted from audio data into textual information and is expressed in a format that can be analyzed using natural language processing.

[0459] 3. "Search Data" means data including ranking information for keywords and search queries that users search on the Internet or internal systems.

[0460] 4. "Natural language processing algorithms" refers to the algorithms used by AI models to analyze text data, including technologies such as speech recognition and semantic analysis.

[0461] 5. "AI Model" means a model of artificial intelligence trained using machine learning techniques and designed to perform specific tasks automatically.

[0462] 6. "Virtual FAQ" refers to automatically generated questions and answers frequently asked by customers, including information to resolve customer concerns.

[0463] 7. "Learning Content" refers to educational materials that store staff and customers can use to learn about new services and products, including text, images, videos, etc.

[0464] 8. "Promotional tools" are materials and advertising materials used to effectively introduce new services and products to customers, including posters, pamphlets, etc.

[0465] 9. An "AI customer avatar" is a virtual customer character generated based on various attribute information and is a model for conducting simulations.

[0466] 10. "Simulation" is a methodology for using AI customer avatars to recreate and analyze anticipated customer reactions and concerns.

[0467] This invention is a system that uses voice data and search data to analyze customer reactions and concerns, and generates and provides virtual FAQs, learning content, and sales promotion tools based on the results. This system is implemented as follows.

[0468] System Overview

[0469] The system includes means for collecting voice data and converting it into text data, means for collecting search data, means for analyzing customer responses and concerns, means for generating virtual FAQs, learning content, and promotional tools, and means for displaying them.

[0470] Hardware and software used

[0471] Terminal: Voice recorder (to record conversations with customers), network connection device (to transmit voice data)

[0472] Server: High-performance computer (data processing and storage), software (data conversion and analysis)

[0473] Google Cloud Speech-to-Text API (converts voice data into text data)

[0474] Machine learning libraries (e.g., TensorFlow) (used to train AI models)

[0475] Natural language processing algorithms (analyzing customer reactions and concerns)

[0476] CAIL (Cognitive AI Library) (AI customer avatar generation and simulation)

[0477] Python's Flask framework (generating virtual FAQs)

[0478] e-learning platform (e.g. Moodle) (learning content generation)

[0479] Adobe InDesign script (generate promotional tools)

[0480] Example of a system

[0481] Information gathering phase

[0482] The device uses a voice recorder to record conversations with customers. The recorded audio data (e.g., audio123.wav) is sent to a server via the network. At the same time, the server uses the Google Search API to collect user search ranking data (e.g., "new product features," "new product price," etc.).

[0483] Data analysis phase

[0484] The server uses the Google Cloud Speech-to-Text API to convert the received voice data into text data (e.g., "What are the features of the new product?"). It then uses TensorFlow to train an AI model using the collected voice and search data. It then uses CAIL to generate AI customer avatars based on multiple attributes and runs simulations. This simulation allows it to analyze what questions and complaints a particular customer avatar might have about the new service.

[0485] FAQ and tool generation phase

[0486] Based on the analysis results, the server generates a virtual FAQ using Python's Flask framework. For example, it creates an FAQ page containing answers to questions such as, "What are the features of the new product?". It also uses the same results to create learning content for e-learning platforms such as Moodle. This includes materials and videos with detailed explanations of the new product's features and sales strategies. It also uses Adobe InDesign scripts to automatically design promotional brochures and posters.

[0487] Results delivery phase

[0488] The server provides the generated virtual FAQs, learning content, and promotional tools to users via an interface module (e.g., a web portal). For example, a login page for the web portal can be created so that store staff can access the FAQs and learning content. It can also send a QR code or link to users via email, providing them in an easily accessible format.

[0489] Prompt Sentence Examples

[0490] "Please tell me more about the pricing for this service."

[0491] This system makes it possible to quickly and accurately grasp specific customer reactions and concerns, and effectively provide on-site staff with the information and tools they need.

[0492] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0493] Step 1:

[0494] The terminal uses a voice recorder to collect voice data. Specifically, the voice recorder records the conversation between the store staff and the customer while they are explaining about new products. The input is the conversation with the customer, and the output is the voice data (e.g., audio123.wav). After the voice data is collected, it is sent to the server via the network.

[0495] Step 2:

[0496] The server converts the received voice data into text data. Specifically, it uses the Google Cloud Speech-to-Text API to convert the voice data (input: audio123.wav) into text data (output: text data "What are the features of the new product?"). The converted text data is stored in the server's database.

[0497] Step 3:

[0498] The server collects search data from online search systems. Specifically, it uses the Google Search API to obtain search ranking data for keywords related to new products (input: API request). The output is the search keywords and their ranking data (e.g., "new product features," "new product price," etc.). This data is also stored in the server's database.

[0499] Step 4:

[0500] The server uses the collected voice data and search data to train an AI model. Specifically, it uses a machine learning library (e.g., TensorFlow) to train the AI ​​model based on natural language processing algorithms (input: text data and search data). The output is a trained AI model. This AI model has the ability to analyze customer responses and concerns.

[0501] Step 5:

[0502] The server uses the trained AI model to generate AI customer avatars based on multiple attributes and run simulations. Specifically, it uses CAIL (Cognitive AI Library) to generate multiple customer avatars based on attribute information such as age, gender, and purchase history (input: attribute information). The output is the generated AI customer avatar, which is then used to run simulations and predict customer reactions and concerns.

[0503] Step 6:

[0504] The server generates a virtual FAQ based on the simulation results. Specifically, it uses the Python Flask framework to create an FAQ page containing frequently asked questions and their answers (input: simulation results). The output is a virtual FAQ page (e.g., "What are the features of the new product?" and its answer).

[0505] Step 7:

[0506] The server generates learning content and promotional tools based on the analysis results. Specifically, it creates learning modules to be installed on e-learning platforms (e.g., Moodle) (input: simulation results), and automatically generates promotional brochures and posters using Adobe InDesign scripts (input: simulation results). The output is learning content and promotional tools (e.g., brochures and posters).

[0507] Step 8:

[0508] The server provides the generated virtual FAQs, learning content, and promotional tools to users via the interface module. Specifically, it creates a login page for a web portal from which store staff can access the FAQs and learning content (input: generated content and tools). The output is the content and tools provided in a form that users can use (e.g., web links, QR codes).

[0509] (Application example 1)

[0510] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0511] To improve the efficiency and accuracy of customer service, it is necessary to analyze customer reactions and concerns in real time and enable store staff to quickly respond appropriately based on that information. However, traditional systems struggled to efficiently integrate these functions, and lacked support tools to improve the quality of customer service. Furthermore, there were insufficient means to identify customer questions and concerns in advance when introducing new services, and to provide effective learning content and promotional tools. As a result, customer satisfaction declined, hindering sales growth.

[0512] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0513] In this invention, the server includes means for collecting voice data, means for converting the collected voice data into text data, means for collecting search data, means for analyzing customer responses and concerns using the voice data and the search data, means for generating a virtual FAQ based on the analysis results, means for generating learning content and promotional tools based on the analysis results, means for displaying the generated virtual FAQ, learning content, and promotional tools, and means for providing a smartphone application that supports customer service using the virtual FAQ, learning content, and promotional tools generated based on the analysis results. This enables real-time understanding of customer responses and concerns and prompt appropriate responses. Furthermore, by identifying customer questions and concerns in advance when introducing a new service and providing effective learning content and promotional tools, the burden on store staff can be reduced, leading to improved customer satisfaction and increased sales.

[0514] "Voice data" refers to acoustic signal data that records conversations between customers and store staff.

[0515] "Text data" refers to data obtained by converting voice data into text using voice recognition technology.

[0516] "Search Data" refers to data that includes customer online search behavior and ranking information.

[0517] "Customer reactions and concerns" refers to feedback and reactions such as interest, questions, and concerns that customers show in-store.

[0518] "Analysis" refers to the act of using AI models and natural language processing algorithms to analyze customer responses and concerns from voice and search data.

[0519] A "virtual FAQ" is a virtual FAQ (Frequently Asked Questions) that is automatically generated based on the analysis results and contains questions that customers frequently ask and their answers.

[0520] "Learning content" refers to educational materials and content provided to store staff to help them learn about new services.

[0521] "Sales promotional tools" are promotional materials and tools such as POPs and pamphlets used to promote the sale of products and services.

[0522] "Display" refers to the act of visually providing the generated virtual FAQs, learning content, and promotional tools to users.

[0523] A "smartphone application" is a software program that is installed on a smartphone device to assist with customer service.

[0524] Information gathering phase

[0525] To collect voice data, the server records conversations between customers and store staff using a voice recorder used in the store. The recorded voice data is sent to the server in real time. At the same time, the server obtains user search ranking data from an online search system, thereby collecting information related to the customer's interests and questions.

[0526] Data analysis phase

[0527] The server converts the received voice data into text data using speech recognition technology. Services such as Google Cloud Speech-to-Text and IBM Watson Speech to Text are used for speech recognition. Next, an AI model (e.g., BERT or GPT-4) is trained based on the voice data and search data. This AI model uses natural language processing algorithms to automatically analyze customer responses and concerns. The server also generates an AI model based on multiple attributes and performs simulations. Google Cloud Natural Language and Amazon Comprehend are commonly used NLP services.

[0528] FAQ and tool generation phase

[0529] The server generates a virtual FAQ based on the analysis results. This virtual FAQ contains frequently asked questions and their answers, and aggregates information useful for responding to customers. Based on the same results, the server also automatically generates learning content and promotional tools (e.g., POPs, pamphlets, etc.) for store staff to learn about new services. The generated virtual FAQ, learning content, and promotional tools are stored on the server.

[0530] Results delivery phase

[0531] The generated virtual FAQs, learning content, and promotional tools are provided to store staff via a smartphone application. Staff can view these through the application interface. This application enables staff to respond to customer questions quickly and accurately. Staff can also use the learning content to efficiently acquire knowledge about new services.

[0532] Specific examples

[0533] For example, when a physical store introduces a new electronic payment service, if a customer asks staff questions about how to use it, fees, or security, the store can use the app to get real-time answers.

[0534] Prompt Sentence Examples

[0535] Customer: "Can you show me how to use the new electronic payment service?"

[0536] Application: "The basic steps for using the new electronic payment service are to first download the app and then create an account. Detailed instructions can be found in the following FAQ."

[0537] In this way, the system of the present invention can grasp and analyze customer reactions and concerns in real time, creating an environment in which they can be responded to efficiently, thereby improving customer satisfaction and increasing sales.

[0538] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0539] Step 1: Collecting audio data

[0540] The user (store staff) uses a voice recorder to record conversations with customers to collect voice data. After the voice data is collected, it is sent to the server in real time. The input is the voice data, and the output is the voice data sent to the server.

[0541] Step 2: Convert audio data to text

[0542] The server converts the received voice data into text data using speech recognition technology. This process uses services such as Google Cloud Speech-to-Text and IBM Watson Speech to Text. The input is voice data and the output is text data. Specific operations include the process of converting voice data into a string of characters.

[0543] Step 3: Collect search data

[0544] The server retrieves user search ranking data from the online search system. This data indicates what questions and keywords customers frequently search. The input is the search data retrieved from the online search system, and the output is the search data used for analysis.

[0545] Step 4: Analyze customer reactions and concerns

[0546] The server trains an AI model (e.g., BERT or GPT-4) based on the received text data and search data to analyze customer reactions and concerns. It uses an NLP algorithm, such as Google Cloud Natural Language or Amazon Comprehend. The input is the text data and search data, and the output is the analysis results of customer reactions and concerns. Specific operations include inputting data, training the model, and analyzing reactions and concerns.

[0547] Step 5: Generate a virtual FAQ

[0548] The server generates a virtual FAQ based on the analysis results. This virtual FAQ contains frequently asked questions and their answers from customers. The input is the analysis results on customer reactions and concerns, and the output is the virtual FAQ. Specific operations include generating patterns of questions and answers.

[0549] Step 6: Generate learning content and collateral

[0550] The server automatically generates learning content and promotional tools based on the same analysis results. The generated learning content includes educational materials about new services, and the promotional tools include POP displays and pamphlets. The input is the analysis results regarding customer reactions and concerns, and the output is learning content and promotional tools. Specific operations include generating educational materials and promotional materials.

[0551] Step 7: Delivering results

[0552] The generated virtual FAQs, learning content, and promotional tools are provided to store staff via a smartphone application. Users (store staff) can use this application to access the generated information and quickly and accurately answer customer questions. The input is the virtual FAQs, learning content, and promotional tools, and the output is customer support based on them. Specific operations include displaying and using the information via the smartphone application.

[0553] Through these steps, the inventive system is able to grasp customer reactions and concerns in real time, enabling appropriate responses to be taken quickly.

[0554] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0555] The present invention combines a system that uses voice data and search data to analyze customer reactions and concerns, and generates and provides virtual FAQs, learning content, and sales promotion tools based on the results, with an emotion engine that recognizes user emotions. Specific embodiments of the present invention will be described below.

[0556] System Overview

[0557] Information gathering phase

[0558] The terminal uses a voice recorder used in the store to record conversations with customers. This conversation includes questions and feedback from customers. The server collects voice data from the voice recorder at regular intervals and receives it via the network.

[0559] The server sends the voice data to the emotion engine for emotion recognition. The voice data is then sent to the speech recognition engine for conversion into text data. The converted text data and the recognized emotion data are stored in a database. At the same time, the server obtains the user's search ranking data and stores this data in the database.

[0560] Data analysis phase

[0561] The server uses voice data, search ranking data, and sentiment data to begin training the AI ​​model, which uses natural language processing (NLP) algorithms to better analyze customer reactions and concerns.

[0562] The server generates AI customer avatars based on multiple attributes and simulates how each avatar will react and feel toward new services.

[0563] FAQ and tool generation phase

[0564] The server generates a virtual FAQ based on the simulation results, which contains questions and answers based on each customer avatar's reactions, concerns, and emotional data.

[0565] The server generates learning content based on the simulation results. This includes manuals and online learning materials to help store staff deepen their understanding of the new service. It also takes into account emotional data to identify points requiring special attention and specific countermeasures.

[0566] The server automatically generates promotional tools, such as POP displays and pamphlets, to effectively introduce and advertise new services.

[0567] Results delivery phase

[0568] The generated virtual FAQs, learning content, and sales promotion tools are provided to the store staff (users) via an interface module in an accessible state, allowing users to make the necessary preparations before introducing the service.

[0569] For example, when users (store staff) prepare to serve customers before introducing a new service, they can check FAQs for specific questions and points to note based on emotional data, making it easier to elicit positive customer reactions to the new service.

[0570] In this way, the system predicts customer reactions and concerns when introducing new services and provides appropriate responses, thereby increasing customer satisfaction and supporting the success of the service. The addition of an emotion engine makes it possible to more accurately grasp the customer's emotional state and respond in a way that reflects that information, resulting in even more effective service provision.

[0571] The processing flow will be explained below.

[0572] Specific flow of program processing

[0573] Information gathering phase

[0574] Step 1:

[0575] The terminal uses a voice recorder used in the store to record conversations with customers, including questions and feedback from the customers.

[0576] Step 2:

[0577] The server collects the audio data from the voice recorder at regular intervals, which involves the process of transferring the data over the network.

[0578] Step 3:

[0579] The server sends the collected voice data to an emotion engine to recognize the customer's emotions, for example, whether the customer is expressing negative emotions or interest in the service.

[0580] Step 4:

[0581] The server sends the voice data to a speech recognition engine and converts it into text data. The converted text data and the recognized emotion data are stored in a database.

[0582] Step 5:

[0583] The server retrieves user search ranking data, including search keywords and frequently asked questions related to the new service, and stores this information in a database.

[0584] Data analysis phase

[0585] Step 6:

[0586] The server then uses the collected voice data, search ranking data, and sentiment data to begin training the AI ​​model, using natural language processing (NLP) algorithms.

[0587] Step 7:

[0588] The server uses an AI model to analyze customer reactions and concerns, such as which parts of the service customers are likely to have questions about and what emotions they are likely to experience.

[0589] Step 8:

[0590] The server generates AI customer avatars based on multiple attributes, including age, gender, and region, and simulates how each avatar would react to new services.

[0591] FAQ and tool generation phase

[0592] Step 9:

[0593] Based on the simulation results, the server generates a virtual FAQ, which includes questions and answers based on each customer avatar's reactions, concerns, and emotional data.

[0594] Step 10:

[0595] The server generates learning content based on the simulation results. This includes manuals and online learning materials to help store staff deepen their understanding of the new service. It also takes into account emotional data to identify points requiring special attention and specific countermeasures.

[0596] Step 11:

[0597] The server automatically generates promotional tools, such as POP displays and pamphlets, to effectively introduce and advertise new services.

[0598] Results delivery phase

[0599] Step 12:

[0600] Users (store staff) can access the generated virtual FAQs, learning content, and sales promotion tools through the interface module, allowing users to make the necessary preparations before introducing new services.

[0601] Step 13:

[0602] When users actually introduce new services, they can use the various tools and content generated to respond to customers. For example, they can increase customer satisfaction by quickly responding to customer questions based on FAQs and implementing methods to alleviate customer concerns based on emotional data.

[0603] Example 2

[0604] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0605] Conventional customer support systems have difficulty accurately grasping customer reactions and emotions, making it difficult to provide appropriate responses quickly. Furthermore, there is a lack of technology to effectively utilize customer feedback and search data to generate services and promotional tools. This makes it difficult to improve customer satisfaction and support the successful introduction of services.

[0606] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0607] In this invention, the server includes means for collecting voice data, means for transmitting the collected voice data to an emotion engine and recognizing customer emotions, means for converting the emotion-recognized voice data into text data, means for storing the converted text data and the recognized emotion data in a database, means for collecting search data, means for training an AI model using the voice data, the emotion data, and the search data to analyze customer reactions and concerns, means for generating an AI customer avatar based on the analysis results and simulating reactions and emotions to a new service, means for generating a virtual FAQ based on the simulation results, means for generating learning content and promotional tools based on the simulation results, and means for displaying the generated virtual FAQ, learning content, and promotional tools. This enables accurate understanding of customer emotions and prompt provision of appropriate countermeasures based on the understanding. Furthermore, customer feedback and search data can be utilized to generate effective virtual FAQs, learning content, and promotional tools, thereby improving customer satisfaction and supporting successful service introduction.

[0608] "Voice data" refers to information recorded in digital format from the conversations and voices of people such as customers and staff.

[0609] "Emotion engine" is a general term for algorithms and technologies that analyze voice data and identify the emotional state contained within it (e.g., joy, anger, sadness, etc.).

[0610] "Text data" is information in the form of a string of characters converted from voice data by a voice recognition engine.

[0611] A "database" is a system or device for systematically storing and managing collected and converted data.

[0612] "Search Data" means records and statistical information about users' search behavior on the Internet and internal systems.

[0613] "AI Model" means an artificial intelligence model trained using machine learning and natural language processing algorithms and used to analyze customer responses and concerns.

[0614] "Customer avatars" are virtual customer profiles generated based on AI models to simulate reactions and emotions toward new services.

[0615] "Simulation results" are the predicted results of virtual reactions and emotions by AI customer avatars.

[0616] A "virtual FAQ" is a virtual document that compiles frequently asked questions from customers and their answers, generated based on the simulation results.

[0617] "Learning content" refers to educational materials and online learning materials that help store staff understand new services and use them to assist them in dealing with customers.

[0618] "Promotional tools" are promotional materials (e.g., POPs, pamphlets, etc.) used to effectively introduce and advertise new services.

[0619] An "interface module" is a system component or software that provides the generated virtual FAQs, learning content, and promotional tools in an accessible form to users.

[0620] "Users" are store staff and other related parties who operate the system and use the generated virtual FAQs, learning content, and promotional tools.

[0621] The present invention combines a system that uses voice data and search data to analyze customer reactions and concerns, and generates and provides virtual FAQs, learning content, and sales promotion tools based on the results, with an emotion engine that recognizes user emotions. Specific embodiments of the present invention will be described below.

[0622] 1. Information gathering phase

[0623] The terminal uses a voice recorder used in the store to record conversations with customers. This conversation includes questions and feedback from customers. The terminal sends this voice data to a server at regular intervals. The voice data is recorded with a timestamp, making it possible to determine the exact point in time at which customer feedback was given.

[0624] 2. Emotion and voice recognition phase

[0625] The server sends the received voice data to an emotion engine to recognize the customer's emotions. The emotion engine uses a voice analysis algorithm to identify emotional states such as "joy," "anger," and "sadness." The server then sends the voice data to a speech recognition engine, which converts it into text data. The converted text data is a string of characters that directly represents the content of the conversation, including specific questions and feedback.

[0626] 3. Data storage phase

[0627] The server stores the converted text data and the recognized emotion data in a database. The database is used to identify changes and patterns over time. At the same time, the server obtains users' search ranking data and stores this data in the database. This search data includes the keywords and topics that users most frequently searched for over a certain period of time.

[0628] 4. AI model training phase

[0629] The server trains an AI model using voice data, search ranking data, and sentiment data. The AI ​​model uses natural language processing (NLP) algorithms to more accurately analyze customer reactions and concerns. The AI ​​model also generates an AI customer avatar based on multiple attributes. The customer avatar has a virtual profile and simulates reactions and emotions toward new services.

[0630] 5. FAQ and tool generation phase

[0631] The server generates a virtual FAQ based on the simulation results. This FAQ includes questions and answers based on the reactions, concerns, and emotional data of each customer avatar. The server also generates learning content and promotional tools based on the simulation results. The learning content includes manuals and online learning materials to help store staff deepen their understanding of the new service. Points that require particular attention and specific countermeasures are also added, improving the quality of customer service. The promotional tools include POP displays and pamphlets to effectively introduce and promote the new service.

[0632] 6. Results delivery phase

[0633] The generated virtual FAQs, learning content, and sales promotion tools are provided to the store staff (users) via the interface module. This allows the users to make the necessary preparations before introducing the service. For example, when users (store staff) prepare to deal with customers before introducing a new service, they can check FAQs for specific questions and points to note based on emotion data. This makes it easier to elicit a positive customer response to the new service.

[0634] Specific examples

[0635] For example, when a user is preparing to introduce a new service, they can input a prompt such as, "Please tell us the questions that customers are most likely to ask when the new service is introduced. Also, please simulate appropriate answers and customer reactions." Based on this prompt, the server uses an AI model to generate the most appropriate virtual FAQ and provides questions and answers based on emotional data. In this way, users can smoothly introduce new services and increase customer satisfaction.

[0636] This system efficiently utilizes voice and search data to accurately grasp customer emotions and reactions, enabling it to provide prompt and appropriate responses. Furthermore, by combining a generative AI model with an emotion engine, the system achieves more effective service delivery.

[0637] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0638] Step 1:

[0639] The device records conversations with customers using a voice recorder.

[0640] Specific operation: The terminal activates a voice recorder installed in the store and records the conversation between the customer and the staff. The voice data is recorded with a timestamp and saved on the terminal.

[0641] Step 2:

[0642] The server sends the voice data to an emotion engine to recognize the customer's emotion.

[0643] Input: Audio data received from the device

[0644] Output: Recognized emotion data

[0645] Specific operation: The server periodically transmits voice data acquired from the device to the emotion engine. The emotion engine uses a voice analysis algorithm to identify emotional states such as "joy," "anger," and "sadness." The recognized emotions are returned to the server in data format.

[0646] Step 3:

[0647] The server sends the voice data to a voice recognition engine and converts it into text data.

[0648] Input: Audio data received from the device

[0649] Output: Converted text data

[0650] Specific operation: The server sends the voice data to a speech recognition engine, which converts the voice into text. For example, the voice saying "I want to know more about this new service" is converted into text "I want to know more about this new service."

[0651] Step 4:

[0652] The server stores the text data and emotion data in a database.

[0653] Input: converted text data and recognized emotion data

[0654] Output: Data stored in the database

[0655] Specific operation: The server stores the converted text data and emotion data in a database, including timestamps and conversation context.

[0656] Step 5:

[0657] The server retrieves the user's search ranking data and stores it in a database.

[0658] Input: Search data from websites or internal systems

[0659] Output: Search ranking data stored in a database

[0660] What it does: The server retrieves user search ranking data from the internet or internal systems and stores it in a database, including the most searched keywords and topics for the past week.

[0661] Step 6:

[0662] The server trains the AI ​​model using voice data, search ranking data, and emotion data.

[0663] Input: Voice data, search ranking data, emotion data

[0664] Output: A trained AI model

[0665] How it works: The server feeds this data to an AI model, which trains it with natural language processing (NLP) algorithms. The AI ​​model is trained to analyze customer reactions and concerns.

[0666] Step 7:

[0667] The server generates AI customer avatars and simulates their reactions and emotions to new services.

[0668] Input: A trained AI model

[0669] Output: Simulated customer avatar reactions and emotions

[0670] How it works: The server uses the trained AI model to generate multiple customer avatars, each with a virtual profile that simulates their reactions and emotions to the new service.

[0671] Step 8:

[0672] The server generates a virtual FAQ based on the simulation results.

[0673] Input: Simulation results

[0674] Output: The generated virtual FAQ

[0675] Specific operation: Based on the simulation results of the AI ​​customer avatar, the server automatically generates a virtual FAQ that compiles frequently asked questions from customers and their answers.

[0676] Step 9:

[0677] The server generates learning content and promotional tools based on the simulation results.

[0678] Input: Simulation results

[0679] Output: Generated learning content and promotional tools

[0680] Specific operation: Based on the simulation results of the AI ​​customer avatar, the server generates learning content (e.g., manuals, online learning materials) and promotional tools (e.g., POPs, pamphlets) for store staff.

[0681] Step 10:

[0682] The server provides the generated virtual FAQs, learning content, and promotional tools to the user via an interface module.

[0683] Input: Generated virtual FAQs, learning content, promotional tools

[0684] Output: Information provided to the user

[0685] Specific operation: The server sends the generated virtual FAQ, learning content, and sales promotion tools to the interface module, making them accessible to users. Through this, users (store staff) can view and use the information they need.

[0686] (Application example 2)

[0687] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0688] In factory work environments, there is a need to reduce the confusion and mistakes that occur when workers introduce new robots and equipment, thereby improving production efficiency and safety. However, conventional systems generally use standalone FAQs and operation manuals, which do not fully consider workers' reactions and emotions. As a result, it is difficult to quickly and accurately respond to actual problems and concerns on the shop floor.

[0689] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting voice data, means for converting the collected voice data into text data, means for collecting search data, means for analyzing the worker's reactions and concerns using the voice data and the search data, means for generating a virtual FAQ based on the analysis results, means for recognizing the worker's emotional state based on the analysis results and generating an operation manual, means for displaying the generated virtual FAQ, operation manual, and warning system, means for training an AI model based on multiple task data, and means for presenting improvement suggestions and points to note based on simulation results generated by the AI ​​model. This makes it possible to grasp the worker's emotions and actual reactions in real time and provide prompt and accurate countermeasures.

[0690] "Voice data" refers to voice information collected from workers and equipment at the work site.

[0691] "Means for collecting voice data" refers to devices and methods for recording and collecting voice from workers, such as voice data acquisition devices and microphones within the factory.

[0692] "Text data" refers to character information obtained by analyzing voice data.

[0693] "Means for converting voice data into text data" refers to a device or method that converts collected voice data into text information using voice recognition software or the like.

[0694] "Search data" refers to information such as workers' search behavior and frequently used keywords.

[0695] "Means for collecting search data" refers to devices and methods for acquiring and recording the keywords searched by workers and their behavioral history.

[0696] "Means for analyzing reactions and concerns" refers to algorithms or software that analyze worker reactions and concerns based on collected data.

[0697] "Virtual FAQ" refers to a collection of hypothetical questions and answers based on each worker's reactions and concerns.

[0698] "Means for generating virtual FAQs" refers to a device or method that automatically generates answers to problems or questions that workers may encounter based on the results of data analysis.

[0699] "Means for recognizing emotional states" refers to algorithms or software that identify and analyze emotions from the tone and content of a worker's voice.

[0700] An "operation manual" refers to a document or guideline that explains how to operate a device or system.

[0701] "Means for generating an operating manual" refers to a device or method for creating an optimized operating procedure manual based on emotion data and reaction data.

[0702] "Warning system" refers to a notification system that alerts workers to important points and dangers.

[0703] "Means for displaying the generated virtual FAQ, operation manual, and warning system" refers to devices and methods for providing information visually or audibly to workers.

[0704] "Means for training AI models" refers to algorithms and software used to train machine learning models using collected data.

[0705] "Means for presenting improvement suggestions and points to note" refers to devices and methods that provide workers with suggestions and points to note for improving work efficiency and safety based on the results generated by a trained AI model.

[0706] System program generation

[0707] In the system that realizes this application example, the server performs processing in the following steps. First, to collect voice data, it collects voice data from voice data capture devices and microphones in the factory. Next, it converts the collected voice data into text data using voice recognition software (e.g., Google Cloud Speech-to-Text). Simultaneously with the converted text data, it extracts emotion data using an emotion recognition engine (e.g., EmotionAPI). This data is then stored in a database (e.g., MySQL).

[0708] Next, the server collects search data, including keywords used by the user and their search history. This information is also stored in a database.

[0709] The server analyzes the worker's reactions and concerns using the collected voice data, text data, search data, and emotion data. It uses NLP algorithms (e.g., spaCy) and machine learning models (e.g., TensorFlow) for the analysis. It generates a virtual FAQ based on the analysis results. It also recognizes the worker's emotional state based on the emotion data and generates an operation manual. An interface module (e.g., React.js) is used to present these results to the worker visually or audibly.

[0710] Furthermore, the AI ​​model is trained based on multiple task data sets, and based on the simulation results generated from the trained AI model, it presents improvement suggestions and points to be aware of to the worker.

[0711] Data Processing Description

[0712] The server converts the voice data collected from the voice data capture device and microphone into text data using Google Cloud Speech-to-Text. At the same time, it extracts emotion data using the Emotion API. This text data and emotion data are stored in a database (MySQL). Search data is tracked by the worker's search behavior and stored in the database.

[0713] The server analyzes the collected data using spaCy and TensorFlow to generate a virtual FAQ. It also recognizes the worker's emotional state based on the emotion data and generates an operation manual or warning system as needed. The generated virtual FAQ, operation manual, and warning system are presented to the worker using an interface module (React.js).

[0714] Examples of concrete examples and prompts

[0715] For example, if a new robot is being introduced at a factory and staff are confused, the system can extract concerns such as "The machine suddenly stopped working" or "I don't understand the error message" from the conversation data and automatically generate FAQs, which are then displayed on the smart glasses' display to assist with countermeasures in real time.

[0716] Example prompt sentence:

[0717] "Analyze staff conversation data regarding the introduction of a new factory robot to identify sources of confusion and bewilderment. Also, generate FAQs and operation manuals that take their emotional state into account."

[0718] This makes it possible to grasp the emotions and actual reactions of workers in real time and provide quick and appropriate countermeasures.

[0719] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0720] Step 1:

[0721] Audio data collection

[0722] Voice data acquisition devices (microphones, etc.) in the factory collect the voices of workers in real time. The collected voice data is sent to a server via a network. The input is the voice data of the workers, and the output is the voice data sent to the server.

[0723] Step 2:

[0724] Converting audio data to text

[0725] The server converts the received voice data into text data using speech recognition software (Google Cloud Speech-to-Text). The input is voice data and the output is text data. Specifically, the server sends the voice file to the API and receives the returned text.

[0726] Step 3:

[0727] Extracting Emotional Data

[0728] After obtaining text data from the voice data, the server uses an emotion recognition engine (Emotion API) to extract emotional data. The input is voice data and the output is emotional data. Specifically, the voice data is sent to the emotion recognition API, and emotion labels and numerical values ​​are received as analysis results.

[0729] Step 4:

[0730] Data storage

[0731] The server stores the converted text data and extracted emotion data in a database (MySQL). The input is text data and emotion data, and the output is the data stored in the database. Specifically, it executes SQL queries to insert each data into the appropriate table.

[0732] Step 5:

[0733] Search data collection

[0734] The server collects the keywords searched by the worker and their behavioral history and stores them in a database. The input is the search query, and the output is the search data stored in the database. Specifically, the server captures the search history and inserts it into the appropriate table.

[0735] Step 6:

[0736] Analysis of reactions and concerns

[0737] The server uses the collected voice, text, emotion, and search data to analyze the worker's reactions and concerns using an NLP algorithm (spaCy) and a machine learning model (TensorFlow). The input is the aforementioned data set, and the output is the analysis results. Specifically, the text data is tokenized and input into the model to obtain prediction results.

[0738] Step 7:

[0739] Virtual FAQ generation

[0740] The server generates a virtual FAQ based on the analysis results of reactions and concerns. The input is the analysis results, and the output is the virtual FAQ. Specifically, the analysis results are applied to a pre-prepared template to generate an FAQ document.

[0741] Step 8:

[0742] Generate operation manuals

[0743] The server recognizes the worker's emotional state based on the emotional data and generates an operation manual. The input is the emotional data and analysis results, and the output is the operation manual. Specifically, it generates a manual document with additional precautions and procedures according to the worker's emotional state.

[0744] Step 9:

[0745] Generate an alert system

[0746] The server generates a warning system based on the generated operation manual and emotion data. The input is the operation manual and emotion data, and the output is a warning message. Specifically, it sets up notifications to notify users of dangerous situations and important precautions in real time.

[0747] Step 10:

[0748] Presentation of information

[0749] Users use an interface module (React.js) to visually or audibly access and display the generated virtual FAQ, operation manual, and warning system. The input is the various documents generated, and the output is the information presented visually or audibly. Specifically, the information is displayed through a web application.

[0750] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0751] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0752] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0753] [Third embodiment]

[0754] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0755] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0756] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0757] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0758] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0759] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0760] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0761] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0762] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0763] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0764] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0765] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0766] The present invention is a system that uses voice data and search data to analyze customer reactions and concerns, and generates and provides virtual FAQs, learning content, and sales promotion tools based on the results. This system is implemented as follows.

[0767] System Overview

[0768] Information gathering phase

[0769] The terminal records conversations with customers using a voice recorder used in the store. This voice data is sent to multiple servers via the network. At the same time, the servers obtain user search ranking data from an online search system.

[0770] Data analysis phase

[0771] The server first converts the received voice data into text data using speech recognition. It then trains an AI model based on the voice data and search data. This AI model uses natural language processing (NLP) algorithms to automatically analyze customer responses and concerns. The server then generates and simulates an AI customer avatar based on multiple attributes.

[0772] FAQ and tool generation phase

[0773] Based on the simulation results, the server generates a virtual FAQ, which includes frequently asked questions and their answers. It also generates educational content and promotional tools (POPs, pamphlets, etc.) for store staff to learn about new services.

[0774] Results delivery phase

[0775] The generated virtual FAQs, learning content, and sales promotion tools are provided to the store staff via an interface module in an accessible state, allowing users to understand customer reactions and concerns before implementing the service and prepare appropriate countermeasures in advance.

[0776] Specific examples

[0777] Collecting and analyzing customer interaction data

[0778] Users (store staff) use voice recorders to record their daily conversations with customers. For example, they can record conversations while explaining a new service. The data is sent to a server in real time and converted into text data using voice recognition technology.

[0779] Search data collection and analysis

[0780] At the same time, the server obtains search rankings for keywords related to the new service from the search system, including information on what questions users frequently search.

[0781] Training and simulating AI models

[0782] The server uses the collected voice and search data to train an AI model that analyzes likely customer reactions and concerns, such as the likelihood that many customers will have questions about the pricing structure of a particular service.

[0783] Virtual FAQ and tool generation

[0784] Based on the analysis results, the server automatically generates virtual FAQs, learning content, and sales promotion tools, allowing users to check specific questions and answers and what they need to learn in advance, while also providing marketing materials to encourage purchases.

[0785] In this way, the system provides a concrete means for anticipating customer reactions and concerns when introducing new services and providing appropriate responses.

[0786] The processing flow will be explained below.

[0787] Specific flow of program processing

[0788] Information gathering phase

[0789] Step 1:

[0790] The terminal uses a voice recorder used in the store to record conversations with customers, including questions and feedback from the customers.

[0791] Step 2:

[0792] The server collects the audio data from the voice recorder at regular intervals, which involves the process of transferring the data over the network.

[0793] Step 3:

[0794] The server sends the voice data to a speech recognition engine and converts it into text data, which is then stored in a database.

[0795] Step 4:

[0796] The server retrieves user search ranking data, including search keywords and frequently asked questions related to the new service, and stores this information in a database.

[0797] Data analysis phase

[0798] Step 5:

[0799] The server takes in the voice data and search ranking data and begins training the AI ​​model, which uses natural language processing (NLP) algorithms.

[0800] Step 6:

[0801] The server uses an AI model to analyze customer reactions and concerns, for example, determining which parts of the service customers are likely to have questions about and what their concerns are.

[0802] Step 7:

[0803] The server generates AI customer avatars based on multiple attributes, including age, gender, and region, and simulates how each avatar would react to new services.

[0804] FAQ and tool generation phase

[0805] Step 8:

[0806] The server generates a virtual FAQ based on the simulation results, which includes frequently asked questions and answers derived from the reactions and concerns of each customer avatar.

[0807] Step 9:

[0808] The server generates learning content based on the simulation results, similar to the virtual FAQ, including manuals and online learning materials to help store staff deepen their understanding of the new service.

[0809] Step 10:

[0810] The server automatically generates promotional tools, such as POP displays and pamphlets, to effectively introduce and advertise new services.

[0811] Results delivery phase

[0812] Step 11:

[0813] Users (store staff) can access the generated virtual FAQs, learning content, and sales promotion tools through the interface module, allowing users to make the necessary preparations before introducing new services.

[0814] Step 12:

[0815] When users actually implement new services, they can use the generated tools and content to respond to customers, thereby increasing customer satisfaction.

[0816] Example 1

[0817] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0818] Today's consumers have diverse and complex needs, and it is essential to accurately understand those needs and provide them with appropriate information. However, traditional methods make it difficult to fully grasp customers' specific reactions and concerns, making it impossible to generate effective FAQs, learning content, and sales promotion tools. Furthermore, manually analyzing voice data and search data is laborious and time-consuming, placing a significant burden on on-site staff. A system that can solve these problems is needed.

[0819] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0820] In this invention, the server includes means for collecting voice data, means for converting the collected voice data into text data, means for collecting search data, means for training an AI model using a natural language processing algorithm to analyze customer reactions and concerns using the voice data and the search data, means for generating a virtual FAQ based on the analysis results, means for generating learning content and promotional tools based on the analysis results, means for generating an AI customer avatar based on multiple attributes and performing a simulation, and means for displaying the generated virtual FAQ, learning content, and promotional tools. This makes it possible to quickly and accurately grasp specific customer reactions and concerns and effectively provide the information and tools needed by on-site staff.

[0821] 1. "Voice data" refers to data that represents acoustic signals in digital format, and is a record of conversations with customers, etc.

[0822] 2. "Text data" means data that has been converted from audio data into textual information and is expressed in a format that can be analyzed using natural language processing.

[0823] 3. "Search Data" means data including ranking information for keywords and search queries that users search on the Internet or internal systems.

[0824] 4. "Natural language processing algorithms" refers to the algorithms used by AI models to analyze text data, including technologies such as speech recognition and semantic analysis.

[0825] 5. "AI Model" means a model of artificial intelligence trained using machine learning techniques and designed to perform specific tasks automatically.

[0826] 6. "Virtual FAQ" refers to automatically generated questions and answers frequently asked by customers, including information to resolve customer concerns.

[0827] 7. "Learning Content" refers to educational materials that store staff and customers can use to learn about new services and products, including text, images, videos, etc.

[0828] 8. "Promotional tools" are materials and advertising materials used to effectively introduce new services and products to customers, including posters, pamphlets, etc.

[0829] 9. An "AI customer avatar" is a virtual customer character generated based on various attribute information and is a model for conducting simulations.

[0830] 10. "Simulation" is a methodology for using AI customer avatars to recreate and analyze anticipated customer reactions and concerns.

[0831] This invention is a system that uses voice data and search data to analyze customer reactions and concerns, and generates and provides virtual FAQs, learning content, and sales promotion tools based on the results. This system is implemented as follows.

[0832] System Overview

[0833] The system includes means for collecting voice data and converting it into text data, means for collecting search data, means for analyzing customer responses and concerns, means for generating virtual FAQs, learning content, and promotional tools, and means for displaying them.

[0834] Hardware and software used

[0835] Terminal: Voice recorder (to record conversations with customers), network connection device (to transmit voice data)

[0836] Server: High-performance computer (data processing and storage), software (data conversion and analysis)

[0837] Google Cloud Speech-to-Text API (converts voice data into text data)

[0838] Machine learning libraries (e.g., TensorFlow) (used to train AI models)

[0839] Natural language processing algorithms (analyzing customer reactions and concerns)

[0840] CAIL (Cognitive AI Library) (AI customer avatar generation and simulation)

[0841] Python's Flask framework (generating virtual FAQs)

[0842] e-learning platform (e.g. Moodle) (learning content generation)

[0843] Adobe InDesign script (generate promotional tools)

[0844] Example of a system

[0845] Information gathering phase

[0846] The device uses a voice recorder to record conversations with customers. The recorded audio data (e.g., audio123.wav) is sent to a server via the network. At the same time, the server uses the Google Search API to collect user search ranking data (e.g., "new product features," "new product price," etc.).

[0847] Data analysis phase

[0848] The server uses the Google Cloud Speech-to-Text API to convert the received voice data into text data (e.g., "What are the features of the new product?"). It then uses TensorFlow to train an AI model using the collected voice and search data. It then uses CAIL to generate AI customer avatars based on multiple attributes and runs simulations. This simulation allows it to analyze what questions and complaints a particular customer avatar might have about the new service.

[0849] FAQ and tool generation phase

[0850] Based on the analysis results, the server generates a virtual FAQ using Python's Flask framework. For example, it creates an FAQ page containing answers to questions such as, "What are the features of the new product?". It also uses the same results to create learning content for e-learning platforms such as Moodle. This includes materials and videos with detailed explanations of the new product's features and sales strategies. It also uses Adobe InDesign scripts to automatically design promotional brochures and posters.

[0851] Results delivery phase

[0852] The server provides the generated virtual FAQs, learning content, and promotional tools to users via an interface module (e.g., a web portal). For example, a login page for the web portal can be created so that store staff can access the FAQs and learning content. It can also send a QR code or link to users via email, providing them in an easily accessible format.

[0853] Prompt Sentence Examples

[0854] "Please tell me more about the pricing for this service."

[0855] This system makes it possible to quickly and accurately grasp specific customer reactions and concerns, and effectively provide on-site staff with the information and tools they need.

[0856] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0857] Step 1:

[0858] The terminal uses a voice recorder to collect voice data. Specifically, the voice recorder records the conversation between the store staff and the customer while they are explaining about new products. The input is the conversation with the customer, and the output is the voice data (e.g., audio123.wav). After the voice data is collected, it is sent to the server via the network.

[0859] Step 2:

[0860] The server converts the received voice data into text data. Specifically, it uses the Google Cloud Speech-to-Text API to convert the voice data (input: audio123.wav) into text data (output: text data "What are the features of the new product?"). The converted text data is stored in the server's database.

[0861] Step 3:

[0862] The server collects search data from online search systems. Specifically, it uses the Google Search API to obtain search ranking data for keywords related to new products (input: API request). The output is the search keywords and their ranking data (e.g., "new product features," "new product price," etc.). This data is also stored in the server's database.

[0863] Step 4:

[0864] The server uses the collected voice data and search data to train an AI model. Specifically, it uses a machine learning library (e.g., TensorFlow) to train the AI ​​model based on natural language processing algorithms (input: text data and search data). The output is a trained AI model. This AI model has the ability to analyze customer responses and concerns.

[0865] Step 5:

[0866] The server uses the trained AI model to generate AI customer avatars based on multiple attributes and run simulations. Specifically, it uses CAIL (Cognitive AI Library) to generate multiple customer avatars based on attribute information such as age, gender, and purchase history (input: attribute information). The output is the generated AI customer avatar, which is then used to run simulations and predict customer reactions and concerns.

[0867] Step 6:

[0868] The server generates a virtual FAQ based on the simulation results. Specifically, it uses the Python Flask framework to create an FAQ page containing frequently asked questions and their answers (input: simulation results). The output is a virtual FAQ page (e.g., "What are the features of the new product?" and its answer).

[0869] Step 7:

[0870] The server generates learning content and promotional tools based on the analysis results. Specifically, it creates learning modules to be installed on e-learning platforms (e.g., Moodle) (input: simulation results), and automatically generates promotional brochures and posters using Adobe InDesign scripts (input: simulation results). The output is learning content and promotional tools (e.g., brochures and posters).

[0871] Step 8:

[0872] The server provides the generated virtual FAQs, learning content, and promotional tools to users via the interface module. Specifically, it creates a login page for a web portal from which store staff can access the FAQs and learning content (input: generated content and tools). The output is the content and tools provided in a form that users can use (e.g., web links, QR codes).

[0873] (Application example 1)

[0874] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0875] To improve the efficiency and accuracy of customer service, it is necessary to analyze customer reactions and concerns in real time and enable store staff to quickly respond appropriately based on that information. However, traditional systems struggled to efficiently integrate these functions, and lacked support tools to improve the quality of customer service. Furthermore, there were insufficient means to identify customer questions and concerns in advance when introducing new services, and to provide effective learning content and promotional tools. As a result, customer satisfaction declined, hindering sales growth.

[0876] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0877] In this invention, the server includes means for collecting voice data, means for converting the collected voice data into text data, means for collecting search data, means for analyzing customer responses and concerns using the voice data and the search data, means for generating a virtual FAQ based on the analysis results, means for generating learning content and promotional tools based on the analysis results, means for displaying the generated virtual FAQ, learning content, and promotional tools, and means for providing a smartphone application that supports customer service using the virtual FAQ, learning content, and promotional tools generated based on the analysis results. This enables real-time understanding of customer responses and concerns and prompt appropriate responses. Furthermore, by identifying customer questions and concerns in advance when introducing a new service and providing effective learning content and promotional tools, the burden on store staff can be reduced, leading to improved customer satisfaction and increased sales.

[0878] "Voice data" refers to acoustic signal data that records conversations between customers and store staff.

[0879] "Text data" refers to data obtained by converting voice data into text using voice recognition technology.

[0880] "Search Data" refers to data that includes customer online search behavior and ranking information.

[0881] "Customer reactions and concerns" refers to feedback and reactions such as interest, questions, and concerns that customers show in-store.

[0882] "Analysis" refers to the act of using AI models and natural language processing algorithms to analyze customer responses and concerns from voice and search data.

[0883] A "virtual FAQ" is a virtual FAQ (Frequently Asked Questions) that is automatically generated based on the analysis results and contains questions that customers frequently ask and their answers.

[0884] "Learning content" refers to educational materials and content provided to store staff to help them learn about new services.

[0885] "Sales promotional tools" are promotional materials and tools such as POPs and pamphlets used to promote the sale of products and services.

[0886] "Display" refers to the act of visually providing the generated virtual FAQs, learning content, and promotional tools to users.

[0887] A "smartphone application" is a software program that is installed on a smartphone device to assist with customer service.

[0888] Information gathering phase

[0889] To collect voice data, the server records conversations between customers and store staff using a voice recorder used in the store. The recorded voice data is sent to the server in real time. At the same time, the server obtains user search ranking data from an online search system, thereby collecting information related to the customer's interests and questions.

[0890] Data analysis phase

[0891] The server converts the received voice data into text data using speech recognition technology. Services such as Google Cloud Speech-to-Text and IBM Watson Speech to Text are used for speech recognition. Next, an AI model (e.g., BERT or GPT-4) is trained based on the voice data and search data. This AI model uses natural language processing algorithms to automatically analyze customer responses and concerns. The server also generates an AI model based on multiple attributes and performs simulations. Google Cloud Natural Language and Amazon Comprehend are commonly used NLP services.

[0892] FAQ and tool generation phase

[0893] The server generates a virtual FAQ based on the analysis results. This virtual FAQ contains frequently asked questions and their answers, and aggregates information useful for responding to customers. Based on the same results, the server also automatically generates learning content and promotional tools (e.g., POPs, pamphlets, etc.) for store staff to learn about new services. The generated virtual FAQ, learning content, and promotional tools are stored on the server.

[0894] Results delivery phase

[0895] The generated virtual FAQs, learning content, and promotional tools are provided to store staff via a smartphone application. Staff can view these through the application interface. This application enables staff to respond to customer questions quickly and accurately. Staff can also use the learning content to efficiently acquire knowledge about new services.

[0896] Specific examples

[0897] For example, when a physical store introduces a new electronic payment service, if a customer asks staff questions about how to use it, fees, or security, the store can use the app to get real-time answers.

[0898] Prompt Sentence Examples

[0899] Customer: "Can you show me how to use the new electronic payment service?"

[0900] Application: "The basic steps for using the new electronic payment service are to first download the app and then create an account. Detailed instructions can be found in the following FAQ."

[0901] In this way, the system of the present invention can grasp and analyze customer reactions and concerns in real time, creating an environment in which they can be responded to efficiently, thereby improving customer satisfaction and increasing sales.

[0902] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0903] Step 1: Collecting audio data

[0904] The user (store staff) uses a voice recorder to record conversations with customers to collect voice data. After the voice data is collected, it is sent to the server in real time. The input is the voice data, and the output is the voice data sent to the server.

[0905] Step 2: Convert audio data to text

[0906] The server converts the received voice data into text data using speech recognition technology. This process uses services such as Google Cloud Speech-to-Text and IBM Watson Speech to Text. The input is voice data and the output is text data. Specific operations include the process of converting voice data into a string of characters.

[0907] Step 3: Collect search data

[0908] The server retrieves user search ranking data from the online search system. This data indicates what questions and keywords customers frequently search. The input is the search data retrieved from the online search system, and the output is the search data used for analysis.

[0909] Step 4: Analyze customer reactions and concerns

[0910] The server trains an AI model (e.g., BERT or GPT-4) based on the received text data and search data to analyze customer reactions and concerns. It uses an NLP algorithm, such as Google Cloud Natural Language or Amazon Comprehend. The input is the text data and search data, and the output is the analysis results of customer reactions and concerns. Specific operations include inputting data, training the model, and analyzing reactions and concerns.

[0911] Step 5: Generate a virtual FAQ

[0912] The server generates a virtual FAQ based on the analysis results. This virtual FAQ contains frequently asked questions and their answers from customers. The input is the analysis results on customer reactions and concerns, and the output is the virtual FAQ. Specific operations include generating patterns of questions and answers.

[0913] Step 6: Generate learning content and collateral

[0914] The server automatically generates learning content and promotional tools based on the same analysis results. The generated learning content includes educational materials about new services, and the promotional tools include POP displays and pamphlets. The input is the analysis results regarding customer reactions and concerns, and the output is learning content and promotional tools. Specific operations include generating educational materials and promotional materials.

[0915] Step 7: Delivering results

[0916] The generated virtual FAQs, learning content, and promotional tools are provided to store staff via a smartphone application. Users (store staff) can use this application to access the generated information and quickly and accurately answer customer questions. The input is the virtual FAQs, learning content, and promotional tools, and the output is customer support based on them. Specific operations include displaying and using the information via the smartphone application.

[0917] Through these steps, the inventive system is able to grasp customer reactions and concerns in real time, enabling appropriate responses to be taken quickly.

[0918] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0919] The present invention combines a system that uses voice data and search data to analyze customer reactions and concerns, and generates and provides virtual FAQs, learning content, and sales promotion tools based on the results, with an emotion engine that recognizes user emotions. Specific embodiments of the present invention will be described below.

[0920] System Overview

[0921] Information gathering phase

[0922] The terminal uses a voice recorder used in the store to record conversations with customers. This conversation includes questions and feedback from customers. The server collects voice data from the voice recorder at regular intervals and receives it via the network.

[0923] The server sends the voice data to the emotion engine for emotion recognition. The voice data is then sent to the speech recognition engine for conversion into text data. The converted text data and the recognized emotion data are stored in a database. At the same time, the server obtains the user's search ranking data and stores this data in the database.

[0924] Data analysis phase

[0925] The server uses voice data, search ranking data, and sentiment data to begin training the AI ​​model, which uses natural language processing (NLP) algorithms to better analyze customer reactions and concerns.

[0926] The server generates AI customer avatars based on multiple attributes and simulates how each avatar will react and feel toward new services.

[0927] FAQ and tool generation phase

[0928] The server generates a virtual FAQ based on the simulation results, which contains questions and answers based on each customer avatar's reactions, concerns, and emotional data.

[0929] The server generates learning content based on the simulation results. This includes manuals and online learning materials to help store staff deepen their understanding of the new service. It also takes into account emotional data to identify points requiring special attention and specific countermeasures.

[0930] The server automatically generates promotional tools, such as POP displays and pamphlets, to effectively introduce and advertise new services.

[0931] Results delivery phase

[0932] The generated virtual FAQs, learning content, and sales promotion tools are provided to the store staff (users) via an interface module in an accessible state, allowing users to make the necessary preparations before introducing the service.

[0933] For example, when users (store staff) prepare to serve customers before introducing a new service, they can check FAQs for specific questions and points to note based on emotional data, making it easier to elicit positive customer reactions to the new service.

[0934] In this way, the system predicts customer reactions and concerns when introducing new services and provides appropriate responses, thereby increasing customer satisfaction and supporting the success of the service. The addition of an emotion engine makes it possible to more accurately grasp the customer's emotional state and respond in a way that reflects that information, resulting in even more effective service provision.

[0935] The processing flow will be explained below.

[0936] Specific flow of program processing

[0937] Information gathering phase

[0938] Step 1:

[0939] The terminal uses a voice recorder used in the store to record conversations with customers, including questions and feedback from the customers.

[0940] Step 2:

[0941] The server collects the audio data from the voice recorder at regular intervals, which involves the process of transferring the data over the network.

[0942] Step 3:

[0943] The server sends the collected voice data to an emotion engine to recognize the customer's emotions, for example, whether the customer is expressing negative emotions or interest in the service.

[0944] Step 4:

[0945] The server sends the voice data to a speech recognition engine and converts it into text data. The converted text data and the recognized emotion data are stored in a database.

[0946] Step 5:

[0947] The server retrieves user search ranking data, including search keywords and frequently asked questions related to the new service, and stores this information in a database.

[0948] Data analysis phase

[0949] Step 6:

[0950] The server then uses the collected voice data, search ranking data, and sentiment data to begin training the AI ​​model, using natural language processing (NLP) algorithms.

[0951] Step 7:

[0952] The server uses an AI model to analyze customer reactions and concerns, such as which parts of the service customers are likely to have questions about and what emotions they are likely to experience.

[0953] Step 8:

[0954] The server generates AI customer avatars based on multiple attributes, including age, gender, and region, and simulates how each avatar would react to new services.

[0955] FAQ and tool generation phase

[0956] Step 9:

[0957] Based on the simulation results, the server generates a virtual FAQ, which includes questions and answers based on each customer avatar's reactions, concerns, and emotional data.

[0958] Step 10:

[0959] The server generates learning content based on the simulation results. This includes manuals and online learning materials to help store staff deepen their understanding of the new service. It also takes into account emotional data to identify points requiring special attention and specific countermeasures.

[0960] Step 11:

[0961] The server automatically generates promotional tools, such as POP displays and pamphlets, to effectively introduce and advertise new services.

[0962] Results delivery phase

[0963] Step 12:

[0964] Users (store staff) can access the generated virtual FAQs, learning content, and sales promotion tools through the interface module, allowing users to make the necessary preparations before introducing new services.

[0965] Step 13:

[0966] When users actually introduce new services, they can use the various tools and content generated to respond to customers. For example, they can increase customer satisfaction by quickly responding to customer questions based on FAQs and implementing methods to alleviate customer concerns based on emotional data.

[0967] Example 2

[0968] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0969] Conventional customer support systems have difficulty accurately grasping customer reactions and emotions, making it difficult to provide appropriate responses quickly. Furthermore, there is a lack of technology to effectively utilize customer feedback and search data to generate services and promotional tools. This makes it difficult to improve customer satisfaction and support the successful introduction of services.

[0970] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0971] In this invention, the server includes means for collecting voice data, means for transmitting the collected voice data to an emotion engine and recognizing customer emotions, means for converting the emotion-recognized voice data into text data, means for storing the converted text data and the recognized emotion data in a database, means for collecting search data, means for training an AI model using the voice data, the emotion data, and the search data to analyze customer reactions and concerns, means for generating an AI customer avatar based on the analysis results and simulating reactions and emotions to a new service, means for generating a virtual FAQ based on the simulation results, means for generating learning content and promotional tools based on the simulation results, and means for displaying the generated virtual FAQ, learning content, and promotional tools. This enables accurate understanding of customer emotions and prompt provision of appropriate countermeasures based on the understanding. Furthermore, customer feedback and search data can be utilized to generate effective virtual FAQs, learning content, and promotional tools, thereby improving customer satisfaction and supporting successful service introduction.

[0972] "Voice data" refers to information recorded in digital format from the conversations and voices of people such as customers and staff.

[0973] "Emotion engine" is a general term for algorithms and technologies that analyze voice data and identify the emotional state contained within it (e.g., joy, anger, sadness, etc.).

[0974] "Text data" is information in the form of a string of characters converted from voice data by a voice recognition engine.

[0975] A "database" is a system or device for systematically storing and managing collected and converted data.

[0976] "Search Data" means records and statistical information about users' search behavior on the Internet and internal systems.

[0977] "AI Model" means an artificial intelligence model trained using machine learning and natural language processing algorithms and used to analyze customer responses and concerns.

[0978] "Customer avatars" are virtual customer profiles generated based on AI models to simulate reactions and emotions toward new services.

[0979] "Simulation results" are the predicted results of virtual reactions and emotions by AI customer avatars.

[0980] A "virtual FAQ" is a virtual document that compiles frequently asked questions from customers and their answers, generated based on the simulation results.

[0981] "Learning content" refers to educational materials and online learning materials that help store staff understand new services and use them to assist them in dealing with customers.

[0982] "Promotional tools" are promotional materials (e.g., POPs, pamphlets, etc.) used to effectively introduce and advertise new services.

[0983] An "interface module" is a system component or software that provides the generated virtual FAQs, learning content, and promotional tools in an accessible form to users.

[0984] "Users" are store staff and other related parties who operate the system and use the generated virtual FAQs, learning content, and promotional tools.

[0985] The present invention combines a system that uses voice data and search data to analyze customer reactions and concerns, and generates and provides virtual FAQs, learning content, and sales promotion tools based on the results, with an emotion engine that recognizes user emotions. Specific embodiments of the present invention will be described below.

[0986] 1. Information gathering phase

[0987] The terminal uses a voice recorder used in the store to record conversations with customers. This conversation includes questions and feedback from customers. The terminal sends this voice data to a server at regular intervals. The voice data is recorded with a timestamp, making it possible to determine the exact point in time at which customer feedback was given.

[0988] 2. Emotion and voice recognition phase

[0989] The server sends the received voice data to an emotion engine to recognize the customer's emotions. The emotion engine uses a voice analysis algorithm to identify emotional states such as "joy," "anger," and "sadness." The server then sends the voice data to a speech recognition engine, which converts it into text data. The converted text data is a string of characters that directly represents the content of the conversation, including specific questions and feedback.

[0990] 3. Data storage phase

[0991] The server stores the converted text data and the recognized emotion data in a database. The database is used to identify changes and patterns over time. At the same time, the server obtains users' search ranking data and stores this data in the database. This search data includes the keywords and topics that users most frequently searched for over a certain period of time.

[0992] 4. AI model training phase

[0993] The server trains an AI model using voice data, search ranking data, and sentiment data. The AI ​​model uses natural language processing (NLP) algorithms to more accurately analyze customer reactions and concerns. The AI ​​model also generates an AI customer avatar based on multiple attributes. The customer avatar has a virtual profile and simulates reactions and emotions toward new services.

[0994] 5. FAQ and tool generation phase

[0995] The server generates a virtual FAQ based on the simulation results. This FAQ includes questions and answers based on the reactions, concerns, and emotional data of each customer avatar. The server also generates learning content and promotional tools based on the simulation results. The learning content includes manuals and online learning materials to help store staff deepen their understanding of the new service. Points that require particular attention and specific countermeasures are also added, improving the quality of customer service. The promotional tools include POP displays and pamphlets to effectively introduce and promote the new service.

[0996] 6. Results delivery phase

[0997] The generated virtual FAQs, learning content, and sales promotion tools are provided to the store staff (users) via the interface module. This allows the users to make the necessary preparations before introducing the service. For example, when users (store staff) prepare to deal with customers before introducing a new service, they can check FAQs for specific questions and points to note based on emotion data. This makes it easier to elicit a positive customer response to the new service.

[0998] Specific examples

[0999] For example, when a user is preparing to introduce a new service, they can input a prompt such as, "Please tell us the questions that customers are most likely to ask when the new service is introduced. Also, please simulate appropriate answers and customer reactions." Based on this prompt, the server uses an AI model to generate the most appropriate virtual FAQ and provides questions and answers based on emotional data. In this way, users can smoothly introduce new services and increase customer satisfaction.

[1000] This system efficiently utilizes voice and search data to accurately grasp customer emotions and reactions, enabling it to provide prompt and appropriate responses. Furthermore, by combining a generative AI model with an emotion engine, the system achieves more effective service delivery.

[1001] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1002] Step 1:

[1003] The device records conversations with customers using a voice recorder.

[1004] Specific operation: The terminal activates a voice recorder installed in the store and records the conversation between the customer and the staff. The voice data is recorded with a timestamp and saved on the terminal.

[1005] Step 2:

[1006] The server sends the voice data to an emotion engine to recognize the customer's emotion.

[1007] Input: Audio data received from the device

[1008] Output: Recognized emotion data

[1009] Specific operation: The server periodically transmits voice data acquired from the device to the emotion engine. The emotion engine uses a voice analysis algorithm to identify emotional states such as "joy," "anger," and "sadness." The recognized emotions are returned to the server in data format.

[1010] Step 3:

[1011] The server sends the voice data to a voice recognition engine and converts it into text data.

[1012] Input: Audio data received from the device

[1013] Output: Converted text data

[1014] Specific operation: The server sends the voice data to a speech recognition engine, which converts the voice into text. For example, the voice saying "I want to know more about this new service" is converted into text "I want to know more about this new service."

[1015] Step 4:

[1016] The server stores the text data and emotion data in a database.

[1017] Input: converted text data and recognized emotion data

[1018] Output: Data stored in the database

[1019] Specific operation: The server stores the converted text data and emotion data in a database, including timestamps and conversation context.

[1020] Step 5:

[1021] The server retrieves the user's search ranking data and stores it in a database.

[1022] Input: Search data from websites or internal systems

[1023] Output: Search ranking data stored in a database

[1024] What it does: The server retrieves user search ranking data from the internet or internal systems and stores it in a database, including the most searched keywords and topics for the past week.

[1025] Step 6:

[1026] The server trains the AI ​​model using voice data, search ranking data, and emotion data.

[1027] Input: Voice data, search ranking data, emotion data

[1028] Output: A trained AI model

[1029] How it works: The server feeds this data to an AI model, which trains it with natural language processing (NLP) algorithms. The AI ​​model is trained to analyze customer reactions and concerns.

[1030] Step 7:

[1031] The server generates AI customer avatars and simulates their reactions and emotions to new services.

[1032] Input: A trained AI model

[1033] Output: Simulated customer avatar reactions and emotions

[1034] How it works: The server uses the trained AI model to generate multiple customer avatars, each with a virtual profile that simulates their reactions and emotions to the new service.

[1035] Step 8:

[1036] The server generates a virtual FAQ based on the simulation results.

[1037] Input: Simulation results

[1038] Output: The generated virtual FAQ

[1039] Specific operation: Based on the simulation results of the AI ​​customer avatar, the server automatically generates a virtual FAQ that compiles frequently asked questions from customers and their answers.

[1040] Step 9:

[1041] The server generates learning content and promotional tools based on the simulation results.

[1042] Input: Simulation results

[1043] Output: Generated learning content and promotional tools

[1044] Specific operation: Based on the simulation results of the AI ​​customer avatar, the server generates learning content (e.g., manuals, online learning materials) and promotional tools (e.g., POPs, pamphlets) for store staff.

[1045] Step 10:

[1046] The server provides the generated virtual FAQs, learning content, and promotional tools to the user via an interface module.

[1047] Input: Generated virtual FAQs, learning content, promotional tools

[1048] Output: Information provided to the user

[1049] Specific operation: The server sends the generated virtual FAQ, learning content, and sales promotion tools to the interface module, making them accessible to users. Through this, users (store staff) can view and use the information they need.

[1050] (Application example 2)

[1051] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1052] In factory work environments, there is a need to reduce the confusion and mistakes that occur when workers introduce new robots and equipment, thereby improving production efficiency and safety. However, conventional systems generally use standalone FAQs and operation manuals, which do not fully consider workers' reactions and emotions. As a result, it is difficult to quickly and accurately respond to actual problems and concerns on the shop floor.

[1053] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting voice data, means for converting the collected voice data into text data, means for collecting search data, means for analyzing the worker's reactions and concerns using the voice data and the search data, means for generating a virtual FAQ based on the analysis results, means for recognizing the worker's emotional state based on the analysis results and generating an operation manual, means for displaying the generated virtual FAQ, operation manual, and warning system, means for training an AI model based on multiple task data, and means for presenting improvement suggestions and points to note based on simulation results generated by the AI ​​model. This makes it possible to grasp the worker's emotions and actual reactions in real time and provide prompt and accurate countermeasures.

[1054] "Voice data" refers to voice information collected from workers and equipment at the work site.

[1055] "Means for collecting voice data" refers to devices and methods for recording and collecting voice from workers, such as voice data acquisition devices and microphones within the factory.

[1056] "Text data" refers to character information obtained by analyzing voice data.

[1057] "Means for converting voice data into text data" refers to a device or method that converts collected voice data into text information using voice recognition software or the like.

[1058] "Search data" refers to information such as workers' search behavior and frequently used keywords.

[1059] "Means for collecting search data" refers to devices and methods for acquiring and recording the keywords searched by workers and their behavioral history.

[1060] "Means for analyzing reactions and concerns" refers to algorithms or software that analyze worker reactions and concerns based on collected data.

[1061] "Virtual FAQ" refers to a collection of hypothetical questions and answers based on each worker's reactions and concerns.

[1062] "Means for generating virtual FAQs" refers to a device or method that automatically generates answers to problems or questions that workers may encounter based on the results of data analysis.

[1063] "Means for recognizing emotional states" refers to algorithms or software that identify and analyze emotions from the tone and content of a worker's voice.

[1064] An "operation manual" refers to a document or guideline that explains how to operate a device or system.

[1065] "Means for generating an operating manual" refers to a device or method for creating an optimized operating procedure manual based on emotion data and reaction data.

[1066] "Warning system" refers to a notification system that alerts workers to important points and dangers.

[1067] "Means for displaying the generated virtual FAQ, operation manual, and warning system" refers to devices and methods for providing information visually or audibly to workers.

[1068] "Means for training AI models" refers to algorithms and software used to train machine learning models using collected data.

[1069] "Means for presenting improvement suggestions and points to note" refers to devices and methods that provide workers with suggestions and points to note for improving work efficiency and safety based on the results generated by a trained AI model.

[1070] System program generation

[1071] In the system that realizes this application example, the server performs processing in the following steps. First, to collect voice data, it collects voice data from voice data capture devices and microphones in the factory. Next, it converts the collected voice data into text data using voice recognition software (e.g., Google Cloud Speech-to-Text). Simultaneously with the converted text data, it extracts emotion data using an emotion recognition engine (e.g., EmotionAPI). This data is then stored in a database (e.g., MySQL).

[1072] Next, the server collects search data, including keywords used by the user and their search history. This information is also stored in a database.

[1073] The server analyzes the worker's reactions and concerns using the collected voice data, text data, search data, and emotion data. It uses NLP algorithms (e.g., spaCy) and machine learning models (e.g., TensorFlow) for the analysis. It generates a virtual FAQ based on the analysis results. It also recognizes the worker's emotional state based on the emotion data and generates an operation manual. An interface module (e.g., React.js) is used to present these results to the worker visually or audibly.

[1074] Furthermore, the AI ​​model is trained based on multiple task data sets, and based on the simulation results generated from the trained AI model, it presents improvement suggestions and points to be aware of to the worker.

[1075] Data Processing Description

[1076] The server converts the voice data collected from the voice data capture device and microphone into text data using Google Cloud Speech-to-Text. At the same time, it extracts emotion data using the Emotion API. This text data and emotion data are stored in a database (MySQL). Search data is tracked by the worker's search behavior and stored in the database.

[1077] The server analyzes the collected data using spaCy and TensorFlow to generate a virtual FAQ. It also recognizes the worker's emotional state based on the emotion data and generates an operation manual or warning system as needed. The generated virtual FAQ, operation manual, and warning system are presented to the worker using an interface module (React.js).

[1078] Examples of concrete examples and prompts

[1079] For example, if a new robot is being introduced at a factory and staff are confused, the system can extract concerns such as "The machine suddenly stopped working" or "I don't understand the error message" from the conversation data and automatically generate FAQs, which are then displayed on the smart glasses' display to assist with countermeasures in real time.

[1080] Example prompt sentence:

[1081] "Analyze staff conversation data regarding the introduction of a new factory robot to identify sources of confusion and bewilderment. Also, generate FAQs and operation manuals that take their emotional state into account."

[1082] This makes it possible to grasp the emotions and actual reactions of workers in real time and provide quick and appropriate countermeasures.

[1083] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1084] Step 1:

[1085] Audio data collection

[1086] Voice data acquisition devices (microphones, etc.) in the factory collect the voices of workers in real time. The collected voice data is sent to a server via a network. The input is the voice data of the workers, and the output is the voice data sent to the server.

[1087] Step 2:

[1088] Converting audio data to text

[1089] The server converts the received voice data into text data using speech recognition software (Google Cloud Speech-to-Text). The input is voice data and the output is text data. Specifically, the server sends the voice file to the API and receives the returned text.

[1090] Step 3:

[1091] Extracting Emotional Data

[1092] After obtaining text data from the voice data, the server uses an emotion recognition engine (Emotion API) to extract emotional data. The input is voice data and the output is emotional data. Specifically, the voice data is sent to the emotion recognition API, and emotion labels and numerical values ​​are received as analysis results.

[1093] Step 4:

[1094] Data storage

[1095] The server stores the converted text data and extracted emotion data in a database (MySQL). The input is text data and emotion data, and the output is the data stored in the database. Specifically, it executes SQL queries to insert each data into the appropriate table.

[1096] Step 5:

[1097] Search data collection

[1098] The server collects the keywords searched by the worker and their behavioral history and stores them in a database. The input is the search query, and the output is the search data stored in the database. Specifically, the server captures the search history and inserts it into the appropriate table.

[1099] Step 6:

[1100] Analysis of reactions and concerns

[1101] The server uses the collected voice, text, emotion, and search data to analyze the worker's reactions and concerns using an NLP algorithm (spaCy) and a machine learning model (TensorFlow). The input is the aforementioned data set, and the output is the analysis results. Specifically, the text data is tokenized and input into the model to obtain prediction results.

[1102] Step 7:

[1103] Virtual FAQ generation

[1104] The server generates a virtual FAQ based on the analysis results of reactions and concerns. The input is the analysis results, and the output is the virtual FAQ. Specifically, the analysis results are applied to a pre-prepared template to generate an FAQ document.

[1105] Step 8:

[1106] Generate operation manuals

[1107] The server recognizes the worker's emotional state based on the emotional data and generates an operation manual. The input is the emotional data and analysis results, and the output is the operation manual. Specifically, it generates a manual document with additional precautions and procedures according to the worker's emotional state.

[1108] Step 9:

[1109] Generate an alert system

[1110] The server generates a warning system based on the generated operation manual and emotion data. The input is the operation manual and emotion data, and the output is a warning message. Specifically, it sets up notifications to notify users of dangerous situations and important precautions in real time.

[1111] Step 10:

[1112] Presentation of information

[1113] Users use an interface module (React.js) to visually or audibly access and display the generated virtual FAQ, operation manual, and warning system. The input is the various documents generated, and the output is the information presented visually or audibly. Specifically, the information is displayed through a web application.

[1114] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1115] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1116] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1117] [Fourth embodiment]

[1118] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1119] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1120] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1121] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1122] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1123] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1124] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1125] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1126] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1127] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1128] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1129] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1130] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1131] The present invention is a system that uses voice data and search data to analyze customer reactions and concerns, and generates and provides virtual FAQs, learning content, and sales promotion tools based on the results. This system is implemented as follows.

[1132] System Overview

[1133] Information gathering phase

[1134] The terminal records conversations with customers using a voice recorder used in the store. This voice data is sent to multiple servers via the network. At the same time, the servers obtain user search ranking data from an online search system.

[1135] Data analysis phase

[1136] The server first converts the received voice data into text data using speech recognition. It then trains an AI model based on the voice data and search data. This AI model uses natural language processing (NLP) algorithms to automatically analyze customer responses and concerns. The server then generates and simulates an AI customer avatar based on multiple attributes.

[1137] FAQ and tool generation phase

[1138] Based on the simulation results, the server generates a virtual FAQ, which includes frequently asked questions and their answers. It also generates educational content and promotional tools (POPs, pamphlets, etc.) for store staff to learn about new services.

[1139] Results delivery phase

[1140] The generated virtual FAQs, learning content, and sales promotion tools are provided to the store staff via an interface module in an accessible state, allowing users to understand customer reactions and concerns before implementing the service and prepare appropriate countermeasures in advance.

[1141] Specific examples

[1142] Collecting and analyzing customer interaction data

[1143] Users (store staff) use voice recorders to record their daily conversations with customers. For example, they can record conversations while explaining a new service. The data is sent to a server in real time and converted into text data using voice recognition technology.

[1144] Search data collection and analysis

[1145] At the same time, the server obtains search rankings for keywords related to the new service from the search system, including information on what questions users frequently search.

[1146] Training and simulating AI models

[1147] The server uses the collected voice and search data to train an AI model that analyzes likely customer reactions and concerns, such as the likelihood that many customers will have questions about the pricing structure of a particular service.

[1148] Virtual FAQ and tool generation

[1149] Based on the analysis results, the server automatically generates virtual FAQs, learning content, and sales promotion tools, allowing users to check specific questions and answers and what they need to learn in advance, while also providing marketing materials to encourage purchases.

[1150] In this way, the system provides a concrete means for anticipating customer reactions and concerns when introducing new services and providing appropriate responses.

[1151] The processing flow will be explained below.

[1152] Specific flow of program processing

[1153] Information gathering phase

[1154] Step 1:

[1155] The terminal uses a voice recorder used in the store to record conversations with customers, including questions and feedback from the customers.

[1156] Step 2:

[1157] The server collects the audio data from the voice recorder at regular intervals, which involves the process of transferring the data over the network.

[1158] Step 3:

[1159] The server sends the voice data to a speech recognition engine and converts it into text data, which is then stored in a database.

[1160] Step 4:

[1161] The server retrieves user search ranking data, including search keywords and frequently asked questions related to the new service, and stores this information in a database.

[1162] Data analysis phase

[1163] Step 5:

[1164] The server takes in the voice data and search ranking data and begins training the AI ​​model, which uses natural language processing (NLP) algorithms.

[1165] Step 6:

[1166] The server uses an AI model to analyze customer reactions and concerns, for example, determining which parts of the service customers are likely to have questions about and what their concerns are.

[1167] Step 7:

[1168] The server generates AI customer avatars based on multiple attributes, including age, gender, and region, and simulates how each avatar would react to new services.

[1169] FAQ and tool generation phase

[1170] Step 8:

[1171] The server generates a virtual FAQ based on the simulation results, which includes frequently asked questions and answers derived from the reactions and concerns of each customer avatar.

[1172] Step 9:

[1173] The server generates learning content based on the simulation results, similar to the virtual FAQ, including manuals and online learning materials to help store staff deepen their understanding of the new service.

[1174] Step 10:

[1175] The server automatically generates promotional tools, such as POP displays and pamphlets, to effectively introduce and advertise new services.

[1176] Results delivery phase

[1177] Step 11:

[1178] Users (store staff) can access the generated virtual FAQs, learning content, and sales promotion tools through the interface module, allowing users to make the necessary preparations before introducing new services.

[1179] Step 12:

[1180] When users actually implement new services, they can use the generated tools and content to respond to customers, thereby increasing customer satisfaction.

[1181] Example 1

[1182] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1183] Today's consumers have diverse and complex needs, and it is essential to accurately understand those needs and provide them with appropriate information. However, traditional methods make it difficult to fully grasp customers' specific reactions and concerns, making it impossible to generate effective FAQs, learning content, and sales promotion tools. Furthermore, manually analyzing voice data and search data is laborious and time-consuming, placing a significant burden on on-site staff. A system that can solve these problems is needed.

[1184] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1185] In this invention, the server includes means for collecting voice data, means for converting the collected voice data into text data, means for collecting search data, means for training an AI model using a natural language processing algorithm to analyze customer reactions and concerns using the voice data and the search data, means for generating a virtual FAQ based on the analysis results, means for generating learning content and promotional tools based on the analysis results, means for generating an AI customer avatar based on multiple attributes and performing a simulation, and means for displaying the generated virtual FAQ, learning content, and promotional tools. This makes it possible to quickly and accurately grasp specific customer reactions and concerns and effectively provide the information and tools needed by on-site staff.

[1186] 1. "Voice data" refers to data that represents acoustic signals in digital format, and is a record of conversations with customers, etc.

[1187] 2. "Text data" means data that has been converted from audio data into textual information and is expressed in a format that can be analyzed using natural language processing.

[1188] 3. "Search Data" means data including ranking information for keywords and search queries that users search on the Internet or internal systems.

[1189] 4. "Natural language processing algorithms" refers to the algorithms used by AI models to analyze text data, including technologies such as speech recognition and semantic analysis.

[1190] 5. "AI Model" means a model of artificial intelligence trained using machine learning techniques and designed to perform specific tasks automatically.

[1191] 6. "Virtual FAQ" refers to automatically generated questions and answers frequently asked by customers, including information to resolve customer concerns.

[1192] 7. "Learning Content" refers to educational materials that store staff and customers can use to learn about new services and products, including text, images, videos, etc.

[1193] 8. "Promotional tools" are materials and advertising materials used to effectively introduce new services and products to customers, including posters, pamphlets, etc.

[1194] 9. An "AI customer avatar" is a virtual customer character generated based on various attribute information and is a model for conducting simulations.

[1195] 10. "Simulation" is a methodology for using AI customer avatars to recreate and analyze anticipated customer reactions and concerns.

[1196] This invention is a system that uses voice data and search data to analyze customer reactions and concerns, and generates and provides virtual FAQs, learning content, and sales promotion tools based on the results. This system is implemented as follows.

[1197] System Overview

[1198] The system includes means for collecting voice data and converting it into text data, means for collecting search data, means for analyzing customer responses and concerns, means for generating virtual FAQs, learning content, and promotional tools, and means for displaying them.

[1199] Hardware and software used

[1200] Terminal: Voice recorder (to record conversations with customers), network connection device (to transmit voice data)

[1201] Server: High-performance computer (data processing and storage), software (data conversion and analysis)

[1202] Google Cloud Speech-to-Text API (converts voice data into text data)

[1203] Machine learning libraries (e.g., TensorFlow) (used to train AI models)

[1204] Natural language processing algorithms (analyzing customer reactions and concerns)

[1205] CAIL (Cognitive AI Library) (AI customer avatar generation and simulation)

[1206] Python's Flask framework (generating virtual FAQs)

[1207] e-learning platform (e.g. Moodle) (learning content generation)

[1208] Adobe InDesign script (generate promotional tools)

[1209] Example of a system

[1210] Information gathering phase

[1211] The device uses a voice recorder to record conversations with customers. The recorded audio data (e.g., audio123.wav) is sent to a server via the network. At the same time, the server uses the Google Search API to collect user search ranking data (e.g., "new product features," "new product price," etc.).

[1212] Data analysis phase

[1213] The server uses the Google Cloud Speech-to-Text API to convert the received voice data into text data (e.g., "What are the features of the new product?"). It then uses TensorFlow to train an AI model using the collected voice and search data. It then uses CAIL to generate AI customer avatars based on multiple attributes and runs simulations. This simulation allows it to analyze what questions and complaints a particular customer avatar might have about the new service.

[1214] FAQ and tool generation phase

[1215] Based on the analysis results, the server generates a virtual FAQ using Python's Flask framework. For example, it creates an FAQ page containing answers to questions such as, "What are the features of the new product?". It also uses the same results to create learning content for e-learning platforms such as Moodle. This includes materials and videos with detailed explanations of the new product's features and sales strategies. It also uses Adobe InDesign scripts to automatically design promotional brochures and posters.

[1216] Results delivery phase

[1217] The server provides the generated virtual FAQs, learning content, and promotional tools to users via an interface module (e.g., a web portal). For example, a login page for the web portal can be created so that store staff can access the FAQs and learning content. It can also send a QR code or link to users via email, providing them in an easily accessible format.

[1218] Prompt Sentence Examples

[1219] "Please tell me more about the pricing for this service."

[1220] This system makes it possible to quickly and accurately grasp specific customer reactions and concerns, and effectively provide on-site staff with the information and tools they need.

[1221] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1222] Step 1:

[1223] The terminal uses a voice recorder to collect voice data. Specifically, the voice recorder records the conversation between the store staff and the customer while they are explaining about new products. The input is the conversation with the customer, and the output is the voice data (e.g., audio123.wav). After the voice data is collected, it is sent to the server via the network.

[1224] Step 2:

[1225] The server converts the received voice data into text data. Specifically, it uses the Google Cloud Speech-to-Text API to convert the voice data (input: audio123.wav) into text data (output: text data "What are the features of the new product?"). The converted text data is stored in the server's database.

[1226] Step 3:

[1227] The server collects search data from online search systems. Specifically, it uses the Google Search API to obtain search ranking data for keywords related to new products (input: API request). The output is the search keywords and their ranking data (e.g., "new product features," "new product price," etc.). This data is also stored in the server's database.

[1228] Step 4:

[1229] The server uses the collected voice data and search data to train an AI model. Specifically, it uses a machine learning library (e.g., TensorFlow) to train the AI ​​model based on natural language processing algorithms (input: text data and search data). The output is a trained AI model. This AI model has the ability to analyze customer responses and concerns.

[1230] Step 5:

[1231] The server uses the trained AI model to generate AI customer avatars based on multiple attributes and run simulations. Specifically, it uses CAIL (Cognitive AI Library) to generate multiple customer avatars based on attribute information such as age, gender, and purchase history (input: attribute information). The output is the generated AI customer avatar, which is then used to run simulations and predict customer reactions and concerns.

[1232] Step 6:

[1233] The server generates a virtual FAQ based on the simulation results. Specifically, it uses the Python Flask framework to create an FAQ page containing frequently asked questions and their answers (input: simulation results). The output is a virtual FAQ page (e.g., "What are the features of the new product?" and its answer).

[1234] Step 7:

[1235] The server generates learning content and promotional tools based on the analysis results. Specifically, it creates learning modules to be installed on e-learning platforms (e.g., Moodle) (input: simulation results), and automatically generates promotional brochures and posters using Adobe InDesign scripts (input: simulation results). The output is learning content and promotional tools (e.g., brochures and posters).

[1236] Step 8:

[1237] The server provides the generated virtual FAQs, learning content, and promotional tools to users via the interface module. Specifically, it creates a login page for a web portal from which store staff can access the FAQs and learning content (input: generated content and tools). The output is the content and tools provided in a form that users can use (e.g., web links, QR codes).

[1238] (Application example 1)

[1239] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1240] To improve the efficiency and accuracy of customer service, it is necessary to analyze customer reactions and concerns in real time and enable store staff to quickly respond appropriately based on that information. However, traditional systems struggled to efficiently integrate these functions, and lacked support tools to improve the quality of customer service. Furthermore, there were insufficient means to identify customer questions and concerns in advance when introducing new services, and to provide effective learning content and promotional tools. As a result, customer satisfaction declined, hindering sales growth.

[1241] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1242] In this invention, the server includes means for collecting voice data, means for converting the collected voice data into text data, means for collecting search data, means for analyzing customer responses and concerns using the voice data and the search data, means for generating a virtual FAQ based on the analysis results, means for generating learning content and promotional tools based on the analysis results, means for displaying the generated virtual FAQ, learning content, and promotional tools, and means for providing a smartphone application that supports customer service using the virtual FAQ, learning content, and promotional tools generated based on the analysis results. This enables real-time understanding of customer responses and concerns and prompt appropriate responses. Furthermore, by identifying customer questions and concerns in advance when introducing a new service and providing effective learning content and promotional tools, the burden on store staff can be reduced, leading to improved customer satisfaction and increased sales.

[1243] "Voice data" refers to acoustic signal data that records conversations between customers and store staff.

[1244] "Text data" refers to data obtained by converting voice data into text using voice recognition technology.

[1245] "Search Data" refers to data that includes customer online search behavior and ranking information.

[1246] "Customer reactions and concerns" refers to feedback and reactions such as interest, questions, and concerns that customers show in-store.

[1247] "Analysis" refers to the act of using AI models and natural language processing algorithms to analyze customer responses and concerns from voice and search data.

[1248] A "virtual FAQ" is a virtual FAQ (Frequently Asked Questions) that is automatically generated based on the analysis results and contains questions that customers frequently ask and their answers.

[1249] "Learning content" refers to educational materials and content provided to store staff to help them learn about new services.

[1250] "Sales promotional tools" are promotional materials and tools such as POPs and pamphlets used to promote the sale of products and services.

[1251] "Display" refers to the act of visually providing the generated virtual FAQs, learning content, and promotional tools to users.

[1252] A "smartphone application" is a software program that is installed on a smartphone device to assist with customer service.

[1253] Information gathering phase

[1254] To collect voice data, the server records conversations between customers and store staff using a voice recorder used in the store. The recorded voice data is sent to the server in real time. At the same time, the server obtains user search ranking data from an online search system, thereby collecting information related to the customer's interests and questions.

[1255] Data analysis phase

[1256] The server converts the received voice data into text data using speech recognition technology. Services such as Google Cloud Speech-to-Text and IBM Watson Speech to Text are used for speech recognition. Next, an AI model (e.g., BERT or GPT-4) is trained based on the voice data and search data. This AI model uses natural language processing algorithms to automatically analyze customer responses and concerns. The server also generates an AI model based on multiple attributes and performs simulations. Google Cloud Natural Language and Amazon Comprehend are commonly used NLP services.

[1257] FAQ and tool generation phase

[1258] The server generates a virtual FAQ based on the analysis results. This virtual FAQ contains frequently asked questions and their answers, and aggregates information useful for responding to customers. Based on the same results, the server also automatically generates learning content and promotional tools (e.g., POPs, pamphlets, etc.) for store staff to learn about new services. The generated virtual FAQ, learning content, and promotional tools are stored on the server.

[1259] Results delivery phase

[1260] The generated virtual FAQs, learning content, and promotional tools are provided to store staff via a smartphone application. Staff can view these through the application interface. This application enables staff to respond to customer questions quickly and accurately. Staff can also use the learning content to efficiently acquire knowledge about new services.

[1261] Specific examples

[1262] For example, when a physical store introduces a new electronic payment service, if a customer asks staff questions about how to use it, fees, or security, the store can use the app to get real-time answers.

[1263] Prompt Sentence Examples

[1264] Customer: "Can you show me how to use the new electronic payment service?"

[1265] Application: "The basic steps for using the new electronic payment service are to first download the app and then create an account. Detailed instructions can be found in the following FAQ."

[1266] In this way, the system of the present invention can grasp and analyze customer reactions and concerns in real time, creating an environment in which they can be responded to efficiently, thereby improving customer satisfaction and increasing sales.

[1267] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1268] Step 1: Collecting audio data

[1269] The user (store staff) uses a voice recorder to record conversations with customers to collect voice data. After the voice data is collected, it is sent to the server in real time. The input is the voice data, and the output is the voice data sent to the server.

[1270] Step 2: Convert audio data to text

[1271] The server converts the received voice data into text data using speech recognition technology. This process uses services such as Google Cloud Speech-to-Text and IBM Watson Speech to Text. The input is voice data and the output is text data. Specific operations include the process of converting voice data into a string of characters.

[1272] Step 3: Collect search data

[1273] The server retrieves user search ranking data from the online search system. This data indicates what questions and keywords customers frequently search. The input is the search data retrieved from the online search system, and the output is the search data used for analysis.

[1274] Step 4: Analyze customer reactions and concerns

[1275] The server trains an AI model (e.g., BERT or GPT-4) based on the received text data and search data to analyze customer reactions and concerns. It uses an NLP algorithm, such as Google Cloud Natural Language or Amazon Comprehend. The input is the text data and search data, and the output is the analysis results of customer reactions and concerns. Specific operations include inputting data, training the model, and analyzing reactions and concerns.

[1276] Step 5: Generate a virtual FAQ

[1277] The server generates a virtual FAQ based on the analysis results. This virtual FAQ contains frequently asked questions and their answers from customers. The input is the analysis results on customer reactions and concerns, and the output is the virtual FAQ. Specific operations include generating patterns of questions and answers.

[1278] Step 6: Generate learning content and collateral

[1279] The server automatically generates learning content and promotional tools based on the same analysis results. The generated learning content includes educational materials about new services, and the promotional tools include POP displays and pamphlets. The input is the analysis results regarding customer reactions and concerns, and the output is learning content and promotional tools. Specific operations include generating educational materials and promotional materials.

[1280] Step 7: Delivering results

[1281] The generated virtual FAQs, learning content, and promotional tools are provided to store staff via a smartphone application. Users (store staff) can use this application to access the generated information and quickly and accurately answer customer questions. The input is the virtual FAQs, learning content, and promotional tools, and the output is customer support based on them. Specific operations include displaying and using the information via the smartphone application.

[1282] Through these steps, the inventive system is able to grasp customer reactions and concerns in real time, enabling appropriate responses to be taken quickly.

[1283] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1284] The present invention combines a system that uses voice data and search data to analyze customer reactions and concerns, and generates and provides virtual FAQs, learning content, and sales promotion tools based on the results, with an emotion engine that recognizes user emotions. Specific embodiments of the present invention will be described below.

[1285] System Overview

[1286] Information gathering phase

[1287] The terminal uses a voice recorder used in the store to record conversations with customers. This conversation includes questions and feedback from customers. The server collects voice data from the voice recorder at regular intervals and receives it via the network.

[1288] The server sends the voice data to the emotion engine for emotion recognition. The voice data is then sent to the speech recognition engine for conversion into text data. The converted text data and the recognized emotion data are stored in a database. At the same time, the server obtains the user's search ranking data and stores this data in the database.

[1289] Data analysis phase

[1290] The server uses voice data, search ranking data, and sentiment data to begin training the AI ​​model, which uses natural language processing (NLP) algorithms to better analyze customer reactions and concerns.

[1291] The server generates AI customer avatars based on multiple attributes and simulates how each avatar will react and feel toward new services.

[1292] FAQ and tool generation phase

[1293] The server generates a virtual FAQ based on the simulation results, which contains questions and answers based on each customer avatar's reactions, concerns, and emotional data.

[1294] The server generates learning content based on the simulation results. This includes manuals and online learning materials to help store staff deepen their understanding of the new service. It also takes into account emotional data to identify points requiring special attention and specific countermeasures.

[1295] The server automatically generates promotional tools, such as POP displays and pamphlets, to effectively introduce and advertise new services.

[1296] Results delivery phase

[1297] The generated virtual FAQs, learning content, and sales promotion tools are provided to the store staff (users) via an interface module in an accessible state, allowing users to make the necessary preparations before introducing the service.

[1298] For example, when users (store staff) prepare to serve customers before introducing a new service, they can check FAQs for specific questions and points to note based on emotional data, making it easier to elicit positive customer reactions to the new service.

[1299] In this way, the system predicts customer reactions and concerns when introducing new services and provides appropriate responses, thereby increasing customer satisfaction and supporting the success of the service. The addition of an emotion engine makes it possible to more accurately grasp the customer's emotional state and respond in a way that reflects that information, resulting in even more effective service provision.

[1300] The processing flow will be explained below.

[1301] Specific flow of program processing

[1302] Information gathering phase

[1303] Step 1:

[1304] The terminal uses a voice recorder used in the store to record conversations with customers, including questions and feedback from the customers.

[1305] Step 2:

[1306] The server collects the audio data from the voice recorder at regular intervals, which involves the process of transferring the data over the network.

[1307] Step 3:

[1308] The server sends the collected voice data to an emotion engine to recognize the customer's emotions, for example, whether the customer is expressing negative emotions or interest in the service.

[1309] Step 4:

[1310] The server sends the voice data to a speech recognition engine and converts it into text data. The converted text data and the recognized emotion data are stored in a database.

[1311] Step 5:

[1312] The server retrieves user search ranking data, including search keywords and frequently asked questions related to the new service, and stores this information in a database.

[1313] Data analysis phase

[1314] Step 6:

[1315] The server then uses the collected voice data, search ranking data, and sentiment data to begin training the AI ​​model, using natural language processing (NLP) algorithms.

[1316] Step 7:

[1317] The server uses an AI model to analyze customer reactions and concerns, such as which parts of the service customers are likely to have questions about and what emotions they are likely to experience.

[1318] Step 8:

[1319] The server generates AI customer avatars based on multiple attributes, including age, gender, and region, and simulates how each avatar would react to new services.

[1320] FAQ and tool generation phase

[1321] Step 9:

[1322] Based on the simulation results, the server generates a virtual FAQ, which includes questions and answers based on each customer avatar's reactions, concerns, and emotional data.

[1323] Step 10:

[1324] The server generates learning content based on the simulation results. This includes manuals and online learning materials to help store staff deepen their understanding of the new service. It also takes into account emotional data to identify points requiring special attention and specific countermeasures.

[1325] Step 11:

[1326] The server automatically generates promotional tools, such as POP displays and pamphlets, to effectively introduce and advertise new services.

[1327] Results delivery phase

[1328] Step 12:

[1329] Users (store staff) can access the generated virtual FAQs, learning content, and sales promotion tools through the interface module, allowing users to make the necessary preparations before introducing new services.

[1330] Step 13:

[1331] When users actually introduce new services, they can use the various tools and content generated to respond to customers. For example, they can increase customer satisfaction by quickly responding to customer questions based on FAQs and implementing methods to alleviate customer concerns based on emotional data.

[1332] Example 2

[1333] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1334] Conventional customer support systems have difficulty accurately grasping customer reactions and emotions, making it difficult to provide appropriate responses quickly. Furthermore, there is a lack of technology to effectively utilize customer feedback and search data to generate services and promotional tools. This makes it difficult to improve customer satisfaction and support the successful introduction of services.

[1335] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1336] In this invention, the server includes means for collecting voice data, means for transmitting the collected voice data to an emotion engine and recognizing customer emotions, means for converting the emotion-recognized voice data into text data, means for storing the converted text data and the recognized emotion data in a database, means for collecting search data, means for training an AI model using the voice data, the emotion data, and the search data to analyze customer reactions and concerns, means for generating an AI customer avatar based on the analysis results and simulating reactions and emotions to a new service, means for generating a virtual FAQ based on the simulation results, means for generating learning content and promotional tools based on the simulation results, and means for displaying the generated virtual FAQ, learning content, and promotional tools. This enables accurate understanding of customer emotions and prompt provision of appropriate countermeasures based on the understanding. Furthermore, customer feedback and search data can be utilized to generate effective virtual FAQs, learning content, and promotional tools, thereby improving customer satisfaction and supporting successful service introduction.

[1337] "Voice data" refers to information recorded in digital format from the conversations and voices of people such as customers and staff.

[1338] "Emotion engine" is a general term for algorithms and technologies that analyze voice data and identify the emotional state contained within it (e.g., joy, anger, sadness, etc.).

[1339] "Text data" is information in the form of a string of characters converted from voice data by a voice recognition engine.

[1340] A "database" is a system or device for systematically storing and managing collected and converted data.

[1341] "Search Data" means records and statistical information about users' search behavior on the Internet and internal systems.

[1342] "AI Model" means an artificial intelligence model trained using machine learning and natural language processing algorithms and used to analyze customer responses and concerns.

[1343] "Customer avatars" are virtual customer profiles generated based on AI models to simulate reactions and emotions toward new services.

[1344] "Simulation results" are the predicted results of virtual reactions and emotions by AI customer avatars.

[1345] A "virtual FAQ" is a virtual document that compiles frequently asked questions from customers and their answers, generated based on the simulation results.

[1346] "Learning content" refers to educational materials and online learning materials that help store staff understand new services and use them to assist them in dealing with customers.

[1347] "Promotional tools" are promotional materials (e.g., POPs, pamphlets, etc.) used to effectively introduce and advertise new services.

[1348] An "interface module" is a system component or software that provides the generated virtual FAQs, learning content, and promotional tools in an accessible form to users.

[1349] "Users" are store staff and other related parties who operate the system and use the generated virtual FAQs, learning content, and promotional tools.

[1350] The present invention combines a system that uses voice data and search data to analyze customer reactions and concerns, and generates and provides virtual FAQs, learning content, and sales promotion tools based on the results, with an emotion engine that recognizes user emotions. Specific embodiments of the present invention will be described below.

[1351] 1. Information gathering phase

[1352] The terminal uses a voice recorder used in the store to record conversations with customers. This conversation includes questions and feedback from customers. The terminal sends this voice data to a server at regular intervals. The voice data is recorded with a timestamp, making it possible to determine the exact point in time at which customer feedback was given.

[1353] 2. Emotion and voice recognition phase

[1354] The server sends the received voice data to an emotion engine to recognize the customer's emotions. The emotion engine uses a voice analysis algorithm to identify emotional states such as "joy," "anger," and "sadness." The server then sends the voice data to a speech recognition engine, which converts it into text data. The converted text data is a string of characters that directly represents the content of the conversation, including specific questions and feedback.

[1355] 3. Data storage phase

[1356] The server stores the converted text data and the recognized emotion data in a database. The database is used to identify changes and patterns over time. At the same time, the server obtains users' search ranking data and stores this data in the database. This search data includes the keywords and topics that users most frequently searched for over a certain period of time.

[1357] 4. AI model training phase

[1358] The server trains an AI model using voice data, search ranking data, and sentiment data. The AI ​​model uses natural language processing (NLP) algorithms to more accurately analyze customer reactions and concerns. The AI ​​model also generates an AI customer avatar based on multiple attributes. The customer avatar has a virtual profile and simulates reactions and emotions toward new services.

[1359] 5. FAQ and tool generation phase

[1360] The server generates a virtual FAQ based on the simulation results. This FAQ includes questions and answers based on the reactions, concerns, and emotional data of each customer avatar. The server also generates learning content and promotional tools based on the simulation results. The learning content includes manuals and online learning materials to help store staff deepen their understanding of the new service. Points that require particular attention and specific countermeasures are also added, improving the quality of customer service. The promotional tools include POP displays and pamphlets to effectively introduce and promote the new service.

[1361] 6. Results delivery phase

[1362] The generated virtual FAQs, learning content, and sales promotion tools are provided to the store staff (users) via the interface module. This allows the users to make the necessary preparations before introducing the service. For example, when users (store staff) prepare to deal with customers before introducing a new service, they can check FAQs for specific questions and points to note based on emotion data. This makes it easier to elicit a positive customer response to the new service.

[1363] Specific examples

[1364] For example, when a user is preparing to introduce a new service, they can input a prompt such as, "Please tell us the questions that customers are most likely to ask when the new service is introduced. Also, please simulate appropriate answers and customer reactions." Based on this prompt, the server uses an AI model to generate the most appropriate virtual FAQ and provides questions and answers based on emotional data. In this way, users can smoothly introduce new services and increase customer satisfaction.

[1365] This system efficiently utilizes voice and search data to accurately grasp customer emotions and reactions, enabling it to provide prompt and appropriate responses. Furthermore, by combining a generative AI model with an emotion engine, the system achieves more effective service delivery.

[1366] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1367] Step 1:

[1368] The device records conversations with customers using a voice recorder.

[1369] Specific operation: The terminal activates a voice recorder installed in the store and records the conversation between the customer and the staff. The voice data is recorded with a timestamp and saved on the terminal.

[1370] Step 2:

[1371] The server sends the voice data to an emotion engine to recognize the customer's emotion.

[1372] Input: Audio data received from the device

[1373] Output: Recognized emotion data

[1374] Specific operation: The server periodically transmits voice data acquired from the device to the emotion engine. The emotion engine uses a voice analysis algorithm to identify emotional states such as "joy," "anger," and "sadness." The recognized emotions are returned to the server in data format.

[1375] Step 3:

[1376] The server sends the voice data to a voice recognition engine and converts it into text data.

[1377] Input: Audio data received from the device

[1378] Output: Converted text data

[1379] Specific operation: The server sends the voice data to a speech recognition engine, which converts the voice into text. For example, the voice saying "I want to know more about this new service" is converted into text "I want to know more about this new service."

[1380] Step 4:

[1381] The server stores the text data and emotion data in a database.

[1382] Input: converted text data and recognized emotion data

[1383] Output: Data stored in the database

[1384] Specific operation: The server stores the converted text data and emotion data in a database, including timestamps and conversation context.

[1385] Step 5:

[1386] The server retrieves the user's search ranking data and stores it in a database.

[1387] Input: Search data from websites or internal systems

[1388] Output: Search ranking data stored in a database

[1389] What it does: The server retrieves user search ranking data from the internet or internal systems and stores it in a database, including the most searched keywords and topics for the past week.

[1390] Step 6:

[1391] The server trains the AI ​​model using voice data, search ranking data, and emotion data.

[1392] Input: Voice data, search ranking data, emotion data

[1393] Output: A trained AI model

[1394] How it works: The server feeds this data to an AI model, which trains it with natural language processing (NLP) algorithms. The AI ​​model is trained to analyze customer reactions and concerns.

[1395] Step 7:

[1396] The server generates AI customer avatars and simulates their reactions and emotions to new services.

[1397] Input: A trained AI model

[1398] Output: Simulated customer avatar reactions and emotions

[1399] How it works: The server uses the trained AI model to generate multiple customer avatars, each with a virtual profile that simulates their reactions and emotions to the new service.

[1400] Step 8:

[1401] The server generates a virtual FAQ based on the simulation results.

[1402] Input: Simulation results

[1403] Output: The generated virtual FAQ

[1404] Specific operation: Based on the simulation results of the AI ​​customer avatar, the server automatically generates a virtual FAQ that compiles frequently asked questions from customers and their answers.

[1405] Step 9:

[1406] The server generates learning content and promotional tools based on the simulation results.

[1407] Input: Simulation results

[1408] Output: Generated learning content and promotional tools

[1409] Specific operation: Based on the simulation results of the AI ​​customer avatar, the server generates learning content (e.g., manuals, online learning materials) and promotional tools (e.g., POPs, pamphlets) for store staff.

[1410] Step 10:

[1411] The server provides the generated virtual FAQs, learning content, and promotional tools to the user via an interface module.

[1412] Input: Generated virtual FAQs, learning content, promotional tools

[1413] Output: Information provided to the user

[1414] Specific operation: The server sends the generated virtual FAQ, learning content, and sales promotion tools to the interface module, making them accessible to users. Through this, users (store staff) can view and use the information they need.

[1415] (Application example 2)

[1416] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1417] In factory work environments, there is a need to reduce the confusion and mistakes that occur when workers introduce new robots and equipment, thereby improving production efficiency and safety. However, conventional systems generally use standalone FAQs and operation manuals, which do not fully consider workers' reactions and emotions. As a result, it is difficult to quickly and accurately respond to actual problems and concerns on the shop floor.

[1418] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting voice data, means for converting the collected voice data into text data, means for collecting search data, means for analyzing the worker's reactions and concerns using the voice data and the search data, means for generating a virtual FAQ based on the analysis results, means for recognizing the worker's emotional state based on the analysis results and generating an operation manual, means for displaying the generated virtual FAQ, operation manual, and warning system, means for training an AI model based on multiple task data, and means for presenting improvement suggestions and points to note based on simulation results generated by the AI ​​model. This makes it possible to grasp the worker's emotions and actual reactions in real time and provide prompt and accurate countermeasures.

[1419] "Voice data" refers to voice information collected from workers and equipment at the work site.

[1420] "Means for collecting voice data" refers to devices and methods for recording and collecting voice from workers, such as voice data acquisition devices and microphones within the factory.

[1421] "Text data" refers to character information obtained by analyzing voice data.

[1422] "Means for converting voice data into text data" refers to a device or method that converts collected voice data into text information using voice recognition software or the like.

[1423] "Search data" refers to information such as workers' search behavior and frequently used keywords.

[1424] "Means for collecting search data" refers to devices and methods for acquiring and recording the keywords searched by workers and their behavioral history.

[1425] "Means for analyzing reactions and concerns" refers to algorithms or software that analyze worker reactions and concerns based on collected data.

[1426] "Virtual FAQ" refers to a collection of hypothetical questions and answers based on each worker's reactions and concerns.

[1427] "Means for generating virtual FAQs" refers to a device or method that automatically generates answers to problems or questions that workers may encounter based on the results of data analysis.

[1428] "Means for recognizing emotional states" refers to algorithms or software that identify and analyze emotions from the tone and content of a worker's voice.

[1429] An "operation manual" refers to a document or guideline that explains how to operate a device or system.

[1430] "Means for generating an operating manual" refers to a device or method for creating an optimized operating procedure manual based on emotion data and reaction data.

[1431] "Warning system" refers to a notification system that alerts workers to important points and dangers.

[1432] "Means for displaying the generated virtual FAQ, operation manual, and warning system" refers to devices and methods for providing information visually or audibly to workers.

[1433] "Means for training AI models" refers to algorithms and software used to train machine learning models using collected data.

[1434] "Means for presenting improvement suggestions and points to note" refers to devices and methods that provide workers with suggestions and points to note for improving work efficiency and safety based on the results generated by a trained AI model.

[1435] System program generation

[1436] In the system that realizes this application example, the server performs processing in the following steps. First, to collect voice data, it collects voice data from voice data capture devices and microphones in the factory. Next, it converts the collected voice data into text data using voice recognition software (e.g., Google Cloud Speech-to-Text). Simultaneously with the converted text data, it extracts emotion data using an emotion recognition engine (e.g., EmotionAPI). This data is then stored in a database (e.g., MySQL).

[1437] Next, the server collects search data, including keywords used by the user and their search history. This information is also stored in a database.

[1438] The server analyzes the worker's reactions and concerns using the collected voice data, text data, search data, and emotion data. It uses NLP algorithms (e.g., spaCy) and machine learning models (e.g., TensorFlow) for the analysis. It generates a virtual FAQ based on the analysis results. It also recognizes the worker's emotional state based on the emotion data and generates an operation manual. An interface module (e.g., React.js) is used to present these results to the worker visually or audibly.

[1439] Furthermore, the AI ​​model is trained based on multiple task data sets, and based on the simulation results generated from the trained AI model, it presents improvement suggestions and points to be aware of to the worker.

[1440] Data Processing Description

[1441] The server converts the voice data collected from the voice data capture device and microphone into text data using Google Cloud Speech-to-Text. At the same time, it extracts emotion data using the Emotion API. This text data and emotion data are stored in a database (MySQL). Search data is tracked by the worker's search behavior and stored in the database.

[1442] The server analyzes the collected data using spaCy and TensorFlow to generate a virtual FAQ. It also recognizes the worker's emotional state based on the emotion data and generates an operation manual or warning system as needed. The generated virtual FAQ, operation manual, and warning system are presented to the worker using an interface module (React.js).

[1443] Examples of concrete examples and prompts

[1444] For example, if a new robot is being introduced at a factory and staff are confused, the system can extract concerns such as "The machine suddenly stopped working" or "I don't understand the error message" from the conversation data and automatically generate FAQs, which are then displayed on the smart glasses' display to assist with countermeasures in real time.

[1445] Example prompt sentence:

[1446] "Analyze staff conversation data regarding the introduction of a new factory robot to identify sources of confusion and bewilderment. Also, generate FAQs and operation manuals that take their emotional state into account."

[1447] This makes it possible to grasp the emotions and actual reactions of workers in real time and provide quick and appropriate countermeasures.

[1448] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1449] Step 1:

[1450] Audio data collection

[1451] Voice data acquisition devices (microphones, etc.) in the factory collect the voices of workers in real time. The collected voice data is sent to a server via a network. The input is the voice data of the workers, and the output is the voice data sent to the server.

[1452] Step 2:

[1453] Converting audio data to text

[1454] The server converts the received voice data into text data using speech recognition software (Google Cloud Speech-to-Text). The input is voice data and the output is text data. Specifically, the server sends the voice file to the API and receives the returned text.

[1455] Step 3:

[1456] Extracting Emotional Data

[1457] After obtaining text data from the voice data, the server uses an emotion recognition engine (Emotion API) to extract emotional data. The input is voice data and the output is emotional data. Specifically, the voice data is sent to the emotion recognition API, and emotion labels and numerical values ​​are received as analysis results.

[1458] Step 4:

[1459] Data storage

[1460] The server stores the converted text data and extracted emotion data in a database (MySQL). The input is text data and emotion data, and the output is the data stored in the database. Specifically, it executes SQL queries to insert each data into the appropriate table.

[1461] Step 5:

[1462] Search data collection

[1463] The server collects the keywords searched by the worker and their behavioral history and stores them in a database. The input is the search query, and the output is the search data stored in the database. Specifically, the server captures the search history and inserts it into the appropriate table.

[1464] Step 6:

[1465] Analysis of reactions and concerns

[1466] The server uses the collected voice, text, emotion, and search data to analyze the worker's reactions and concerns using an NLP algorithm (spaCy) and a machine learning model (TensorFlow). The input is the aforementioned data set, and the output is the analysis results. Specifically, the text data is tokenized and input into the model to obtain prediction results.

[1467] Step 7:

[1468] Virtual FAQ generation

[1469] The server generates a virtual FAQ based on the analysis results of reactions and concerns. The input is the analysis results, and the output is the virtual FAQ. Specifically, the analysis results are applied to a pre-prepared template to generate an FAQ document.

[1470] Step 8:

[1471] Generate operation manuals

[1472] The server recognizes the worker's emotional state based on the emotional data and generates an operation manual. The input is the emotional data and analysis results, and the output is the operation manual. Specifically, it generates a manual document with additional precautions and procedures according to the worker's emotional state.

[1473] Step 9:

[1474] Generate an alert system

[1475] The server generates a warning system based on the generated operation manual and emotion data. The input is the operation manual and emotion data, and the output is a warning message. Specifically, it sets up notifications to notify users of dangerous situations and important precautions in real time.

[1476] Step 10:

[1477] Presentation of information

[1478] Users use an interface module (React.js) to visually or audibly access and display the generated virtual FAQ, operation manual, and warning system. The input is the various documents generated, and the output is the information presented visually or audibly. Specifically, the information is displayed through a web application.

[1479] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1480] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1481] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1482] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1483] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1484] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1485] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1486] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1487] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1488] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1489] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1490] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1491] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1492] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1493] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1494] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1495] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1496] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1497] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1498] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1499] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1500] The following is further disclosed regarding the above embodiment.

[1501] (Claim 1)

[1502] means for collecting audio data;

[1503] A means for converting the collected voice data into text data;

[1504] a means for collecting search data;

[1505] means for analyzing customer responses and concerns using the voice data and the search data;

[1506] A means for generating a virtual FAQ based on the analysis results;

[1507] means for generating learning content and promotional tools based on the analysis results;

[1508] a means for displaying the generated virtual FAQs and learning content, and promotional tools;

[1509] A system including:

[1510] (Claim 2)

[1511] 2. The system according to claim 1, wherein the means for collecting voice data acquires the voice data from a voice recorder used in the store.

[1512] (Claim 3)

[1513] 2. The system according to claim 1, wherein the means for collecting search data acquires user search rankings.

[1514] "Example 1"

[1515] (Claim 1)

[1516] means for collecting audio data;

[1517] A means for converting the collected voice data into text data;

[1518] a means for collecting search data;

[1519] means for training an AI model using a natural language processing algorithm to analyze customer responses and concerns using the voice data and the search data;

[1520] A means for generating a virtual FAQ based on the analysis results;

[1521] means for generating learning content and promotional tools based on the analysis results;

[1522] A means for generating and simulating AI customer avatars based on multiple attributes;

[1523] a means for displaying the generated virtual FAQs and learning content, and promotional tools;

[1524] A system including:

[1525] (Claim 2)

[1526] 2. The system according to claim 1, wherein the means for collecting voice data acquires the voice data from a voice recorder used in the store.

[1527] (Claim 3)

[1528] 2. The system according to claim 1, wherein the means for collecting search data acquires user search ranking data.

[1529] "Application Example 1"

[1530] (Claim 1)

[1531] means for collecting audio data;

[1532] A means for converting the collected voice data into text data;

[1533] a means for collecting search data;

[1534] means for analyzing customer responses and concerns using the voice data and the search data;

[1535] A means for generating a virtual FAQ based on the analysis results;

[1536] means for generating learning content and promotional tools based on the analysis results;

[1537] a means for displaying the generated virtual FAQs and learning content, and promotional tools;

[1538] A means for providing a smartphone application that supports customer service by utilizing virtual FAQs, learning content, and sales promotion tools generated based on the analysis results; and

[1539] A system including:

[1540] (Claim 2)

[1541] The system of claim 1, wherein the means for collecting voice data acquires voice data from a voice recorder used in the store, and further supports staff training using virtual FAQs generated based on the analysis results.

[1542] (Claim 3)

[1543] 2. The system of claim 1, wherein the means for collecting search data obtains user search rankings, and further comprises an interface for displaying the generated virtual FAQs and learning content in real time.

[1544] "Example 2: Combining Emotion Engines"

[1545] (Claim 1)

[1546] means for collecting audio data;

[1547] a means for transmitting the collected voice data to an emotion engine to recognize the emotion of the customer;

[1548] A means for converting emotion-recognized voice data into text data;

[1549] a means for storing the converted text data and the recognized emotion data in a database;

[1550] a means for collecting search data;

[1551] means for training an AI model using the voice data, the emotion data, and the search data to analyze customer responses and concerns;

[1552] A means for generating AI customer avatars based on the analysis results and simulating reactions and emotions to new services;

[1553] a means for generating a virtual FAQ based on the simulation results;

[1554] means for generating learning content and promotional tools based on the simulation results;

[1555] a means for displaying the generated virtual FAQs and learning content, and promotional tools;

[1556] A system including:

[1557] (Claim 2)

[1558] 2. The system according to claim 1, wherein the means for collecting voice data acquires the voice data from a voice recorder used in the store.

[1559] (Claim 3)

[1560] 2. The system according to claim 1, wherein the means for collecting search data acquires user search rankings.

[1561] "Application example 2 when combining emotion engines"

[1562] (Claim 1)

[1563] means for collecting audio data;

[1564] A means for converting the collected voice data into text data;

[1565] a means for collecting search data;

[1566] means for analyzing worker responses and concerns using the voice data and the searched data;

[1567] A means for generating a virtual FAQ based on the analysis results;

[1568] a means for recognizing the emotional state of the worker based on the analysis result and generating an operation manual;

[1569] a means for displaying the generated virtual FAQ and operation manual, and a warning system;

[1570] A means of training an AI model based on multiple task data;

[1571] A means for presenting improvement suggestions and points to be noted based on the simulation results generated from the AI ​​model;

[1572] A system including:

[1573] (Claim 2)

[1574] 2. The system according to claim 1, wherein the means for collecting voice data acquires the voice data from a voice data acquisition device in a factory.

[1575] (Claim 3)

[1576] 2. The system according to claim 1, wherein the means for collecting search data acquires search rankings of users. [Explanation of symbols]

[1577] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for collecting audio data; A means for converting the collected voice data into text data; a means for collecting search data; means for analyzing customer responses and concerns using the voice data and the search data; A means for generating a virtual FAQ based on the analysis results; means for generating learning content and promotional tools based on the analysis results; a means for displaying the generated virtual FAQs and learning content, and promotional tools; A system including:

2. 2. The system according to claim 1, wherein the means for collecting voice data acquires the voice data from a voice recorder used in the store.

3. 2. The system of claim 1, wherein the means for collecting search data acquires user search rankings.

Images