system

Abstract

Provide a system. The method includes: means for connecting an audio device to a user's mobile device; means for recording a conversation between a user and a customer via said audio device; means for transmitting the recorded voice data to a server in real time; means for converting voice data into text data in the server; A means for storing the converted text data in a database; means for retrieving relevant historical data from said database; A system that includes a means for providing search results to a user.

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In recent years, there has been a demand for stores to improve their complaint handling and customer service quality in order to increase customer satisfaction. However, many stores do not record interactions with customers, making it difficult to refer to past responses when a problem occurs. This can lead to inappropriate complaint handling and a loss of customer trust. Inventory checks are also often done manually, which can delay the provision of service to customers. Furthermore, there is also the problem that the customer service know-how of talented staff is not sufficiently shared, resulting in an inconsistency in the overall quality of customer service. An effective system is needed to overcome these challenges. [Means for solving the problem]

[0005] The present invention provides a means for connecting an audio device to a user's mobile device. It also includes a means for recording conversations between a user and a customer via the audio device and transmitting the audio data to a server in real time. It also includes a means for the server to convert the received audio data into text data and store the converted text data in a database. It also includes a means for searching the database for relevant past data in response to a user request and providing the search results to the user. It also includes a means for acquiring and providing inventory information in real time in response to a user request. It also aims to improve overall customer service quality by providing a means for collecting and sharing records of excellent employees' service.

[0006] An "audio device" is an electronic device for collecting and transmitting sound to a user's mobile device.

[0007] "User" refers to store staff or employees who use this system to deal with customers.

[0008] "Mobile Device" means a smartphone, tablet, or other portable electronic device used to process voice data in conjunction with the System.

[0009] "Server" refers to a computer system for receiving, processing, storing, and otherwise processing audio data.

[0010] "Audio Data" means digital audio files containing conversations between Users and Customers collected by an audio device.

[0011] "Text data" refers to character information converted from audio data.

[0012] "Database" refers to a digital storage system for storing and managing audio and text data.

[0013] "Inventory information" refers to data regarding the inventory status of products and goods at a store.

[0014] "Complaints" refer to dissatisfaction or complaints from customers, and the purpose is to record the content of those complaints.

[0015] "Interaction records" refers to the content of the interactions and responses a user has with customers, and includes saving and managing such data.

[0016] "Real-time" refers to data being processed and transferred almost simultaneously, without delay.

[0017] "Excellent employees" refer to store staff who are highly rated for their customer service and attitude.

[0018] "Sharing" refers to the act of making certain data or information accessible to multiple users. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8]FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0020] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0021] First, the terms used in the following description will be explained.

[0022] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0023] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0024] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0025] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0026] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0027] [First embodiment]

[0028] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0029] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0030] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0031] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0032] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0033] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0034] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0035] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0036] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0037] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0038] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0039] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0040] As an embodiment of the present invention, a system is constructed in which an audio device, a mobile device, and a server are linked together in order to improve the efficiency of customer service in a store.

[0041] 1. Basic system configuration

[0042] User (store staff): Uses this system to record conversations with customers, manage complaints, check inventory, and share customer service know-how.

[0043] Audio device: A device worn by the user that captures conversations with customers.

[0044] Mobile device (smartphone or tablet): Responsible for receiving audio data from the audio device and sending it to the server.

[0045] Server: Converts received voice data into text and stores it in a database. It also manages inventory information and provides information upon user request. It also collects and shares records of outstanding employees' interactions.

[0046] 2. System Operation Flow

[0047] 1. Connecting the user to the audio device

[0048] Device: The user powers on the audio device and launches the app on their mobile device, which pairs with the audio device via Bluetooth.

[0049] Server: If pairing is successful, the server receives the device ID, checks the device registration status, and approves the connection.

[0050] 2. Collection and transmission of voice data

[0051] Device: When a user initiates a customer interaction, the audio device collects the voice and transmits the collected voice data to the mobile device in real time.

[0052] Device: The mobile device temporarily stores the collected voice data and transfers it to the server in batches.

[0053] 3. Audio Data Processing

[0054] Server: Receives voice data and converts it into text data using a speech recognition engine.

[0055] Server: The converted text data is saved in a database.

[0056] 4. Search for past claims data

[0057] Users: Use the search function in the app when handling a complaint by entering keywords related to previous complaints.

[0058] Server: Searches for relevant historical data in the database and provides it to the user.

[0059] 5. Check inventory information

[0060] User: When a customer requests an inventory check, they enter the product name and barcode through the app.

[0061] Server: Retrieves inventory information from the database and provides it to users in real time.

[0062] Terminal: Displays inventory information to the user and communicates it to the customer.

[0063] 6. Sharing customer service know-how

[0064] Server: Continuously collects and stores records of excellent employees' responses in a know-how database.

[0065] Users: When they want to learn about a specific situation or how to respond in the app, they can select a category or enter keywords.

[0066] Server: Searches for relevant information from the know-how database and provides it to the user.

[0067] Examples:

[0068] Example 1: Handling complaints

[0069] If a customer complains that "the response to the previous product exchange was slow,"

[0070] User: Enters "product exchange" and performs a search.

[0071] Server: Searches past conversation records related to "product exchange" and provides three results to the user's device.

[0072] Users: Check past responses and use them as a reference for responding quickly.

[0073] Example 2: Checking inventory

[0074] If a customer asks, "Do you have this T-shirt in a medium size?"

[0075] User: Scans the product barcode to request an inventory check.

[0076] Server: Searches the inventory database, finds that there are 5 units of the item in stock, and sends the result to the user.

[0077] User: Tell customer "In stock."

[0078] Example 3: Sharing customer service know-how

[0079] If new staff want to learn "effective upselling methods,"

[0080] User: Searches for "upsell" in the app.

[0081] Server: Extracts effective phrases and techniques related to "upselling" from a know-how database and provides them to users.

[0082] User: Review the information provided and apply it to actual customer service.

[0083] As described above, the embodiment of the present invention realizes efficient customer service and improved service quality in stores through the specific operation procedures and processing flows.

[0084] The processing flow will be explained below.

[0085] Specific processing flow of the system

[0086] 1. Connecting the user to the audio device

[0087] Step 1:

[0088] User: Power on the audio device.

[0089] Step 2:

[0090] Device: Turn on Bluetooth on your smartphone and launch the dedicated app.

[0091] Step 3:

[0092] On your device: The app performs a Bluetooth scan to discover nearby audio devices.

[0093] Step 4:

[0094] Device: Displays a list of detected audio devices and lets the user select a device.

[0095] Step 5:

[0096] User: Approves to pair the selected device.

[0097] Step 6:

[0098] On the device: Notify the user that pairing was successful.

[0099] Step 7:

[0100] Server: Receives the device ID from the mobile device and checks the device registration status.

[0101] Step 8:

[0102] Server: Once registration is confirmed, approve the connection and send a notification to the device that the connection is complete.

[0103] 2. Collection and transmission of voice data

[0104] Step 1:

[0105] User: Presses the start recording button in the app.

[0106] Step 2:

[0107] Device: The audio device collects the conversation and transmits the voice data to the mobile device in real time.

[0108] Step 3:

[0109] Device: Your mobile device temporarily stores the audio data.

[0110] Step 4:

[0111] Terminal: Uploads audio data to the server periodically in batches.

[0112] 3. Audio Data Processing

[0113] Step 1:

[0114] Server: Receives the audio data.

[0115] Step 2:

[0116] Server: The speech recognition engine converts the voice data into text.

[0117] Step 3:

[0118] Server: The converted text data is saved in a database.

[0119] 4. Search for past claims data

[0120] Step 1:

[0121] User: When a complaint occurs, use the complaint search feature in the app.

[0122] Step 2:

[0123] User: Enter keywords related to the complaint in the search box.

[0124] Step 3:

[0125] Device: The app sends a search request to the server.

[0126] Step 4:

[0127] Server: Searches the database for relevant past conversation records.

[0128] Step 5:

[0129] Server: Returns search results to the user's device.

[0130] Step 6:

[0131] Device: Display search results to the user in a list format.

[0132] 5. Check inventory information

[0133] Step 1:

[0134] User: Receives a request from a customer to check inventory.

[0135] Step 2:

[0136] User: Use the in-app inventory check feature to enter the product name or barcode.

[0137] Step 3:

[0138] Terminal: Sends an inventory check request to the server.

[0139] Step 4:

[0140] Server: Retrieves product inventory information from the database.

[0141] Step 5:

[0142] Server: Returns inventory information to the user's device.

[0143] Step 6:

[0144] Terminal: Display inventory information to the user.

[0145] 6. Sharing customer service know-how

[0146] Step 1:

[0147] Server: Continuously collects and stores records of excellent employees' responses in a know-how database.

[0148] Step 2:

[0149] Users: When they want to learn about a specific situation or how to respond in the app, they can select a category or enter keywords.

[0150] Step 3:

[0151] Terminal: Sends a know-how search request to the server.

[0152] Step 4:

[0153] Server: Searches for relevant information from a know-how database.

[0154] Step 5:

[0155] Server: Sends search results to the user's device.

[0156] Step 6:

[0157] Terminal: Displays the provided know-how information to the user.

[0158] With the above processing flow, the system of the present invention can improve the efficiency of customer service for users, and can realize the sharing of know-how on handling complaints, checking inventory, and customer service.

[0159] Example 1

[0160] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0161] Conventional customer service systems have had issues with efficiently collecting and managing conversation records, quickly obtaining inventory information, and sharing excellent customer service know-how. In particular, there is a demand for systems that can efficiently process customer conversation data in real time, quickly search for past complaint handling records, and provide inventory information in a timely manner.

[0162] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0163] In this invention, the server includes means for collecting conversations via an audio device connected to the mobile terminal, means for temporarily storing the collected voice data and transmitting it to the server, and means for converting the voice data into text data and storing it in a database, thereby enabling real-time processing of conversation data with customers, rapid search of past complaint handling records, and timely provision of inventory information.

[0164] An "audio device" is an audio device that allows a user to collect conversations with customers.

[0165] "Mobile device" refers to the mobile device used by the user, such as a smartphone or tablet.

[0166] "Means of collection" refers to the function of using an audio device to record conversations between customers and users and capture them as data.

[0167] "Temporarily stored" means that the collected voice data is temporarily stored on the mobile device until it is batch processed.

[0168] "Batch processing" refers to the process of sending data to a server in bulk based on a set time or data size.

[0169] "Server" means a central processing unit for processing, storing, and providing information to users.

[0170] "Means for converting into text data" refers to the process of converting voice data into text information using a voice recognition engine.

[0171] A "database" is a data storage system for systematically storing and managing converted text data and related information.

[0172] "Searching means" refers to the functionality for locating specific information within a database.

[0173] "Means for providing search results" refers to the function of displaying the information obtained through a search on the user's mobile device.

[0174] "Means of obtaining inventory information in real time" refers to the process of instantly obtaining the latest inventory information from the database and providing it to the user.

[0175] "Excellent employee service records" refers to records of customer service provided by highly rated employees.

[0176] "Means for sharing" refers to the ability to share collected records of excellent employee responses with other users for learning and reference.

[0177] This invention is a system aimed at improving the efficiency of customer service in stores. By linking audio devices, mobile terminals, and servers, this system collects and converts conversation data with customers into text, and provides the functionality to store and search the data in a database.

[0178] Basic system configuration

[0179] User (store staff): Records conversations with customers and uses them to manage complaints, check inventory, and share customer service know-how.

[0180] Audio Device: A device for collecting audio. The user wears this device and records the conversation with the customer.

[0181] Mobile device (smartphone or tablet): A device that receives audio data from an audio device and sends it to a server.

[0182] Server: Converts voice data into text data and stores it in a database. It also provides users with related past data, inventory information, and customer service know-how.

[0183] Overall system operation

[0184] The user turns on the audio device and launches the dedicated app on the mobile device. The audio device connects to the mobile device via Bluetooth. The server receives the device ID, checks the registration status, and then approves the connection. When the user starts talking with a customer, the audio device collects voice data and transmits it to the mobile device in real time. The mobile device temporarily stores the collected voice data and transmits it to the server in batch processing.

[0185] Audio data processing

[0186] The server converts the received voice data into text data using a speech recognition engine (for example, IBM Watson (registered trademark) or Google (registered trademark) Cloud Speech-to-Text). This text data is then saved in a database. The saved data is searchable and can be used as a record of past complaint handling, inventory information, or records of outstanding employee responses.

[0187] Specific examples

[0188] Handling complaints

[0189] If a user receives a complaint from a customer that "the previous product exchange was handled slowly":

[0190] User: Enters "product exchange" into the app and performs a search.

[0191] Server: Searches the database and provides records related to past "commodity exchanges" to the user's device.

[0192] Users: Use search results to act quickly.

[0193] Stock Check

[0194] When a customer asks, "Do you have this item?"

[0195] User: Scans the product barcode to request an inventory check.

[0196] Server: Searches the database and provides real-time inventory status to users.

[0197] User: Tell customer "In stock."

[0198] Sharing customer service know-how

[0199] If you're a new staff member and want to learn "effective upselling methods":

[0200] User: Searches for "upsell" in the app.

[0201] Server: Provides users with relevant phrases and techniques from a know-how database.

[0202] User: Review the information provided and put it into practice when serving customers.

[0203] As a result, this system will improve the efficiency of customer service and service quality in stores.

[0204] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0205] Step 1:

[0206] Connecting your audio device to your mobile device

[0207] Input: The user turns on the audio device and launches a dedicated app on their mobile device.

[0208] Output: The audio device and mobile device are connected via Bluetooth.

[0209] Specific behavior:

[0210] User: Power on the audio device.

[0211] Device: Launch the dedicated app on your mobile device and pair it with your audio device via Bluetooth.

[0212] Server: If pairing is successful, it receives the device ID, checks the device registration status, and approves the connection.

[0213] Step 2:

[0214] User initiates customer interaction

[0215] Input: The user initiates a conversation with the customer.

[0216] Output: The conversation with the customer is collected as audio data.

[0217] Specific behavior:

[0218] User: Initiate a conversation with a customer.

[0219] Audio devices: Collect voice data from user and customer conversations.

[0220] Step 3:

[0221] Temporary storage and batch processing of audio data

[0222] Input: The collected voice data is sent to the mobile device.

[0223] Output: The audio data is temporarily stored and sent to the server in batch processing.

[0224] Specific behavior:

[0225] Audio device: Sends audio data to mobile devices in real time.

[0226] Terminal: Temporarily stores received audio data and starts batch processing when a certain time or data size is reached.

[0227] Terminal: Encrypts the audio data through batch processing and sends it to the server.

[0228] Step 4:

[0229] Converting audio data to text

[0230] Input: The server receives the audio data.

[0231] Output: The audio data is converted to text data.

[0232] Specific behavior:

[0233] Server: Converts the voice data into text data using a speech recognition engine (e.g., IBM Watson or Google Cloud Speech-to-Text).

[0234] Server: Validates the accuracy of the speech recognition results and makes corrections if necessary.

[0235] Step 5:

[0236] Text data storage in a database

[0237] Input: Text data is generated.

[0238] Output: The text data is saved in the database.

[0239] Specific behavior:

[0240] Server: The converted text data is saved in a database.

[0241] Server: Verify that the data was saved correctly and generate a success message.

[0242] Step 6:

[0243] Search past claims data

[0244] Input: The user types a keyword into the app.

[0245] Output: The relevant claim record is displayed on the user's terminal.

[0246] Specific behavior:

[0247] User: Enters a keyword, such as "exchange," into the app and performs a search.

[0248] Server: Searches the database, extracts relevant past complaint handling records, and provides them to the user.

[0249] Step 7:

[0250] Check stock information

[0251] Input: The user types a stock check request into the app.

[0252] Output: The inventory information is displayed on the user's terminal.

[0253] Specific behavior:

[0254] User: Enters product name or barcode into the app to request availability.

[0255] Server: Retrieves inventory information from the database and provides it to users in real time.

[0256] Device: Display inventory information on the user's device and communicate it to the customer.

[0257] Step 8:

[0258] Sharing customer service know-how

[0259] Input: The user types a specific keyword into the app.

[0260] Output: Related response know-how is displayed on the user's terminal.

[0261] Specific behavior:

[0262] User: Enters a keyword into the app, such as "upsell," and performs a search.

[0263] Server: Extracts relevant phrases and techniques from the know-how database and provides them to users.

[0264] User: Review the information provided and put it into practice in customer service.

[0265] (Application example 1)

[0266] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0267] Conventional factory management systems rely on manual processes for managing work orders, checking inventory, handling complaints, and sharing work know-how, which can be inefficient and prone to errors. Performing these tasks efficiently is particularly difficult in large factories. This has led to a demand for automating various tasks within factories, processing data in real time, and improving work efficiency.

[0268] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0269] In this invention, the server includes means for connecting an audio device to a user's mobile device, means for recording conversations between the user and customers via the audio device, means for transmitting the recorded audio data to the server in real time, means for converting the audio data to text data in the server, means for storing the converted text data in a database, means for searching related past data from the database, means for providing the search results to the user, means for inputting and managing work instructions by voice, means for checking inventory information in real time, means for recording and searching complaint responses, and means for classifying and sharing excellent work records, which enables improved work efficiency in factories, more accurate inventory management, faster complaint responses, and the sharing of work know-how.

[0270] "Audio Device" means a device that collects audio and is connected to a User's Mobile Device.

[0271] A "mobile device" is a portable information terminal such as a smartphone or tablet.

[0272] A "recording means" is a method or apparatus for storing audio data collected via an audio device.

[0273] "Means for transmitting in real time" refers to a method or device for instantly transmitting collected voice data to a server.

[0274] A "server" is a central processing unit that processes and stores information on a network.

[0275] The "means for converting into text data" refers to a method or device that has the function of converting voice data into text information.

[0276] A "database" is a system for centrally storing and managing information.

[0277] A "searching means" is a method or device for efficiently locating specific information within a database.

[0278] "Means of providing" refers to the method or device for displaying search results and data to users.

[0279] "Means for managing work instructions" refers to a method or device for organizing and saving work instructions entered by voice so that they can be referenced as needed.

[0280] "Means for checking inventory information" refers to a method or device that allows you to grasp the inventory status of a product in real time.

[0281] "Means for recording complaint handling" refers to a method or device for recording and saving customer complaints and requests as audio data.

[0282] "Means for classifying and sharing excellent work records" refers to methods and devices for collecting effective work methods and know-how, organizing and storing them, and sharing them with other users.

[0283] As an embodiment of the present invention, a method for constructing a factory management system will be described below.

[0284] 1. Basic system configuration

[0285] Users (factory workers): Use this system to input and manage work instructions by voice, check inventory, handle complaints, and share excellent work know-how.

[0286] Audio device: A device worn by the user that collects audio such as instructions and complaints within the factory.

[0287] Mobile device (smartphone or tablet): Temporarily stores audio data received from the audio device and sends it to the server.

[0288] Server: Converts received voice data into text and stores it in a database. It also manages inventory information and provides information upon user request. It also collects and shares excellent work records.

[0289] 2. System Operation Flow

[0290] 1. Connecting the user to the audio device

[0291] Device: The user powers on the audio device and launches the app on their mobile device, which pairs with the audio device via Bluetooth.

[0292] Server: If pairing is successful, the server receives the device ID, checks the device registration status, and approves the connection.

[0293] 2. Collection and transmission of voice data

[0294] Device: When the user begins to speak work instructions, the audio device collects the voice and transmits the collected voice data to the mobile device in real time.

[0295] Device: The mobile device temporarily stores the collected voice data and transfers it to the server in batches.

[0296] 3. Audio Data Processing

[0297] Server: Receives the voice data and converts it into text data using a speech recognition engine (e.g., Google Cloud Speech-to-Text).

[0298] Server: The converted text data is stored in a database (e.g., MySQL (registered trademark)).

[0299] 4. Search for past claims data

[0300] Users: Use the search function in the app when handling a complaint by entering keywords related to previous complaints.

[0301] Server: Searches for relevant historical data in the database and provides it to the user.

[0302] 5. Check inventory information

[0303] User: Enter the product name or barcode by voice.

[0304] Server: Retrieves inventory information from the database and provides it to users in real time.

[0305] Terminal: Displays inventory information to the user and communicates it to the customer.

[0306] 6. Share good work records

[0307] Server: Continuously collects voice data of excellent work and stores it in a know-how database.

[0308] Users: When they want to learn about a specific task or know-how in the app, they select a category or enter keywords.

[0309] Server: Searches for relevant information from the know-how database and provides it to the user.

[0310] Specific examples

[0311] Example 1: Entering and managing work orders

[0312] User: Speak instructions such as "Assemble 3 units of part A."

[0313] Server: Converts the received voice data into text and adds the instruction "Assemble 3 units of part A" to the work list.

[0314] Example 2: Checking inventory

[0315] User: Asks aloud, "What is the inventory of part A, barcode 1234?"

[0316] Server: Searches the inventory database and sends the result, "There are 5 units of part A with barcode 1234 in stock."

[0317] Example 3: Recording complaints

[0318] User: Speaks a complaint: "Machine X is not working properly."

[0319] Server: Converts the received voice data into text and stores it in the claim database.

[0320] User: Search for past complaint history regarding "Machine X".

[0321] Prompt Sentence Examples

[0322] (Voice input): "What is the current inventory?"

[0323] (Generated text): "Checking current inventory. Please wait."

[0324] In this way, the efficiency of work and the quality of service in the factory management system are improved.

[0325] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0326] Step 1:

[0327] Connecting users and audio devices

[0328] User: Power on the audio device and launch the app on their smartphone.

[0329] Input: User interaction to open the audio device and launch the app.

[0330] Output: The audio device and smartphone are paired.

[0331] Device: The app pairs with your audio device via Bluetooth.

[0332] Server: If pairing is successful, the server receives the device ID, checks the device's registration status, and approves the connection.

[0333] Step 2:

[0334] Collection and transmission of voice data

[0335] User: Start a conversation with customers or other workers.

[0336] Input: The user's voice conversation.

[0337] Output: Audio device collects audio data.

[0338] Device: The audio device sends audio data to the smartphone in real time.

[0339] Input: Audio data from the audio device.

[0340] Output: Audio data temporarily stored on your smartphone.

[0341] Terminal: The temporarily saved audio data is transferred to the server in batch processing.

[0342] Step 3:

[0343] Audio data processing

[0344] Server: Receives the voice data and analyzes it using a voice recognition engine.

[0345] Input: Audio data sent from the smartphone.

[0346] Output: Parsed text data.

[0347] Server: Converts voice data into text data using a speech recognition engine (e.g., Google Cloud Speech-to-Text).

[0348] Input: Speech data to the speech recognition engine.

[0349] Output: The content converted to text data.

[0350] Server: The converted text data is stored in a database (e.g. MySQL).

[0351] Step 4:

[0352] Search past claims data

[0353] User: When handling a complaint, use the search function in the app and enter keywords.

[0354] Input: Claim response keywords.

[0355] Output: Search results for historical claims data.

[0356] Server: Searches for relevant historical data in a database.

[0357] Input: A query search to the database.

[0358] Output: Claim data as search results.

[0359] Server: Serves search results to users.

[0360] Step 5:

[0361] Check stock information

[0362] User: Enter the product name or barcode by voice.

[0363] Input: Product name or barcode.

[0364] Output: Inventory information retrieved from the database.

[0365] Server: Retrieves inventory information from the database and provides it to users in real time.

[0366] Input: Inventory information from the database.

[0367] Output: Inventory information provided to the user.

[0368] Terminal: Displays inventory information to the user and communicates it to the customer.

[0369] Step 6:

[0370] Sharing excellent work records

[0371] Server: Continuously collects voice data of excellent work and stores it in a know-how database.

[0372] Input: Excellent work audio data.

[0373] Output: Text data stored in the know-how database.

[0374] Users: When they want to learn about a specific task or know-how in the app, they select a category or enter keywords.

[0375] Input: Keywords related to know-how.

[0376] Output: Relevant know-how information.

[0377] Server: Searches for relevant information from the know-how database and provides it to the user.

[0378] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0379] As a form for implementing the present invention, a system is constructed that combines an audio device, a mobile device, a server, and an emotion engine to improve the efficiency of customer service in stores, handle complaints, check inventory, and share customer service know-how.

[0380] 1. Basic system configuration

[0381] User (store staff): This system records conversations with customers, manages complaints, checks inventory, and shares customer service know-how. It also uses an emotion engine to understand emotional states in real time.

[0382] Audio device: A device worn by the user that captures conversations with customers.

[0383] Mobile device (smartphone or tablet): Responsible for receiving voice data from the audio device and sending it to the server and emotion engine.

[0384] Server: Converts received voice data into text and stores it in a database. It also manages inventory information and provides information upon user request. It also collects and shares records of outstanding employees' interactions.

[0385] Emotion Engine: Analyzes emotions from user and customer voice data to determine their emotional state in real time.

[0386] 2. System Operation Flow

[0387] 1. Connecting the user to the audio device

[0388] Device: The user powers on the audio device and launches the app on their mobile device, which pairs with the audio device via Bluetooth.

[0389] Server: If pairing is successful, the server receives the device ID, checks the device registration status, and approves the connection.

[0390] 2. Collection and transmission of voice data

[0391] Device: When a user initiates a customer interaction, the audio device collects the voice and transmits the collected voice data to the mobile device in real time.

[0392] Terminal: The mobile device temporarily stores the voice data and transfers it to the server and emotion engine in batch processing.

[0393] 3. Audio Data Processing

[0394] Server: Receives the voice data and converts it into text using a speech recognition engine.

[0395] Server: The converted text data is saved in a database.

[0396] 4. Emotional Data Processing

[0397] Emotion Engine: Analyzes emotional state from voice data and notifies the user in real time.

[0398] 5. Search for past claims data

[0399] Users: Use the search function in the app when handling a complaint by entering keywords related to previous complaints.

[0400] Server: Searches for relevant historical data in the database and provides it to the user.

[0401] 6. Check inventory information

[0402] User: When a customer requests an inventory check, they enter the product name and barcode through the app.

[0403] Server: Retrieves inventory information from the database and provides it to users in real time.

[0404] Terminal: Displays inventory information to the user and communicates it to the customer.

[0405] 7. Sharing customer service know-how

[0406] Server: Continuously collects and stores records of excellent employees' responses in a know-how database.

[0407] Users: When they want to learn about a specific situation or how to respond in the app, they can select a category or enter keywords.

[0408] Server: Searches for relevant information from the know-how database and provides it to the user.

[0409] Specific examples

[0410] Example 1: Handling complaints

[0411] If a customer complains that "the response to the previous product exchange was slow,"

[0412] User: Enters "product exchange" and performs a search.

[0413] Server: Searches past conversation records related to "product exchange" and provides three results to the user's device.

[0414] Users: Check past responses and use them as a reference for responding quickly.

[0415] Emotion Engine: It also analyzes customer emotions and advises users on appropriate actions before the issue escalates.

[0416] Example 2: Checking inventory

[0417] If a customer asks, "Do you have this T-shirt in a medium size?"

[0418] User: Scans the product barcode to request an inventory check.

[0419] Server: Searches the inventory database, finds that there are 5 units of the item in stock, and sends the result to the user.

[0420] User: Tell customer "In stock."

[0421] Example 3: Sharing customer service know-how

[0422] If new staff want to learn "effective upselling methods,"

[0423] User: Searches for "upsell" in the app.

[0424] Server: Extracts effective phrases and techniques related to "upselling" from a know-how database and provides them to users.

[0425] User: Review the information provided and apply it to actual customer service.

[0426] Emotion engine: Analyzes the user's emotions and suggests relaxation methods if stress is building up.

[0427] As described above, the embodiment of the present invention realizes efficient customer service and improved service quality in stores through the specific operating procedures and processing flows. In addition, by combining it with an emotion engine, it becomes possible to grasp the user's emotional state in real time and provide more appropriate service.

[0428] The processing flow will be explained below.

[0429] Specific processing flow of the system

[0430] 1. Connecting the user to the audio device

[0431] Step 1:

[0432] User: Power on the audio device.

[0433] Step 2:

[0434] Device: Turn on Bluetooth on your smartphone and launch the dedicated app.

[0435] Step 3:

[0436] On your device: The app performs a Bluetooth scan to discover nearby audio devices.

[0437] Step 4:

[0438] Device: Displays a list of detected audio devices and lets the user select a device.

[0439] Step 5:

[0440] User: Approves to pair the selected device.

[0441] Step 6:

[0442] On the device: Notify the user that pairing was successful.

[0443] Step 7:

[0444] Server: Receives the device ID from the mobile device and checks the device registration status.

[0445] Step 8:

[0446] Server: Once registration is confirmed, approve the connection and send a notification to the device that the connection is complete.

[0447] 2. Collection and transmission of voice data

[0448] Step 1:

[0449] User: Presses the start recording button in the app.

[0450] Step 2:

[0451] Device: The audio device collects the conversation and transmits the voice data to the mobile device in real time.

[0452] Step 3:

[0453] Device: Your mobile device temporarily stores the audio data.

[0454] Step 4:

[0455] Terminal: Uploads audio data to the server periodically in batches.

[0456] 3. Audio Data Processing

[0457] Step 1:

[0458] Server: Receives the audio data.

[0459] Step 2:

[0460] Server: The speech recognition engine converts the voice data into text.

[0461] Step 3:

[0462] Server: The converted text data is saved in a database.

[0463] 4. Emotional Data Processing

[0464] Step 1:

[0465] Emotion Engine: Receives voice data.

[0466] Step 2:

[0467] Emotion engine: Analyzes emotional states from voice data.

[0468] Step 3:

[0469] Emotion Engine: Notifies users of their analyzed emotional state in real time.

[0470] 5. Search for past claims data

[0471] Step 1:

[0472] User: When a complaint occurs, use the complaint search feature in the app.

[0473] Step 2:

[0474] User: Enter keywords related to the complaint in the search box.

[0475] Step 3:

[0476] Device: The app sends a search request to the server.

[0477] Step 4:

[0478] Server: Searches the database for relevant past conversation records.

[0479] Step 5:

[0480] Server: Returns search results to the user's device.

[0481] Step 6:

[0482] Device: Display search results to the user in a list format.

[0483] 6. Check inventory information

[0484] Step 1:

[0485] User: Receives a request from a customer to check inventory.

[0486] Step 2:

[0487] User: Use the in-app inventory check feature to enter the product name or barcode.

[0488] Step 3:

[0489] Terminal: Sends an inventory check request to the server.

[0490] Step 4:

[0491] Server: Retrieves product inventory information from the database.

[0492] Step 5:

[0493] Server: Returns inventory information to the user's device.

[0494] Step 6:

[0495] Terminal: Display inventory information to the user.

[0496] 7. Sharing customer service know-how

[0497] Step 1:

[0498] Server: Continuously collects and stores records of excellent employees' responses in a know-how database.

[0499] Step 2:

[0500] Users: When they want to learn about a specific situation or how to respond in the app, they can select a category or enter keywords.

[0501] Step 3:

[0502] Terminal: Sends a know-how search request to the server.

[0503] Step 4:

[0504] Server: Searches for relevant information from a know-how database.

[0505] Step 5:

[0506] Server: Sends search results to the user's device.

[0507] Step 6:

[0508] Terminal: Displays the provided know-how information to the user.

[0509] With the above processing flow, the system of the present invention can improve the efficiency of customer service, handle complaints, check inventory, and share customer service know-how. In addition, by combining it with an emotion engine, it is possible to grasp the user's emotional state in real time and respond more appropriately.

[0510] Example 2

[0511] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0512] Conventional customer service systems have difficulty efficiently recording conversations between customers and users (store staff) and processing them in real time. Furthermore, systems that can simultaneously support multiple tasks, such as handling complaints, checking inventory, and sharing customer service know-how, are limited. Furthermore, they lack the functionality to instantly grasp the emotional state of customers and optimize responses, resulting in problems such as stress for users and a decline in the quality of customer service. An effective system that can solve these issues is needed.

[0513] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0514] In this invention, the server includes means for connecting an audio device to a user's mobile terminal, means for recording conversations between a user and a customer via the audio device, means for transmitting the recorded voice data to an information processing device in real time, means for converting the voice data into text data in the information processing device, means for saving the converted text data in a storage device, means for searching for related past data from the storage device, means for providing the search results to the user, means for analyzing the emotional state of the customer from the voice data and notifying the user in real time, means for acquiring and providing inventory information in response to a user's request, and means for collecting and sharing records of responses from excellent employees. This makes it possible to improve the efficiency of customer service, speed up complaint responses, improve the accuracy of inventory checks, share customer service know-how, and also to grasp the emotional state of customers in real time and respond appropriately.

[0515] An "audio device" is a device that collects audio and transmits that data to an external device.

[0516] "Users" refer to store staff and service personnel who use this system to provide customer service.

[0517] "Mobile terminal" refers to a small, portable information processing device such as a smartphone or tablet.

[0518] "Customer" refers to a consumer who uses a store or service.

[0519] The term "mobile device" is synonymous with a portable terminal and refers to an information processing device that is carried and used by a user.

[0520] "Information processing device" refers to a computer system for processing voice data and text data.

[0521] "Recording means" refers to devices or software capable of collecting and storing audio data.

[0522] "Voice data" refers to data that represents voice information, including the content of conversations between users and customers, in digital form.

[0523] "Text data" refers to digital data that has been converted from voice data into text information.

[0524] "Storage device" refers to hardware and software for storing data for long periods of time.

[0525] "Means for storing data in a storage device" refers to the functions and processes for storing data in a storage device.

[0526] "Means for searching past data" refers to the function or process of searching for information that matches specific conditions from data stored in a storage device.

[0527] "Means of providing search results to users" refers to the function of displaying or providing the information obtained through a search in a format that is easy for users to view.

[0528] "Means for analyzing emotional state and notifying in real time" refers to the functions and processes for determining emotions from voice data and immediately conveying that information to the user.

[0529] "Means for obtaining and providing inventory information" refers to the function of retrieving inventory information from a database or other storage device and providing it to the user.

[0530] "Means for collecting and sharing records of excellent employee interactions" refers to the functions and processes for collecting records of excellent employee interactions with customers and sharing them with other employees.

[0531] As an embodiment of the present invention, a system is constructed that combines an audio device, a mobile device, a server, and an emotion engine to improve the efficiency of customer service in stores, handle complaints, check inventory, and share customer service know-how. The system aims to collect, analyze, save, and provide conversations between customers and store staff in real time.

[0532] Basic system configuration

[0533] User (store staff)

[0534] Users can use the system to record conversations with customers, manage complaints, check inventory, share customer service know-how, and use an emotion engine to understand the emotional state of their customers and themselves in real time.

[0535] Audio Devices

[0536] The audio device is a user-worn device that collects conversations with customers and connects to a mobile device via Bluetooth.

[0537] Mobile device (smartphone or tablet)

[0538] The mobile device receives voice data from the audio device and sends it to the server and emotion engine, and some voice data is temporarily stored.

[0539] server

[0540] The server converts the received voice data into text and stores it in a database. It also manages inventory information and provides information upon user request. It also has the capacity to collect and share records of excellent employees' interactions.

[0541] Emotion Engine

[0542] The emotion engine analyzes customer and user emotions from voice data and determines their emotional state in real time.

[0543] Specific examples

[0544] Example 1: Handling complaints

[0545] If a customer complains that "the response to the previous product exchange was slow," the system operates as follows:

[0546] User: Enters "product exchange" into the app and performs a search.

[0547] Server: Searches the database for conversation records related to past "product exchanges" and provides the results to the user's mobile device.

[0548] Users: Check past responses and use them as a reference for responding quickly.

[0549] Emotion Engine: Analyzes customer emotions and advises users on appropriate actions before the issue escalates.

[0550] Example 2: Checking inventory

[0551] When a customer asks, "Do you have this T-shirt in size medium?", the system works as follows:

[0552] User: Scans the product barcode to request an inventory check.

[0553] Server: Searches the inventory database, finds that there are 5 units of the item in stock, and sends the result to the user.

[0554] User: Tell customer "In stock."

[0555] Example 3: Sharing customer service know-how

[0556] If a new staff member wants to learn "effective upselling methods," the system works as follows:

[0557] User: Searches for "upsell" in the app.

[0558] Server: Extracts effective phrases and techniques related to "upselling" from a know-how database and provides them to users.

[0559] User: Review the information provided and apply it to actual customer service.

[0560] Emotion engine: Analyzes the user's emotions and suggests relaxation methods if stress is building up.

[0561] In this way, the present invention achieves more efficient customer service and improved service quality in stores through specific operational procedures and processing flows. In addition, by combining it with an emotion engine, it is possible to grasp the user's emotional state in real time and provide more appropriate responses.

[0562] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0563] Step 1:

[0564] Connecting users and audio devices

[0565] Input: The user turns on the audio device and launches a dedicated app on their mobile device.

[0566] Specific operation: The user operates the Bluetooth settings of the mobile device to pair with the audio device. The dedicated app on the mobile device obtains the audio device ID and sends it to the server.

[0567] Data processing: The server checks the received device ID in the database to determine whether it is a registered device. If successful, the server returns a connection acknowledgement message.

[0568] Output: A "Connection successful" message will appear on the mobile device screen.

[0569] Step 2:

[0570] Collection and transmission of voice data

[0571] Input: The user initiates a conversation and the audio device collects the audio.

[0572] What it does: The audio device collects the conversation with the customer and transmits it in real time to the mobile device, which temporarily stores the audio data.

[0573] Data processing: Once a certain amount of voice data has been accumulated, it is transferred to the server and emotion engine in a batch process.

[0574] Output: The server and emotion engine receive the audio data.

[0575] Step 3:

[0576] Audio data processing

[0577] Input: Audio data sent from a mobile device.

[0578] Specific operation: The server converts the voice data into text data using a speech recognition engine such as the Google Cloud Speech-to-Text API.

[0579] Data processing: The converted text data is added with time series and related information (customer ID, date and time, etc.) and stored in a database.

[0580] Output: The saved text data.

[0581] Step 4:

[0582] Emotional Data Processing

[0583] Input: Audio data sent from a mobile device.

[0584] What it does: The emotion engine analyzes voice data to identify the emotional state of customers and users.

[0585] Data processing: The emotion engine determines emotions from the tone, speed, and content of the voice, and analyzes the results in real time.

[0586] Output: Notifications about the emotional state of the customer and user are displayed in a dedicated app on the mobile device.

[0587] Step 5:

[0588] Handling complaints

[0589] Input: User searches for keywords related to the claim in the app.

[0590] How it works: A user enters a keyword, such as "product exchange," into the app, and the server searches the database to extract relevant past complaint data.

[0591] Data processing: The data extracted as search results is sorted in chronological order and provided to the user along with related information.

[0592] Output: The app displays the past complaint handling records.

[0593] Step 6:

[0594] Check stock information

[0595] Input: The user enters the product name or barcode into a dedicated app.

[0596] Specific operation: The user scans an item with a barcode reader or enters the item name. The server searches the inventory database to obtain the stock information for that item.

[0597] Data processing: Inventory information is sent to the user's device in real time.

[0598] Output: Inventory information is displayed on the user's mobile device, and the user communicates stock availability to the customer.

[0599] Step 7:

[0600] Sharing customer service know-how

[0601] Input: A user searches the app for keywords related to a specific interaction.

[0602] Specific operation: The user enters a keyword such as "upsell" into the app. The server searches the know-how database and extracts records of related customer service know-how.

[0603] Data processing: The extracted know-how information is organized by category and provided to users.

[0604] Output: Know-how information is displayed on the app, and users can apply it to actual customer service.

[0605] As described above, this system collects, processes, and provides data according to user requests through each processing step, thereby improving the efficiency of customer service and service quality.

[0606] (Application example 2)

[0607] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0608] Real-time information collection and analysis is essential to improve the efficiency of customer service in stores, the speed of complaint handling, the speed of inventory checks, and the customer service skills of employees. However, conventional systems lack the means to comprehensively solve these issues, making it particularly difficult to incorporate emotion analysis into customer service, efficiently check inventory, and instantly share customer service know-how.

[0609] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for connecting an audio device to the user's mobile device, means for recording conversations between the user and customers via the audio device, means for transmitting the recorded voice data to the server in real time, means for converting the voice data into text data in the server, means for storing the converted text data in a database, means for searching for related past data from the database, means for providing the search results to the user, means for having the user wear smart glasses and collecting the conversation and emotional state through the glasses, means for analyzing the emotional state from the voice data using an emotion engine and notifying the user in real time, and means for displaying inventory information, complaint data, and customer service know-how on the smart glasses. This enables efficient customer service, appropriate responses based on emotion analysis, rapid inventory confirmation, and real-time sharing of customer service know-how.

[0610] An "audio device" is a device that records conversations between users and customers and transmits them as audio data to a mobile device.

[0611] "Mobile Device" means a portable electronic device that receives recorded audio data and transmits the data to a server or emotion engine.

[0612] A "server" is a central processing unit that converts voice data into text data and stores it in a database.

[0613] An "emotion engine" is an analysis device that analyzes emotions from voice data and determines emotional states in real time.

[0614] "Smart glasses" are devices worn by users that collect speech and emotional states and display information.

[0615] A "database" is a collection of information in which voice data is converted into text data and stored.

[0616] "Inventory information" is data that indicates the quantity and condition of a particular product currently in stock in the store.

[0617] "Complaint data" refers to recorded data regarding past complaints and requests from customers.

[0618] "Customer service know-how" is knowledge that allows employees to share records and techniques of excellent customer service.

[0619] "Real-time" refers to data and information being processed immediately, without delay.

[0620] A "search tool" is a method or device for finding relevant information within a database.

[0621] "Notification means" refers to a method or device for notifying the user of information such as the analyzed emotional state.

[0622] A "display" is a method or apparatus for visually presenting information to smart glasses or other devices.

[0623] This invention is a system for improving the efficiency of customer service in stores, handling complaints quickly, checking inventory, and sharing customer service know-how. This system combines audio devices, smart glasses, mobile devices, a server, and an emotion engine.

[0624] Basic system configuration

[0625] User (store staff):

[0626] The user wears smart glasses and uses a mobile device to collect conversations with customers, handle complaints, check inventory, and learn customer service know-how. The emotion engine allows the user to understand the emotional state of both the customer and the user in real time.

[0627] Audio Device:

[0628] A microphone built into smart glasses that collects customer conversations and transmits them to a mobile device.

[0629] Smart Glasses:

[0630] This device collects conversations with customers and their emotional states and displays the information, displaying customer emotion analysis results, inventory information, complaint handling data, and customer service know-how in real time.

[0631] Mobile devices:

[0632] It works in conjunction with smart glasses to send voice data to a server in real time, and also receives converted text data, emotion analysis results, and past data, and sends them to the smart glasses.

[0633] server:

[0634] The system converts voice data into text data and stores it in a database. It also uses an emotion engine to analyze emotions from the voice data and notify users in real time. It also manages complaint histories and inventory information.

[0635] Emotion Engine:

[0636] It analyzes emotions from voice data and determines the user's emotional state in real time. This information is sent to the smart glasses, which then advise the user on how to respond appropriately.

[0637] Hardware and software used

[0638] Hardware:

[0639] Smart glasses (e.g., Google Glass®): A device for collecting speech and emotional state and displaying information.

[0640] Mobile devices (e.g. smartphones and tablets): process the voice data and send it to the server.

[0641] software:

[0642] Speech recognition engine (e.g. Google Speech-to-Text API): Converts voice data into text.

[0643] Sentiment analysis engine (e.g., IBM Watson Tone Analyzer): Analyzes customer and user sentiment.

[0644] Database (e.g. MySQL): Manages claims, inventory, and know-how.

[0645] System operation example

[0646] 1. Stock Check Example:

[0647] A customer asks, "Do you have this shoe in size 27.5cm?"

[0648] Users use smart glasses to scan product names or barcodes.

[0649] The server searches the inventory database.

[0650] The smart glasses will display "in stock" and the user can then direct it to the customer.

[0651] Example prompt sentence:

[0652] "Check if this item is in stock"

[0653] Customer: "Do you have these shoes in size 27.5cm?"

[0654] User: Scan product name or barcode with smart glasses

[0655] System: "Checking inventory..."

[0656] System: Shows "In stock"

[0657] 2. Examples of complaint handling:

[0658] The customer complains that the previous replacement was too slow.

[0659] The user types "exchange" into the smart glasses and searches.

[0660] The server searches past "exchange" claim data.

[0661] Past response results are displayed on the smart glasses, allowing users to respond quickly based on them.

[0662] Example prompt sentence:

[0663] "Check past complaint handling"

[0664] Customer: "The last exchange was slow."

[0665] User: Searches for "exchange" on smart glasses

[0666] System: Display "View past complaint responses"

[0667] In this way, the system will enable more efficient customer service, appropriate responses based on sentiment analysis, quick inventory checks, and real-time sharing of customer service know-how.

[0668] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0669] Step 1:

[0670] The user wears the smart glasses and connects the audio device to the mobile device. Specifically, the user turns on the smart glasses, launches the app on the mobile device, and pairs with the audio device via Bluetooth. The input is the user's operation, and the output is the connection status between the devices.

[0671] Step 2:

[0672] When a user starts a customer interaction, the audio device collects the voice. The input is the conversation between the customer and the user, and the output is the collected voice data. Specifically, the microphone of the audio device records the conversation and transmits it to the mobile device in real time.

[0673] Step 3:

[0674] The mobile device temporarily stores the voice data and transfers it to the server and emotion engine in batch processing. Specifically, once a certain amount of voice data has been collected, it is automatically sent to the server and emotion engine. The input is the collected voice data, and the output is the transmitted voice data.

[0675] Step 4:

[0676] The server receives the voice data and converts it into text data using a speech recognition engine. Specifically, speech recognition software such as the Google Speech-to-Text API is used. The input is voice data and the output is text data.

[0677] Step 5:

[0678] The server stores the text data in a database. Specifically, database software such as MySQL is used to store the data in an appropriate format. The input is text data, and the output is the stored text data.

[0679] Step 6:

[0680] The emotion engine analyzes the emotional state from the voice data and notifies the user in real time. Specifically, emotion analysis software such as IBM Watson Tone Analyzer is used. The input is the voice data, and the output is the analysis result of the emotional state.

[0681] Step 7:

[0682] Users can search for past complaint data or inventory information as needed. Specifically, they enter specific keywords into a mobile device app and execute a search. The input is the search keyword, and the output is the search results for related data.

[0683] Step 8:

[0684] The server searches the database for relevant historical data and provides it to the user. Specifically, it searches the database using an SQL query and sends the results to the mobile device. The input is the search query, and the output is the search result data.

[0685] Step 9:

[0686] The server retrieves inventory information in real time and displays it on the smart glasses. Specifically, it retrieves the latest information from the inventory database and displays the results on the user's smart glasses. The input is an inventory check request, and the output is inventory information.

[0687] Step 10:

[0688] The server collects the customer service records of outstanding employees and stores their customer service know-how in a database. Specifically, it analyzes daily customer service data, extracts effective customer service techniques, and adds them to the database. The input is customer service data, and the output is the analyzed customer service know-how.

[0689] Step 11:

[0690] The user checks customer service know-how using the smart glasses and applies it to actual customer service. Specifically, the user queries know-how for a specific situation and responds based on the information displayed on the smart glasses. The input is a know-how query request, and the output is the displayed know-how information.

[0691] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0692] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0693] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0694] [Second embodiment]

[0695] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0696] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0697] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0698] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0699] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0700] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0701] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0702] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0703] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0704] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0705] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0706] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0707] As an embodiment of the present invention, a system is constructed in which an audio device, a mobile device, and a server are linked together in order to improve the efficiency of customer service in a store.

[0708] 1. Basic system configuration

[0709] User (store staff): Uses this system to record conversations with customers, manage complaints, check inventory, and share customer service know-how.

[0710] Audio device: A device worn by the user that captures conversations with customers.

[0711] Mobile device (smartphone or tablet): Responsible for receiving audio data from the audio device and sending it to the server.

[0712] Server: Converts received voice data into text and stores it in a database. It also manages inventory information and provides information upon user request. It also collects and shares records of outstanding employees' interactions.

[0713] 2. System Operation Flow

[0714] 1. Connecting the user to the audio device

[0715] Device: The user powers on the audio device and launches the app on their mobile device, which pairs with the audio device via Bluetooth.

[0716] Server: If pairing is successful, the server receives the device ID, checks the device registration status, and approves the connection.

[0717] 2. Collection and transmission of voice data

[0718] Device: When a user initiates a customer interaction, the audio device collects the voice and transmits the collected voice data to the mobile device in real time.

[0719] Device: The mobile device temporarily stores the collected voice data and transfers it to the server in batches.

[0720] 3. Audio Data Processing

[0721] Server: Receives voice data and converts it into text data using a speech recognition engine.

[0722] Server: The converted text data is saved in a database.

[0723] 4. Search for past claims data

[0724] Users: Use the search function in the app when handling a complaint by entering keywords related to previous complaints.

[0725] Server: Searches for relevant historical data in the database and provides it to the user.

[0726] 5. Check inventory information

[0727] User: When a customer requests an inventory check, they enter the product name and barcode through the app.

[0728] Server: Retrieves inventory information from the database and provides it to users in real time.

[0729] Terminal: Displays inventory information to the user and communicates it to the customer.

[0730] 6. Sharing customer service know-how

[0731] Server: Continuously collects and stores records of excellent employees' responses in a know-how database.

[0732] Users: When they want to learn about a specific situation or how to respond in the app, they can select a category or enter keywords.

[0733] Server: Searches for relevant information from the know-how database and provides it to the user.

[0734] Examples:

[0735] Example 1: Handling complaints

[0736] If a customer complains that "the response to the previous product exchange was slow,"

[0737] User: Enters "product exchange" and performs a search.

[0738] Server: Searches past conversation records related to "product exchange" and provides three results to the user's device.

[0739] Users: Check past responses and use them as a reference for responding quickly.

[0740] Example 2: Checking inventory

[0741] If a customer asks, "Do you have this T-shirt in a medium size?"

[0742] User: Scans the product barcode to request an inventory check.

[0743] Server: Searches the inventory database, finds that there are 5 units of the item in stock, and sends the result to the user.

[0744] User: Tell customer "In stock."

[0745] Example 3: Sharing customer service know-how

[0746] If new staff want to learn "effective upselling methods,"

[0747] User: Searches for "upsell" in the app.

[0748] Server: Extracts effective phrases and techniques related to "upselling" from a know-how database and provides them to users.

[0749] User: Review the information provided and apply it to actual customer service.

[0750] As described above, the embodiment of the present invention realizes efficient customer service and improved service quality in stores through the specific operation procedures and processing flows.

[0751] The processing flow will be explained below.

[0752] Specific processing flow of the system

[0753] 1. Connecting the user to the audio device

[0754] Step 1:

[0755] User: Power on the audio device.

[0756] Step 2:

[0757] Device: Turn on Bluetooth on your smartphone and launch the dedicated app.

[0758] Step 3:

[0759] On your device: The app performs a Bluetooth scan to discover nearby audio devices.

[0760] Step 4:

[0761] Device: Displays a list of detected audio devices and lets the user select a device.

[0762] Step 5:

[0763] User: Approves to pair the selected device.

[0764] Step 6:

[0765] On the device: Notify the user that pairing was successful.

[0766] Step 7:

[0767] Server: Receives the device ID from the mobile device and checks the device registration status.

[0768] Step 8:

[0769] Server: Once registration is confirmed, approve the connection and send a notification to the device that the connection is complete.

[0770] 2. Collection and transmission of voice data

[0771] Step 1:

[0772] User: Presses the start recording button in the app.

[0773] Step 2:

[0774] Device: The audio device collects the conversation and transmits the voice data to the mobile device in real time.

[0775] Step 3:

[0776] Device: Your mobile device temporarily stores the audio data.

[0777] Step 4:

[0778] Terminal: Uploads audio data to the server periodically in batches.

[0779] 3. Audio Data Processing

[0780] Step 1:

[0781] Server: Receives the audio data.

[0782] Step 2:

[0783] Server: The speech recognition engine converts the voice data into text.

[0784] Step 3:

[0785] Server: The converted text data is saved in a database.

[0786] 4. Search for past claims data

[0787] Step 1:

[0788] User: When a complaint occurs, use the complaint search feature in the app.

[0789] Step 2:

[0790] User: Enter keywords related to the complaint in the search box.

[0791] Step 3:

[0792] Device: The app sends a search request to the server.

[0793] Step 4:

[0794] Server: Searches the database for relevant past conversation records.

[0795] Step 5:

[0796] Server: Returns search results to the user's device.

[0797] Step 6:

[0798] Device: Display search results to the user in a list format.

[0799] 5. Check inventory information

[0800] Step 1:

[0801] User: Receives a request from a customer to check inventory.

[0802] Step 2:

[0803] User: Use the in-app inventory check feature to enter the product name or barcode.

[0804] Step 3:

[0805] Terminal: Sends an inventory check request to the server.

[0806] Step 4:

[0807] Server: Retrieves product inventory information from the database.

[0808] Step 5:

[0809] Server: Returns inventory information to the user's device.

[0810] Step 6:

[0811] Terminal: Display inventory information to the user.

[0812] 6. Sharing customer service know-how

[0813] Step 1:

[0814] Server: Continuously collects and stores records of excellent employees' responses in a know-how database.

[0815] Step 2:

[0816] Users: When they want to learn about a specific situation or how to respond in the app, they can select a category or enter keywords.

[0817] Step 3:

[0818] Terminal: Sends a know-how search request to the server.

[0819] Step 4:

[0820] Server: Searches for relevant information from a know-how database.

[0821] Step 5:

[0822] Server: Sends search results to the user's device.

[0823] Step 6:

[0824] Terminal: Displays the provided know-how information to the user.

[0825] With the above processing flow, the system of the present invention can improve the efficiency of customer service for users, and can realize the sharing of know-how on handling complaints, checking inventory, and customer service.

[0826] Example 1

[0827] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0828] Conventional customer service systems have had issues with efficiently collecting and managing conversation records, quickly obtaining inventory information, and sharing excellent customer service know-how. In particular, there is a demand for systems that can efficiently process customer conversation data in real time, quickly search for past complaint handling records, and provide inventory information in a timely manner.

[0829] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0830] In this invention, the server includes means for collecting conversations via an audio device connected to the mobile terminal, means for temporarily storing the collected voice data and transmitting it to the server, and means for converting the voice data into text data and storing it in a database, thereby enabling real-time processing of conversation data with customers, rapid search of past complaint handling records, and timely provision of inventory information.

[0831] An "audio device" is an audio device that allows a user to collect conversations with customers.

[0832] "Mobile device" refers to the mobile device used by the user, such as a smartphone or tablet.

[0833] "Means of collection" refers to the function of using an audio device to record conversations between customers and users and capture them as data.

[0834] "Temporarily stored" means that the collected voice data is temporarily stored on the mobile device until it is batch processed.

[0835] "Batch processing" refers to the process of sending data to a server in bulk based on a set time or data size.

[0836] "Server" means a central processing unit for processing, storing, and providing information to users.

[0837] "Means for converting into text data" refers to the process of converting voice data into text information using a voice recognition engine.

[0838] A "database" is a data storage system for systematically storing and managing converted text data and related information.

[0839] "Searching means" refers to the functionality for locating specific information within a database.

[0840] "Means for providing search results" refers to the function of displaying the information obtained through a search on the user's mobile device.

[0841] "Means of obtaining inventory information in real time" refers to the process of instantly obtaining the latest inventory information from the database and providing it to the user.

[0842] "Excellent employee service records" refers to records of customer service provided by highly rated employees.

[0843] "Means for sharing" refers to the ability to share collected records of excellent employee responses with other users for learning and reference.

[0844] This invention is a system aimed at improving the efficiency of customer service in stores. By linking audio devices, mobile terminals, and servers, this system collects and converts conversation data with customers into text, and provides the functionality to store and search the data in a database.

[0845] Basic system configuration

[0846] User (store staff): Records conversations with customers and uses them to manage complaints, check inventory, and share customer service know-how.

[0847] Audio Device: A device for collecting audio. The user wears this device and records the conversation with the customer.

[0848] Mobile device (smartphone or tablet): A device that receives audio data from an audio device and sends it to a server.

[0849] Server: Converts voice data into text data and stores it in a database. It also provides users with related past data, inventory information, and customer service know-how.

[0850] Overall system operation

[0851] The user turns on the audio device and launches the dedicated app on the mobile device. The audio device connects to the mobile device via Bluetooth. The server receives the device ID, checks the registration status, and then approves the connection. When the user starts talking with a customer, the audio device collects voice data and transmits it to the mobile device in real time. The mobile device temporarily stores the collected voice data and transmits it to the server in batch processing.

[0852] Audio data processing

[0853] The server converts the received voice data into text data using a speech recognition engine (such as IBM Watson or Google Cloud Speech-to-Text). This text data is then stored in a database. The stored data is searchable and can be used as a record of past complaint handling, inventory information, or records of outstanding employee responses.

[0854] Specific examples

[0855] Handling complaints

[0856] If a user receives a complaint from a customer that "the previous product exchange was handled slowly":

[0857] User: Enters "product exchange" into the app and performs a search.

[0858] Server: Searches the database and provides records related to past "commodity exchanges" to the user's device.

[0859] Users: Use search results to act quickly.

[0860] Stock Check

[0861] When a customer asks, "Do you have this item?"

[0862] User: Scans the product barcode to request an inventory check.

[0863] Server: Searches the database and provides real-time inventory status to users.

[0864] User: Tell customer "In stock."

[0865] Sharing customer service know-how

[0866] If you're a new staff member and want to learn "effective upselling methods":

[0867] User: Searches for "upsell" in the app.

[0868] Server: Provides users with relevant phrases and techniques from a know-how database.

[0869] User: Review the information provided and put it into practice when serving customers.

[0870] As a result, this system will improve the efficiency of customer service and service quality in stores.

[0871] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0872] Step 1:

[0873] Connecting your audio device to your mobile device

[0874] Input: The user turns on the audio device and launches a dedicated app on their mobile device.

[0875] Output: The audio device and mobile device are connected via Bluetooth.

[0876] Specific behavior:

[0877] User: Power on the audio device.

[0878] Device: Launch the dedicated app on your mobile device and pair it with your audio device via Bluetooth.

[0879] Server: If pairing is successful, it receives the device ID, checks the device registration status, and approves the connection.

[0880] Step 2:

[0881] User initiates customer interaction

[0882] Input: The user initiates a conversation with the customer.

[0883] Output: The conversation with the customer is collected as audio data.

[0884] Specific behavior:

[0885] User: Initiate a conversation with a customer.

[0886] Audio devices: Collect voice data from user and customer conversations.

[0887] Step 3:

[0888] Temporary storage and batch processing of audio data

[0889] Input: The collected voice data is sent to the mobile device.

[0890] Output: The audio data is temporarily stored and sent to the server in batch processing.

[0891] Specific behavior:

[0892] Audio device: Sends audio data to mobile devices in real time.

[0893] Terminal: Temporarily stores received audio data and starts batch processing when a certain time or data size is reached.

[0894] Terminal: Encrypts the audio data through batch processing and sends it to the server.

[0895] Step 4:

[0896] Converting audio data to text

[0897] Input: The server receives the audio data.

[0898] Output: The audio data is converted to text data.

[0899] Specific behavior:

[0900] Server: Converts the voice data into text data using a speech recognition engine (e.g., IBM Watson or Google Cloud Speech-to-Text).

[0901] Server: Validates the accuracy of the speech recognition results and makes corrections if necessary.

[0902] Step 5:

[0903] Text data storage in a database

[0904] Input: Text data is generated.

[0905] Output: The text data is saved in the database.

[0906] Specific behavior:

[0907] Server: The converted text data is saved in a database.

[0908] Server: Verify that the data was saved correctly and generate a success message.

[0909] Step 6:

[0910] Search past claims data

[0911] Input: The user types a keyword into the app.

[0912] Output: The relevant claim record is displayed on the user's terminal.

[0913] Specific behavior:

[0914] User: Enters a keyword, such as "exchange," into the app and performs a search.

[0915] Server: Searches the database, extracts relevant past complaint handling records, and provides them to the user.

[0916] Step 7:

[0917] Check stock information

[0918] Input: The user types a stock check request into the app.

[0919] Output: The inventory information is displayed on the user's terminal.

[0920] Specific behavior:

[0921] User: Enters product name or barcode into the app to request availability.

[0922] Server: Retrieves inventory information from the database and provides it to users in real time.

[0923] Device: Display inventory information on the user's device and communicate it to the customer.

[0924] Step 8:

[0925] Sharing customer service know-how

[0926] Input: The user types a specific keyword into the app.

[0927] Output: Related response know-how is displayed on the user's terminal.

[0928] Specific behavior:

[0929] User: Enters a keyword into the app, such as "upsell," and performs a search.

[0930] Server: Extracts relevant phrases and techniques from the know-how database and provides them to users.

[0931] User: Review the information provided and put it into practice in customer service.

[0932] (Application example 1)

[0933] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0934] Conventional factory management systems rely on manual processes for managing work orders, checking inventory, handling complaints, and sharing work know-how, which can be inefficient and prone to errors. Performing these tasks efficiently is particularly difficult in large factories. This has led to a demand for automating various tasks within factories, processing data in real time, and improving work efficiency.

[0935] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0936] In this invention, the server includes means for connecting an audio device to a user's mobile device, means for recording conversations between the user and customers via the audio device, means for transmitting the recorded audio data to the server in real time, means for converting the audio data to text data in the server, means for storing the converted text data in a database, means for searching related past data from the database, means for providing the search results to the user, means for inputting and managing work instructions by voice, means for checking inventory information in real time, means for recording and searching complaint responses, and means for classifying and sharing excellent work records, which enables improved work efficiency in factories, more accurate inventory management, faster complaint responses, and the sharing of work know-how.

[0937] "Audio Device" means a device that collects audio and is connected to a User's Mobile Device.

[0938] A "mobile device" is a portable information terminal such as a smartphone or tablet.

[0939] A "recording means" is a method or apparatus for storing audio data collected via an audio device.

[0940] "Means for transmitting in real time" refers to a method or device for instantly transmitting collected voice data to a server.

[0941] A "server" is a central processing unit that processes and stores information on a network.

[0942] The "means for converting into text data" refers to a method or device that has the function of converting voice data into text information.

[0943] A "database" is a system for centrally storing and managing information.

[0944] A "searching means" is a method or device for efficiently locating specific information within a database.

[0945] "Means of providing" refers to the method or device for displaying search results and data to users.

[0946] "Means for managing work instructions" refers to a method or device for organizing and saving work instructions entered by voice so that they can be referenced as needed.

[0947] "Means for checking inventory information" refers to a method or device that allows you to grasp the inventory status of a product in real time.

[0948] "Means for recording complaint handling" refers to a method or device for recording and saving customer complaints and requests as audio data.

[0949] "Means for classifying and sharing excellent work records" refers to methods and devices for collecting effective work methods and know-how, organizing and storing them, and sharing them with other users.

[0950] As an embodiment of the present invention, a method for constructing a factory management system will be described below.

[0951] 1. Basic system configuration

[0952] Users (factory workers): Use this system to input and manage work instructions by voice, check inventory, handle complaints, and share excellent work know-how.

[0953] Audio device: A device worn by the user that collects audio such as instructions and complaints within the factory.

[0954] Mobile device (smartphone or tablet): Temporarily stores audio data received from the audio device and sends it to the server.

[0955] Server: Converts received voice data into text and stores it in a database. It also manages inventory information and provides information upon user request. It also collects and shares excellent work records.

[0956] 2. System Operation Flow

[0957] 1. Connecting the user to the audio device

[0958] Device: The user powers on the audio device and launches the app on their mobile device, which pairs with the audio device via Bluetooth.

[0959] Server: If pairing is successful, the server receives the device ID, checks the device registration status, and approves the connection.

[0960] 2. Collection and transmission of voice data

[0961] Device: When the user begins to speak work instructions, the audio device collects the voice and transmits the collected voice data to the mobile device in real time.

[0962] Device: The mobile device temporarily stores the collected voice data and transfers it to the server in batches.

[0963] 3. Audio Data Processing

[0964] Server: Receives the voice data and converts it into text data using a speech recognition engine (e.g., Google Cloud Speech-to-Text).

[0965] Server: The converted text data is stored in a database (e.g. MySQL).

[0966] 4. Search for past claims data

[0967] Users: Use the search function in the app when handling a complaint by entering keywords related to previous complaints.

[0968] Server: Searches for relevant historical data in the database and provides it to the user.

[0969] 5. Check inventory information

[0970] User: Enter the product name or barcode by voice.

[0971] Server: Retrieves inventory information from the database and provides it to users in real time.

[0972] Terminal: Displays inventory information to the user and communicates it to the customer.

[0973] 6. Share good work records

[0974] Server: Continuously collects voice data of excellent work and stores it in a know-how database.

[0975] Users: When they want to learn about a specific task or know-how in the app, they select a category or enter keywords.

[0976] Server: Searches for relevant information from the know-how database and provides it to the user.

[0977] Specific examples

[0978] Example 1: Entering and managing work orders

[0979] User: Speak instructions such as "Assemble 3 units of part A."

[0980] Server: Converts the received voice data into text and adds the instruction "Assemble 3 units of part A" to the work list.

[0981] Example 2: Checking inventory

[0982] User: Asks aloud, "What is the inventory of part A, barcode 1234?"

[0983] Server: Searches the inventory database and sends the result, "There are 5 units of part A with barcode 1234 in stock."

[0984] Example 3: Recording complaints

[0985] User: Speaks a complaint: "Machine X is not working properly."

[0986] Server: Converts the received voice data into text and stores it in the claim database.

[0987] User: Search for past complaint history regarding "Machine X".

[0988] Prompt Sentence Examples

[0989] (Voice input): "What is the current inventory?"

[0990] (Generated text): "Checking current inventory. Please wait."

[0991] In this way, the efficiency of work and the quality of service in the factory management system are improved.

[0992] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0993] Step 1:

[0994] Connecting users and audio devices

[0995] User: Power on the audio device and launch the app on their smartphone.

[0996] Input: User interaction to open the audio device and launch the app.

[0997] Output: The audio device and smartphone are paired.

[0998] Device: The app pairs with your audio device via Bluetooth.

[0999] Server: If pairing is successful, the server receives the device ID, checks the device's registration status, and approves the connection.

[1000] Step 2:

[1001] Collection and transmission of voice data

[1002] User: Start a conversation with customers or other workers.

[1003] Input: The user's voice conversation.

[1004] Output: Audio device collects audio data.

[1005] Device: The audio device sends audio data to the smartphone in real time.

[1006] Input: Audio data from the audio device.

[1007] Output: Audio data temporarily stored on your smartphone.

[1008] Terminal: The temporarily saved audio data is transferred to the server in batch processing.

[1009] Step 3:

[1010] Audio data processing

[1011] Server: Receives the voice data and analyzes it using a voice recognition engine.

[1012] Input: Audio data sent from the smartphone.

[1013] Output: Parsed text data.

[1014] Server: Converts voice data into text data using a speech recognition engine (e.g., Google Cloud Speech-to-Text).

[1015] Input: Speech data to the speech recognition engine.

[1016] Output: The content converted to text data.

[1017] Server: The converted text data is stored in a database (e.g. MySQL).

[1018] Step 4:

[1019] Search past claims data

[1020] User: When handling a complaint, use the search function in the app and enter keywords.

[1021] Input: Claim response keywords.

[1022] Output: Search results for historical claims data.

[1023] Server: Searches for relevant historical data in a database.

[1024] Input: A query search to the database.

[1025] Output: Claim data as search results.

[1026] Server: Serves search results to users.

[1027] Step 5:

[1028] Check stock information

[1029] User: Enter the product name or barcode by voice.

[1030] Input: Product name or barcode.

[1031] Output: Inventory information retrieved from the database.

[1032] Server: Retrieves inventory information from the database and provides it to users in real time.

[1033] Input: Inventory information from the database.

[1034] Output: Inventory information provided to the user.

[1035] Terminal: Displays inventory information to the user and communicates it to the customer.

[1036] Step 6:

[1037] Sharing excellent work records

[1038] Server: Continuously collects voice data of excellent work and stores it in a know-how database.

[1039] Input: Excellent work audio data.

[1040] Output: Text data stored in the know-how database.

[1041] Users: When they want to learn about a specific task or know-how in the app, they select a category or enter keywords.

[1042] Input: Keywords related to know-how.

[1043] Output: Relevant know-how information.

[1044] Server: Searches for relevant information from the know-how database and provides it to the user.

[1045] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1046] As a form for implementing the present invention, a system is constructed that combines an audio device, a mobile device, a server, and an emotion engine to improve the efficiency of customer service in stores, handle complaints, check inventory, and share customer service know-how.

[1047] 1. Basic system configuration

[1048] User (store staff): This system records conversations with customers, manages complaints, checks inventory, and shares customer service know-how. It also uses an emotion engine to understand emotional states in real time.

[1049] Audio device: A device worn by the user that captures conversations with customers.

[1050] Mobile device (smartphone or tablet): Responsible for receiving voice data from the audio device and sending it to the server and emotion engine.

[1051] Server: Converts received voice data into text and stores it in a database. It also manages inventory information and provides information upon user request. It also collects and shares records of outstanding employees' interactions.

[1052] Emotion Engine: Analyzes emotions from user and customer voice data to determine their emotional state in real time.

[1053] 2. System Operation Flow

[1054] 1. Connecting the user to the audio device

[1055] Device: The user powers on the audio device and launches the app on their mobile device, which pairs with the audio device via Bluetooth.

[1056] Server: If pairing is successful, the server receives the device ID, checks the device registration status, and approves the connection.

[1057] 2. Collection and transmission of voice data

[1058] Device: When a user initiates a customer interaction, the audio device collects the voice and transmits the collected voice data to the mobile device in real time.

[1059] Terminal: The mobile device temporarily stores the voice data and transfers it to the server and emotion engine in batch processing.

[1060] 3. Audio Data Processing

[1061] Server: Receives the voice data and converts it into text using a speech recognition engine.

[1062] Server: The converted text data is saved in a database.

[1063] 4. Emotional Data Processing

[1064] Emotion Engine: Analyzes emotional state from voice data and notifies the user in real time.

[1065] 5. Search for past claims data

[1066] Users: Use the search function in the app when handling a complaint by entering keywords related to previous complaints.

[1067] Server: Searches for relevant historical data in the database and provides it to the user.

[1068] 6. Check inventory information

[1069] User: When a customer requests an inventory check, they enter the product name and barcode through the app.

[1070] Server: Retrieves inventory information from the database and provides it to users in real time.

[1071] Terminal: Displays inventory information to the user and communicates it to the customer.

[1072] 7. Sharing customer service know-how

[1073] Server: Continuously collects and stores records of excellent employees' responses in a know-how database.

[1074] Users: When they want to learn about a specific situation or how to respond in the app, they can select a category or enter keywords.

[1075] Server: Searches for relevant information from the know-how database and provides it to the user.

[1076] Specific examples

[1077] Example 1: Handling complaints

[1078] If a customer complains that "the response to the previous product exchange was slow,"

[1079] User: Enters "product exchange" and performs a search.

[1080] Server: Searches past conversation records related to "product exchange" and provides three results to the user's device.

[1081] Users: Check past responses and use them as a reference for responding quickly.

[1082] Emotion Engine: It also analyzes customer emotions and advises users on appropriate actions before the issue escalates.

[1083] Example 2: Checking inventory

[1084] If a customer asks, "Do you have this T-shirt in a medium size?"

[1085] User: Scans the product barcode to request an inventory check.

[1086] Server: Searches the inventory database, finds that there are 5 units of the item in stock, and sends the result to the user.

[1087] User: Tell customer "In stock."

[1088] Example 3: Sharing customer service know-how

[1089] If new staff want to learn "effective upselling methods,"

[1090] User: Searches for "upsell" in the app.

[1091] Server: Extracts effective phrases and techniques related to "upselling" from a know-how database and provides them to users.

[1092] User: Review the information provided and apply it to actual customer service.

[1093] Emotion engine: Analyzes the user's emotions and suggests relaxation methods if stress is building up.

[1094] As described above, the embodiment of the present invention realizes efficient customer service and improved service quality in stores through the specific operating procedures and processing flows. In addition, by combining it with an emotion engine, it becomes possible to grasp the user's emotional state in real time and provide more appropriate service.

[1095] The processing flow will be explained below.

[1096] Specific processing flow of the system

[1097] 1. Connecting the user to the audio device

[1098] Step 1:

[1099] User: Power on the audio device.

[1100] Step 2:

[1101] Device: Turn on Bluetooth on your smartphone and launch the dedicated app.

[1102] Step 3:

[1103] On your device: The app performs a Bluetooth scan to discover nearby audio devices.

[1104] Step 4:

[1105] Device: Displays a list of detected audio devices and lets the user select a device.

[1106] Step 5:

[1107] User: Approves to pair the selected device.

[1108] Step 6:

[1109] On the device: Notify the user that pairing was successful.

[1110] Step 7:

[1111] Server: Receives the device ID from the mobile device and checks the device registration status.

[1112] Step 8:

[1113] Server: Once registration is confirmed, approve the connection and send a notification to the device that the connection is complete.

[1114] 2. Collection and transmission of voice data

[1115] Step 1:

[1116] User: Presses the start recording button in the app.

[1117] Step 2:

[1118] Device: The audio device collects the conversation and transmits the voice data to the mobile device in real time.

[1119] Step 3:

[1120] Device: Your mobile device temporarily stores the audio data.

[1121] Step 4:

[1122] Terminal: Uploads audio data to the server periodically in batches.

[1123] 3. Audio Data Processing

[1124] Step 1:

[1125] Server: Receives the audio data.

[1126] Step 2:

[1127] Server: The speech recognition engine converts the voice data into text.

[1128] Step 3:

[1129] Server: The converted text data is saved in a database.

[1130] 4. Emotional Data Processing

[1131] Step 1:

[1132] Emotion Engine: Receives voice data.

[1133] Step 2:

[1134] Emotion engine: Analyzes emotional states from voice data.

[1135] Step 3:

[1136] Emotion Engine: Notifies users of their analyzed emotional state in real time.

[1137] 5. Search for past claims data

[1138] Step 1:

[1139] User: When a complaint occurs, use the complaint search feature in the app.

[1140] Step 2:

[1141] User: Enter keywords related to the complaint in the search box.

[1142] Step 3:

[1143] Device: The app sends a search request to the server.

[1144] Step 4:

[1145] Server: Searches the database for relevant past conversation records.

[1146] Step 5:

[1147] Server: Returns search results to the user's device.

[1148] Step 6:

[1149] Device: Display search results to the user in a list format.

[1150] 6. Check inventory information

[1151] Step 1:

[1152] User: Receives a request from a customer to check inventory.

[1153] Step 2:

[1154] User: Use the in-app inventory check feature to enter the product name or barcode.

[1155] Step 3:

[1156] Terminal: Sends an inventory check request to the server.

[1157] Step 4:

[1158] Server: Retrieves product inventory information from the database.

[1159] Step 5:

[1160] Server: Returns inventory information to the user's device.

[1161] Step 6:

[1162] Terminal: Display inventory information to the user.

[1163] 7. Sharing customer service know-how

[1164] Step 1:

[1165] Server: Continuously collects and stores records of excellent employees' responses in a know-how database.

[1166] Step 2:

[1167] Users: When they want to learn about a specific situation or how to respond in the app, they can select a category or enter keywords.

[1168] Step 3:

[1169] Terminal: Sends a know-how search request to the server.

[1170] Step 4:

[1171] Server: Searches for relevant information from a know-how database.

[1172] Step 5:

[1173] Server: Sends search results to the user's device.

[1174] Step 6:

[1175] Terminal: Displays the provided know-how information to the user.

[1176] With the above processing flow, the system of the present invention can improve the efficiency of customer service, handle complaints, check inventory, and share customer service know-how. In addition, by combining it with an emotion engine, it is possible to grasp the user's emotional state in real time and respond more appropriately.

[1177] Example 2

[1178] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1179] Conventional customer service systems have difficulty efficiently recording conversations between customers and users (store staff) and processing them in real time. Furthermore, systems that can simultaneously support multiple tasks, such as handling complaints, checking inventory, and sharing customer service know-how, are limited. Furthermore, they lack the functionality to instantly grasp the emotional state of customers and optimize responses, resulting in problems such as stress for users and a decline in the quality of customer service. An effective system that can solve these issues is needed.

[1180] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1181] In this invention, the server includes means for connecting an audio device to a user's mobile terminal, means for recording conversations between a user and a customer via the audio device, means for transmitting the recorded voice data to an information processing device in real time, means for converting the voice data into text data in the information processing device, means for saving the converted text data in a storage device, means for searching for related past data from the storage device, means for providing the search results to the user, means for analyzing the emotional state of the customer from the voice data and notifying the user in real time, means for acquiring and providing inventory information in response to a user's request, and means for collecting and sharing records of responses from excellent employees. This makes it possible to improve the efficiency of customer service, speed up complaint responses, improve the accuracy of inventory checks, share customer service know-how, and also to grasp the emotional state of customers in real time and respond appropriately.

[1182] An "audio device" is a device that collects audio and transmits that data to an external device.

[1183] "Users" refer to store staff and service personnel who use this system to provide customer service.

[1184] "Mobile terminal" refers to a small, portable information processing device such as a smartphone or tablet.

[1185] "Customer" refers to a consumer who uses a store or service.

[1186] The term "mobile device" is synonymous with a portable terminal and refers to an information processing device that is carried and used by a user.

[1187] "Information processing device" refers to a computer system for processing voice data and text data.

[1188] "Recording means" refers to devices or software capable of collecting and storing audio data.

[1189] "Voice data" refers to data that represents voice information, including the content of conversations between users and customers, in digital form.

[1190] "Text data" refers to digital data that has been converted from voice data into text information.

[1191] "Storage device" refers to hardware and software for storing data for long periods of time.

[1192] "Means for storing data in a storage device" refers to the functions and processes for storing data in a storage device.

[1193] "Means for searching past data" refers to the function or process of searching for information that matches specific conditions from data stored in a storage device.

[1194] "Means of providing search results to users" refers to the function of displaying or providing the information obtained through a search in a format that is easy for users to view.

[1195] "Means for analyzing emotional state and notifying in real time" refers to the functions and processes for determining emotions from voice data and immediately conveying that information to the user.

[1196] "Means for obtaining and providing inventory information" refers to the function of retrieving inventory information from a database or other storage device and providing it to the user.

[1197] "Means for collecting and sharing records of excellent employee interactions" refers to the functions and processes for collecting records of excellent employee interactions with customers and sharing them with other employees.

[1198] As an embodiment of the present invention, a system is constructed that combines an audio device, a mobile device, a server, and an emotion engine to improve the efficiency of customer service in stores, handle complaints, check inventory, and share customer service know-how. The system aims to collect, analyze, save, and provide conversations between customers and store staff in real time.

[1199] Basic system configuration

[1200] User (store staff)

[1201] Users can use the system to record conversations with customers, manage complaints, check inventory, share customer service know-how, and use an emotion engine to understand the emotional state of their customers and themselves in real time.

[1202] Audio Devices

[1203] The audio device is a user-worn device that collects conversations with customers and connects to a mobile device via Bluetooth.

[1204] Mobile device (smartphone or tablet)

[1205] The mobile device receives voice data from the audio device and sends it to the server and emotion engine, and some voice data is temporarily stored.

[1206] server

[1207] The server converts the received voice data into text and stores it in a database. It also manages inventory information and provides information upon user request. It also has the capacity to collect and share records of excellent employees' interactions.

[1208] Emotion Engine

[1209] The emotion engine analyzes customer and user emotions from voice data and determines their emotional state in real time.

[1210] Specific examples

[1211] Example 1: Handling complaints

[1212] If a customer complains that "the response to the previous product exchange was slow," the system operates as follows:

[1213] User: Enters "product exchange" into the app and performs a search.

[1214] Server: Searches the database for conversation records related to past "product exchanges" and provides the results to the user's mobile device.

[1215] Users: Check past responses and use them as a reference for responding quickly.

[1216] Emotion Engine: Analyzes customer emotions and advises users on appropriate actions before the issue escalates.

[1217] Example 2: Checking inventory

[1218] When a customer asks, "Do you have this T-shirt in size medium?", the system works as follows:

[1219] User: Scans the product barcode to request an inventory check.

[1220] Server: Searches the inventory database, finds that there are 5 units of the item in stock, and sends the result to the user.

[1221] User: Tell customer "In stock."

[1222] Example 3: Sharing customer service know-how

[1223] If a new staff member wants to learn "effective upselling methods," the system works as follows:

[1224] User: Searches for "upsell" in the app.

[1225] Server: Extracts effective phrases and techniques related to "upselling" from a know-how database and provides them to users.

[1226] User: Review the information provided and apply it to actual customer service.

[1227] Emotion engine: Analyzes the user's emotions and suggests relaxation methods if stress is building up.

[1228] In this way, the present invention achieves more efficient customer service and improved service quality in stores through specific operational procedures and processing flows. In addition, by combining it with an emotion engine, it is possible to grasp the user's emotional state in real time and provide more appropriate responses.

[1229] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1230] Step 1:

[1231] Connecting users and audio devices

[1232] Input: The user turns on the audio device and launches a dedicated app on their mobile device.

[1233] Specific operation: The user operates the Bluetooth settings of the mobile device to pair with the audio device. The dedicated app on the mobile device obtains the audio device ID and sends it to the server.

[1234] Data processing: The server checks the received device ID in the database to determine whether it is a registered device. If successful, the server returns a connection acknowledgement message.

[1235] Output: A "Connection successful" message will appear on the mobile device screen.

[1236] Step 2:

[1237] Collection and transmission of voice data

[1238] Input: The user initiates a conversation and the audio device collects the audio.

[1239] What it does: The audio device collects the conversation with the customer and transmits it in real time to the mobile device, which temporarily stores the audio data.

[1240] Data processing: Once a certain amount of voice data has been accumulated, it is transferred to the server and emotion engine in a batch process.

[1241] Output: The server and emotion engine receive the audio data.

[1242] Step 3:

[1243] Audio data processing

[1244] Input: Audio data sent from a mobile device.

[1245] Specific operation: The server converts the voice data into text data using a speech recognition engine such as the Google Cloud Speech-to-Text API.

[1246] Data processing: The converted text data is added with time series and related information (customer ID, date and time, etc.) and stored in a database.

[1247] Output: The saved text data.

[1248] Step 4:

[1249] Emotional Data Processing

[1250] Input: Audio data sent from a mobile device.

[1251] What it does: The emotion engine analyzes voice data to identify the emotional state of customers and users.

[1252] Data processing: The emotion engine determines emotions from the tone, speed, and content of the voice, and analyzes the results in real time.

[1253] Output: Notifications about the emotional state of the customer and user are displayed in a dedicated app on the mobile device.

[1254] Step 5:

[1255] Handling complaints

[1256] Input: User searches for keywords related to the claim in the app.

[1257] How it works: A user enters a keyword, such as "product exchange," into the app, and the server searches the database to extract relevant past complaint data.

[1258] Data processing: The data extracted as search results is sorted in chronological order and provided to the user along with related information.

[1259] Output: The app displays the past complaint handling records.

[1260] Step 6:

[1261] Check stock information

[1262] Input: The user enters the product name or barcode into a dedicated app.

[1263] Specific operation: The user scans an item with a barcode reader or enters the item name. The server searches the inventory database to obtain the stock information for that item.

[1264] Data processing: Inventory information is sent to the user's device in real time.

[1265] Output: Inventory information is displayed on the user's mobile device, and the user communicates stock availability to the customer.

[1266] Step 7:

[1267] Sharing customer service know-how

[1268] Input: A user searches the app for keywords related to a specific interaction.

[1269] Specific operation: The user enters a keyword such as "upsell" into the app. The server searches the know-how database and extracts records of related customer service know-how.

[1270] Data processing: The extracted know-how information is organized by category and provided to users.

[1271] Output: Know-how information is displayed on the app, and users can apply it to actual customer service.

[1272] As described above, this system collects, processes, and provides data according to user requests through each processing step, thereby improving the efficiency of customer service and service quality.

[1273] (Application example 2)

[1274] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1275] Real-time information collection and analysis is essential to improve the efficiency of customer service in stores, the speed of complaint handling, the speed of inventory checks, and the customer service skills of employees. However, conventional systems lack the means to comprehensively solve these issues, making it particularly difficult to incorporate emotion analysis into customer service, efficiently check inventory, and instantly share customer service know-how.

[1276] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for connecting an audio device to the user's mobile device, means for recording conversations between the user and customers via the audio device, means for transmitting the recorded voice data to the server in real time, means for converting the voice data into text data in the server, means for storing the converted text data in a database, means for searching for related past data from the database, means for providing the search results to the user, means for having the user wear smart glasses and collecting the conversation and emotional state through the glasses, means for analyzing the emotional state from the voice data using an emotion engine and notifying the user in real time, and means for displaying inventory information, complaint data, and customer service know-how on the smart glasses. This enables efficient customer service, appropriate responses based on emotion analysis, rapid inventory confirmation, and real-time sharing of customer service know-how.

[1277] An "audio device" is a device that records conversations between users and customers and transmits them as audio data to a mobile device.

[1278] "Mobile Device" means a portable electronic device that receives recorded audio data and transmits the data to a server or emotion engine.

[1279] A "server" is a central processing unit that converts voice data into text data and stores it in a database.

[1280] An "emotion engine" is an analysis device that analyzes emotions from voice data and determines emotional states in real time.

[1281] "Smart glasses" are devices worn by users that collect speech and emotional states and display information.

[1282] A "database" is a collection of information in which voice data is converted into text data and stored.

[1283] "Inventory information" is data that indicates the quantity and condition of a particular product currently in stock in the store.

[1284] "Complaint data" refers to recorded data regarding past complaints and requests from customers.

[1285] "Customer service know-how" is knowledge that allows employees to share records and techniques of excellent customer service.

[1286] "Real-time" refers to data and information being processed immediately, without delay.

[1287] A "search tool" is a method or device for finding relevant information within a database.

[1288] "Notification means" refers to a method or device for notifying the user of information such as the analyzed emotional state.

[1289] A "display" is a method or apparatus for visually presenting information to smart glasses or other devices.

[1290] This invention is a system for improving the efficiency of customer service in stores, handling complaints quickly, checking inventory, and sharing customer service know-how. This system combines audio devices, smart glasses, mobile devices, a server, and an emotion engine.

[1291] Basic system configuration

[1292] User (store staff):

[1293] The user wears smart glasses and uses a mobile device to collect conversations with customers, handle complaints, check inventory, and learn customer service know-how. The emotion engine allows the user to understand the emotional state of both the customer and the user in real time.

[1294] Audio Device:

[1295] A microphone built into smart glasses that collects customer conversations and transmits them to a mobile device.

[1296] Smart Glasses:

[1297] This device collects conversations with customers and their emotional states and displays the information, displaying customer emotion analysis results, inventory information, complaint handling data, and customer service know-how in real time.

[1298] Mobile devices:

[1299] It works in conjunction with smart glasses to send voice data to a server in real time, and also receives converted text data, emotion analysis results, and past data, and sends them to the smart glasses.

[1300] server:

[1301] The system converts voice data into text data and stores it in a database. It also uses an emotion engine to analyze emotions from the voice data and notify users in real time. It also manages complaint histories and inventory information.

[1302] Emotion Engine:

[1303] It analyzes emotions from voice data and determines the user's emotional state in real time. This information is sent to the smart glasses, which then advise the user on how to respond appropriately.

[1304] Hardware and software used

[1305] Hardware:

[1306] Smart glasses (e.g., Google Glass): devices for collecting speech and emotional state and displaying information.

[1307] Mobile devices (e.g. smartphones and tablets): process the voice data and send it to the server.

[1308] software:

[1309] Speech recognition engine (e.g. Google Speech-to-Text API): Converts voice data into text.

[1310] Sentiment analysis engine (e.g., IBM Watson Tone Analyzer): Analyzes customer and user sentiment.

[1311] Database (e.g. MySQL): Manages claims, inventory, and know-how.

[1312] System operation example

[1313] 1. Stock Check Example:

[1314] A customer asks, "Do you have this shoe in size 27.5cm?"

[1315] Users use smart glasses to scan product names or barcodes.

[1316] The server searches the inventory database.

[1317] The smart glasses will display "in stock" and the user can then direct it to the customer.

[1318] Example prompt sentence:

[1319] "Check if this item is in stock"

[1320] Customer: "Do you have these shoes in size 27.5cm?"

[1321] User: Scan product name or barcode with smart glasses

[1322] System: "Checking inventory..."

[1323] System: Shows "In stock"

[1324] 2. Examples of complaint handling:

[1325] The customer complains that the previous replacement was too slow.

[1326] The user types "exchange" into the smart glasses and searches.

[1327] The server searches past "exchange" claim data.

[1328] Past response results are displayed on the smart glasses, allowing users to respond quickly based on them.

[1329] Example prompt sentence:

[1330] "Check past complaint handling"

[1331] Customer: "The last exchange was slow."

[1332] User: Searches for "exchange" on smart glasses

[1333] System: Display "View past complaint responses"

[1334] In this way, the system will enable more efficient customer service, appropriate responses based on sentiment analysis, quick inventory checks, and real-time sharing of customer service know-how.

[1335] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1336] Step 1:

[1337] The user wears the smart glasses and connects the audio device to the mobile device. Specifically, the user turns on the smart glasses, launches the app on the mobile device, and pairs with the audio device via Bluetooth. The input is the user's operation, and the output is the connection status between the devices.

[1338] Step 2:

[1339] When a user starts a customer interaction, the audio device collects the voice. The input is the conversation between the customer and the user, and the output is the collected voice data. Specifically, the microphone of the audio device records the conversation and transmits it to the mobile device in real time.

[1340] Step 3:

[1341] The mobile device temporarily stores the voice data and transfers it to the server and emotion engine in batch processing. Specifically, once a certain amount of voice data has been collected, it is automatically sent to the server and emotion engine. The input is the collected voice data, and the output is the transmitted voice data.

[1342] Step 4:

[1343] The server receives the voice data and converts it into text data using a speech recognition engine. Specifically, speech recognition software such as the Google Speech-to-Text API is used. The input is voice data and the output is text data.

[1344] Step 5:

[1345] The server stores the text data in a database. Specifically, database software such as MySQL is used to store the data in an appropriate format. The input is text data, and the output is the stored text data.

[1346] Step 6:

[1347] The emotion engine analyzes the emotional state from the voice data and notifies the user in real time. Specifically, emotion analysis software such as IBM Watson Tone Analyzer is used. The input is the voice data, and the output is the analysis result of the emotional state.

[1348] Step 7:

[1349] Users can search for past complaint data or inventory information as needed. Specifically, they enter specific keywords into a mobile device app and execute a search. The input is the search keyword, and the output is the search results for related data.

[1350] Step 8:

[1351] The server searches the database for relevant historical data and provides it to the user. Specifically, it searches the database using an SQL query and sends the results to the mobile device. The input is the search query, and the output is the search result data.

[1352] Step 9:

[1353] The server retrieves inventory information in real time and displays it on the smart glasses. Specifically, it retrieves the latest information from the inventory database and displays the results on the user's smart glasses. The input is an inventory check request, and the output is inventory information.

[1354] Step 10:

[1355] The server collects the customer service records of outstanding employees and stores their customer service know-how in a database. Specifically, it analyzes daily customer service data, extracts effective customer service techniques, and adds them to the database. The input is customer service data, and the output is the analyzed customer service know-how.

[1356] Step 11:

[1357] The user checks customer service know-how using the smart glasses and applies it to actual customer service. Specifically, the user queries know-how for a specific situation and responds based on the information displayed on the smart glasses. The input is a know-how query request, and the output is the displayed know-how information.

[1358] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1359] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1360] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1361] [Third embodiment]

[1362] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1363] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1364] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1365] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1366] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1367] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1368] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1369] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1370] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1371] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1372] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1373] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1374] As an embodiment of the present invention, a system is constructed in which an audio device, a mobile device, and a server are linked together in order to improve the efficiency of customer service in a store.

[1375] 1. Basic system configuration

[1376] User (store staff): Uses this system to record conversations with customers, manage complaints, check inventory, and share customer service know-how.

[1377] Audio device: A device worn by the user that captures conversations with customers.

[1378] Mobile device (smartphone or tablet): Responsible for receiving audio data from the audio device and sending it to the server.

[1379] Server: Converts received voice data into text and stores it in a database. It also manages inventory information and provides information upon user request. It also collects and shares records of outstanding employees' interactions.

[1380] 2. System Operation Flow

[1381] 1. Connecting the user to the audio device

[1382] Device: The user powers on the audio device and launches the app on their mobile device, which pairs with the audio device via Bluetooth.

[1383] Server: If pairing is successful, the server receives the device ID, checks the device registration status, and approves the connection.

[1384] 2. Collection and transmission of voice data

[1385] Device: When a user initiates a customer interaction, the audio device collects the voice and transmits the collected voice data to the mobile device in real time.

[1386] Device: The mobile device temporarily stores the collected voice data and transfers it to the server in batches.

[1387] 3. Audio Data Processing

[1388] Server: Receives voice data and converts it into text data using a speech recognition engine.

[1389] Server: The converted text data is saved in a database.

[1390] 4. Search for past claims data

[1391] Users: Use the search function in the app when handling a complaint by entering keywords related to previous complaints.

[1392] Server: Searches for relevant historical data in the database and provides it to the user.

[1393] 5. Check inventory information

[1394] User: When a customer requests an inventory check, they enter the product name and barcode through the app.

[1395] Server: Retrieves inventory information from the database and provides it to users in real time.

[1396] Terminal: Displays inventory information to the user and communicates it to the customer.

[1397] 6. Sharing customer service know-how

[1398] Server: Continuously collects and stores records of excellent employees' responses in a know-how database.

[1399] Users: When they want to learn about a specific situation or how to respond in the app, they can select a category or enter keywords.

[1400] Server: Searches for relevant information from the know-how database and provides it to the user.

[1401] Examples:

[1402] Example 1: Handling complaints

[1403] If a customer complains that "the response to the previous product exchange was slow,"

[1404] User: Enters "product exchange" and performs a search.

[1405] Server: Searches past conversation records related to "product exchange" and provides three results to the user's device.

[1406] Users: Check past responses and use them as a reference for responding quickly.

[1407] Example 2: Checking inventory

[1408] If a customer asks, "Do you have this T-shirt in a medium size?"

[1409] User: Scans the product barcode to request an inventory check.

[1410] Server: Searches the inventory database, finds that there are 5 units of the item in stock, and sends the result to the user.

[1411] User: Tell customer "In stock."

[1412] Example 3: Sharing customer service know-how

[1413] If new staff want to learn "effective upselling methods,"

[1414] User: Searches for "upsell" in the app.

[1415] Server: Extracts effective phrases and techniques related to "upselling" from a know-how database and provides them to users.

[1416] User: Review the information provided and apply it to actual customer service.

[1417] As described above, the embodiment of the present invention realizes efficient customer service and improved service quality in stores through the specific operation procedures and processing flows.

[1418] The processing flow will be explained below.

[1419] Specific processing flow of the system

[1420] 1. Connecting the user to the audio device

[1421] Step 1:

[1422] User: Power on the audio device.

[1423] Step 2:

[1424] Device: Turn on Bluetooth on your smartphone and launch the dedicated app.

[1425] Step 3:

[1426] On your device: The app performs a Bluetooth scan to discover nearby audio devices.

[1427] Step 4:

[1428] Device: Displays a list of detected audio devices and lets the user select a device.

[1429] Step 5:

[1430] User: Approves to pair the selected device.

[1431] Step 6:

[1432] On the device: Notify the user that pairing was successful.

[1433] Step 7:

[1434] Server: Receives the device ID from the mobile device and checks the device registration status.

[1435] Step 8:

[1436] Server: Once registration is confirmed, approve the connection and send a notification to the device that the connection is complete.

[1437] 2. Collection and transmission of voice data

[1438] Step 1:

[1439] User: Presses the start recording button in the app.

[1440] Step 2:

[1441] Device: The audio device collects the conversation and transmits the voice data to the mobile device in real time.

[1442] Step 3:

[1443] Device: Your mobile device temporarily stores the audio data.

[1444] Step 4:

[1445] Terminal: Uploads audio data to the server periodically in batches.

[1446] 3. Audio Data Processing

[1447] Step 1:

[1448] Server: Receives the audio data.

[1449] Step 2:

[1450] Server: The speech recognition engine converts the voice data into text.

[1451] Step 3:

[1452] Server: The converted text data is saved in a database.

[1453] 4. Search for past claims data

[1454] Step 1:

[1455] User: When a complaint occurs, use the complaint search feature in the app.

[1456] Step 2:

[1457] User: Enter keywords related to the complaint in the search box.

[1458] Step 3:

[1459] Device: The app sends a search request to the server.

[1460] Step 4:

[1461] Server: Searches the database for relevant past conversation records.

[1462] Step 5:

[1463] Server: Returns search results to the user's device.

[1464] Step 6:

[1465] Device: Display search results to the user in a list format.

[1466] 5. Check inventory information

[1467] Step 1:

[1468] User: Receives a request from a customer to check inventory.

[1469] Step 2:

[1470] User: Use the in-app inventory check feature to enter the product name or barcode.

[1471] Step 3:

[1472] Terminal: Sends an inventory check request to the server.

[1473] Step 4:

[1474] Server: Retrieves product inventory information from the database.

[1475] Step 5:

[1476] Server: Returns inventory information to the user's device.

[1477] Step 6:

[1478] Terminal: Display inventory information to the user.

[1479] 6. Sharing customer service know-how

[1480] Step 1:

[1481] Server: Continuously collects and stores records of excellent employees' responses in a know-how database.

[1482] Step 2:

[1483] Users: When they want to learn about a specific situation or how to respond in the app, they can select a category or enter keywords.

[1484] Step 3:

[1485] Terminal: Sends a know-how search request to the server.

[1486] Step 4:

[1487] Server: Searches for relevant information from a know-how database.

[1488] Step 5:

[1489] Server: Sends search results to the user's device.

[1490] Step 6:

[1491] Terminal: Displays the provided know-how information to the user.

[1492] With the above processing flow, the system of the present invention can improve the efficiency of customer service for users, and can realize the sharing of know-how on handling complaints, checking inventory, and customer service.

[1493] Example 1

[1494] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1495] Conventional customer service systems have had issues with efficiently collecting and managing conversation records, quickly obtaining inventory information, and sharing excellent customer service know-how. In particular, there is a demand for systems that can efficiently process customer conversation data in real time, quickly search for past complaint handling records, and provide inventory information in a timely manner.

[1496] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1497] In this invention, the server includes means for collecting conversations via an audio device connected to the mobile terminal, means for temporarily storing the collected voice data and transmitting it to the server, and means for converting the voice data into text data and storing it in a database, thereby enabling real-time processing of conversation data with customers, rapid search of past complaint handling records, and timely provision of inventory information.

[1498] An "audio device" is an audio device that allows a user to collect conversations with customers.

[1499] "Mobile device" refers to the mobile device used by the user, such as a smartphone or tablet.

[1500] "Means of collection" refers to the function of using an audio device to record conversations between customers and users and capture them as data.

[1501] "Temporarily stored" means that the collected voice data is temporarily stored on the mobile device until it is batch processed.

[1502] "Batch processing" refers to the process of sending data to a server in bulk based on a set time or data size.

[1503] "Server" means a central processing unit for processing, storing, and providing information to users.

[1504] "Means for converting into text data" refers to the process of converting voice data into text information using a voice recognition engine.

[1505] A "database" is a data storage system for systematically storing and managing converted text data and related information.

[1506] "Searching means" refers to the functionality for locating specific information within a database.

[1507] "Means for providing search results" refers to the function of displaying the information obtained through a search on the user's mobile device.

[1508] "Means of obtaining inventory information in real time" refers to the process of instantly obtaining the latest inventory information from the database and providing it to the user.

[1509] "Excellent employee service records" refers to records of customer service provided by highly rated employees.

[1510] "Means for sharing" refers to the ability to share collected records of excellent employee responses with other users for learning and reference.

[1511] This invention is a system aimed at improving the efficiency of customer service in stores. By linking audio devices, mobile terminals, and servers, this system collects and converts conversation data with customers into text, and provides the functionality to store and search the data in a database.

[1512] Basic system configuration

[1513] User (store staff): Records conversations with customers and uses them to manage complaints, check inventory, and share customer service know-how.

[1514] Audio Device: A device for collecting audio. The user wears this device and records the conversation with the customer.

[1515] Mobile device (smartphone or tablet): A device that receives audio data from an audio device and sends it to a server.

[1516] Server: Converts voice data into text data and stores it in a database. It also provides users with related past data, inventory information, and customer service know-how.

[1517] Overall system operation

[1518] The user turns on the audio device and launches the dedicated app on the mobile device. The audio device connects to the mobile device via Bluetooth. The server receives the device ID, checks the registration status, and then approves the connection. When the user starts talking with a customer, the audio device collects voice data and transmits it to the mobile device in real time. The mobile device temporarily stores the collected voice data and transmits it to the server in batch processing.

[1519] Audio data processing

[1520] The server converts the received voice data into text data using a speech recognition engine (such as IBM Watson or Google Cloud Speech-to-Text). This text data is then stored in a database. The stored data is searchable and can be used as a record of past complaint handling, inventory information, or records of outstanding employee responses.

[1521] Specific examples

[1522] Handling complaints

[1523] If a user receives a complaint from a customer that "the previous product exchange was handled slowly":

[1524] User: Enters "product exchange" into the app and performs a search.

[1525] Server: Searches the database and provides records related to past "commodity exchanges" to the user's device.

[1526] Users: Use search results to act quickly.

[1527] Stock Check

[1528] When a customer asks, "Do you have this item?"

[1529] User: Scans the product barcode to request an inventory check.

[1530] Server: Searches the database and provides real-time inventory status to users.

[1531] User: Tell customer "In stock."

[1532] Sharing customer service know-how

[1533] If you're a new staff member and want to learn "effective upselling methods":

[1534] User: Searches for "upsell" in the app.

[1535] Server: Provides users with relevant phrases and techniques from a know-how database.

[1536] User: Review the information provided and put it into practice when serving customers.

[1537] As a result, this system will improve the efficiency of customer service and service quality in stores.

[1538] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1539] Step 1:

[1540] Connecting your audio device to your mobile device

[1541] Input: The user turns on the audio device and launches a dedicated app on their mobile device.

[1542] Output: The audio device and mobile device are connected via Bluetooth.

[1543] Specific behavior:

[1544] User: Power on the audio device.

[1545] Device: Launch the dedicated app on your mobile device and pair it with your audio device via Bluetooth.

[1546] Server: If pairing is successful, it receives the device ID, checks the device registration status, and approves the connection.

[1547] Step 2:

[1548] User initiates customer interaction

[1549] Input: The user initiates a conversation with the customer.

[1550] Output: The conversation with the customer is collected as audio data.

[1551] Specific behavior:

[1552] User: Initiate a conversation with a customer.

[1553] Audio devices: Collect voice data from user and customer conversations.

[1554] Step 3:

[1555] Temporary storage and batch processing of audio data

[1556] Input: The collected voice data is sent to the mobile device.

[1557] Output: The audio data is temporarily stored and sent to the server in batch processing.

[1558] Specific behavior:

[1559] Audio device: Sends audio data to mobile devices in real time.

[1560] Terminal: Temporarily stores received audio data and starts batch processing when a certain time or data size is reached.

[1561] Terminal: Encrypts the audio data through batch processing and sends it to the server.

[1562] Step 4:

[1563] Converting audio data to text

[1564] Input: The server receives the audio data.

[1565] Output: The audio data is converted to text data.

[1566] Specific behavior:

[1567] Server: Converts the voice data into text data using a speech recognition engine (e.g., IBM Watson or Google Cloud Speech-to-Text).

[1568] Server: Validates the accuracy of the speech recognition results and makes corrections if necessary.

[1569] Step 5:

[1570] Text data storage in a database

[1571] Input: Text data is generated.

[1572] Output: The text data is saved in the database.

[1573] Specific behavior:

[1574] Server: The converted text data is saved in a database.

[1575] Server: Verify that the data was saved correctly and generate a success message.

[1576] Step 6:

[1577] Search past claims data

[1578] Input: The user types a keyword into the app.

[1579] Output: The relevant claim record is displayed on the user's terminal.

[1580] Specific behavior:

[1581] User: Enters a keyword, such as "exchange," into the app and performs a search.

[1582] Server: Searches the database, extracts relevant past complaint handling records, and provides them to the user.

[1583] Step 7:

[1584] Check stock information

[1585] Input: The user types a stock check request into the app.

[1586] Output: The inventory information is displayed on the user's terminal.

[1587] Specific behavior:

[1588] User: Enters product name or barcode into the app to request availability.

[1589] Server: Retrieves inventory information from the database and provides it to users in real time.

[1590] Device: Display inventory information on the user's device and communicate it to the customer.

[1591] Step 8:

[1592] Sharing customer service know-how

[1593] Input: The user types a specific keyword into the app.

[1594] Output: Related response know-how is displayed on the user's terminal.

[1595] Specific behavior:

[1596] User: Enters a keyword into the app, such as "upsell," and performs a search.

[1597] Server: Extracts relevant phrases and techniques from the know-how database and provides them to users.

[1598] User: Review the information provided and put it into practice in customer service.

[1599] (Application example 1)

[1600] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1601] Conventional factory management systems rely on manual processes for managing work orders, checking inventory, handling complaints, and sharing work know-how, which can be inefficient and prone to errors. Performing these tasks efficiently is particularly difficult in large factories. This has led to a demand for automating various tasks within factories, processing data in real time, and improving work efficiency.

[1602] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1603] In this invention, the server includes means for connecting an audio device to a user's mobile device, means for recording conversations between the user and customers via the audio device, means for transmitting the recorded audio data to the server in real time, means for converting the audio data to text data in the server, means for storing the converted text data in a database, means for searching related past data from the database, means for providing the search results to the user, means for inputting and managing work instructions by voice, means for checking inventory information in real time, means for recording and searching complaint responses, and means for classifying and sharing excellent work records, which enables improved work efficiency in factories, more accurate inventory management, faster complaint responses, and the sharing of work know-how.

[1604] "Audio Device" means a device that collects audio and is connected to a User's Mobile Device.

[1605] A "mobile device" is a portable information terminal such as a smartphone or tablet.

[1606] A "recording means" is a method or apparatus for storing audio data collected via an audio device.

[1607] "Means for transmitting in real time" refers to a method or device for instantly transmitting collected voice data to a server.

[1608] A "server" is a central processing unit that processes and stores information on a network.

[1609] The "means for converting into text data" refers to a method or device that has the function of converting voice data into text information.

[1610] A "database" is a system for centrally storing and managing information.

[1611] A "searching means" is a method or device for efficiently locating specific information within a database.

[1612] "Means of providing" refers to the method or device for displaying search results and data to users.

[1613] "Means for managing work instructions" refers to a method or device for organizing and saving work instructions entered by voice so that they can be referenced as needed.

[1614] "Means for checking inventory information" refers to a method or device that allows you to grasp the inventory status of a product in real time.

[1615] "Means for recording complaint handling" refers to a method or device for recording and saving customer complaints and requests as audio data.

[1616] "Means for classifying and sharing excellent work records" refers to methods and devices for collecting effective work methods and know-how, organizing and storing them, and sharing them with other users.

[1617] As an embodiment of the present invention, a method for constructing a factory management system will be described below.

[1618] 1. Basic system configuration

[1619] Users (factory workers): Use this system to input and manage work instructions by voice, check inventory, handle complaints, and share excellent work know-how.

[1620] Audio device: A device worn by the user that collects audio such as instructions and complaints within the factory.

[1621] Mobile device (smartphone or tablet): Temporarily stores audio data received from the audio device and sends it to the server.

[1622] Server: Converts received voice data into text and stores it in a database. It also manages inventory information and provides information upon user request. It also collects and shares excellent work records.

[1623] 2. System Operation Flow

[1624] 1. Connecting the user to the audio device

[1625] Device: The user powers on the audio device and launches the app on their mobile device, which pairs with the audio device via Bluetooth.

[1626] Server: If pairing is successful, the server receives the device ID, checks the device registration status, and approves the connection.

[1627] 2. Collection and transmission of voice data

[1628] Device: When the user begins to speak work instructions, the audio device collects the voice and transmits the collected voice data to the mobile device in real time.

[1629] Device: The mobile device temporarily stores the collected voice data and transfers it to the server in batches.

[1630] 3. Audio Data Processing

[1631] Server: Receives the voice data and converts it into text data using a speech recognition engine (e.g., Google Cloud Speech-to-Text).

[1632] Server: The converted text data is stored in a database (e.g. MySQL).

[1633] 4. Search for past claims data

[1634] Users: Use the search function in the app when handling a complaint by entering keywords related to previous complaints.

[1635] Server: Searches for relevant historical data in the database and provides it to the user.

[1636] 5. Check inventory information

[1637] User: Enter the product name or barcode by voice.

[1638] Server: Retrieves inventory information from the database and provides it to users in real time.

[1639] Terminal: Displays inventory information to the user and communicates it to the customer.

[1640] 6. Share good work records

[1641] Server: Continuously collects voice data of excellent work and stores it in a know-how database.

[1642] Users: When they want to learn about a specific task or know-how in the app, they select a category or enter keywords.

[1643] Server: Searches for relevant information from the know-how database and provides it to the user.

[1644] Specific examples

[1645] Example 1: Entering and managing work orders

[1646] User: Speak instructions such as "Assemble 3 units of part A."

[1647] Server: Converts the received voice data into text and adds the instruction "Assemble 3 units of part A" to the work list.

[1648] Example 2: Checking inventory

[1649] User: Asks aloud, "What is the inventory of part A, barcode 1234?"

[1650] Server: Searches the inventory database and sends the result, "There are 5 units of part A with barcode 1234 in stock."

[1651] Example 3: Recording complaints

[1652] User: Speaks a complaint: "Machine X is not working properly."

[1653] Server: Converts the received voice data into text and stores it in the claim database.

[1654] User: Search for past complaint history regarding "Machine X".

[1655] Prompt Sentence Examples

[1656] (Voice input): "What is the current inventory?"

[1657] (Generated text): "Checking current inventory. Please wait."

[1658] In this way, the efficiency of work and the quality of service in the factory management system are improved.

[1659] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1660] Step 1:

[1661] Connecting users and audio devices

[1662] User: Power on the audio device and launch the app on their smartphone.

[1663] Input: User interaction to open the audio device and launch the app.

[1664] Output: The audio device and smartphone are paired.

[1665] Device: The app pairs with your audio device via Bluetooth.

[1666] Server: If pairing is successful, the server receives the device ID, checks the device's registration status, and approves the connection.

[1667] Step 2:

[1668] Collection and transmission of voice data

[1669] User: Start a conversation with customers or other workers.

[1670] Input: The user's voice conversation.

[1671] Output: Audio device collects audio data.

[1672] Device: The audio device sends audio data to the smartphone in real time.

[1673] Input: Audio data from the audio device.

[1674] Output: Audio data temporarily stored on your smartphone.

[1675] Terminal: The temporarily saved audio data is transferred to the server in batch processing.

[1676] Step 3:

[1677] Audio data processing

[1678] Server: Receives the voice data and analyzes it using a voice recognition engine.

[1679] Input: Audio data sent from the smartphone.

[1680] Output: Parsed text data.

[1681] Server: Converts voice data into text data using a speech recognition engine (e.g., Google Cloud Speech-to-Text).

[1682] Input: Speech data to the speech recognition engine.

[1683] Output: The content converted to text data.

[1684] Server: The converted text data is stored in a database (e.g. MySQL).

[1685] Step 4:

[1686] Search past claims data

[1687] User: When handling a complaint, use the search function in the app and enter keywords.

[1688] Input: Claim response keywords.

[1689] Output: Search results for historical claims data.

[1690] Server: Searches for relevant historical data in a database.

[1691] Input: A query search to the database.

[1692] Output: Claim data as search results.

[1693] Server: Serves search results to users.

[1694] Step 5:

[1695] Check stock information

[1696] User: Enter the product name or barcode by voice.

[1697] Input: Product name or barcode.

[1698] Output: Inventory information retrieved from the database.

[1699] Server: Retrieves inventory information from the database and provides it to users in real time.

[1700] Input: Inventory information from the database.

[1701] Output: Inventory information provided to the user.

[1702] Terminal: Displays inventory information to the user and communicates it to the customer.

[1703] Step 6:

[1704] Sharing excellent work records

[1705] Server: Continuously collects voice data of excellent work and stores it in a know-how database.

[1706] Input: Excellent work audio data.

[1707] Output: Text data stored in the know-how database.

[1708] Users: When they want to learn about a specific task or know-how in the app, they select a category or enter keywords.

[1709] Input: Keywords related to know-how.

[1710] Output: Relevant know-how information.

[1711] Server: Searches for relevant information from the know-how database and provides it to the user.

[1712] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1713] As a form for implementing the present invention, a system is constructed that combines an audio device, a mobile device, a server, and an emotion engine to improve the efficiency of customer service in stores, handle complaints, check inventory, and share customer service know-how.

[1714] 1. Basic system configuration

[1715] User (store staff): This system records conversations with customers, manages complaints, checks inventory, and shares customer service know-how. It also uses an emotion engine to understand emotional states in real time.

[1716] Audio device: A device worn by the user that captures conversations with customers.

[1717] Mobile device (smartphone or tablet): Responsible for receiving voice data from the audio device and sending it to the server and emotion engine.

[1718] Server: Converts received voice data into text and stores it in a database. It also manages inventory information and provides information upon user request. It also collects and shares records of outstanding employees' interactions.

[1719] Emotion Engine: Analyzes emotions from user and customer voice data to determine their emotional state in real time.

[1720] 2. System Operation Flow

[1721] 1. Connecting the user to the audio device

[1722] Device: The user powers on the audio device and launches the app on their mobile device, which pairs with the audio device via Bluetooth.

[1723] Server: If pairing is successful, the server receives the device ID, checks the device registration status, and approves the connection.

[1724] 2. Collection and transmission of voice data

[1725] Device: When a user initiates a customer interaction, the audio device collects the voice and transmits the collected voice data to the mobile device in real time.

[1726] Terminal: The mobile device temporarily stores the voice data and transfers it to the server and emotion engine in batch processing.

[1727] 3. Audio Data Processing

[1728] Server: Receives the voice data and converts it into text using a speech recognition engine.

[1729] Server: The converted text data is saved in a database.

[1730] 4. Emotional Data Processing

[1731] Emotion Engine: Analyzes emotional state from voice data and notifies the user in real time.

[1732] 5. Search for past claims data

[1733] Users: Use the search function in the app when handling a complaint by entering keywords related to previous complaints.

[1734] Server: Searches for relevant historical data in the database and provides it to the user.

[1735] 6. Check inventory information

[1736] User: When a customer requests an inventory check, they enter the product name and barcode through the app.

[1737] Server: Retrieves inventory information from the database and provides it to users in real time.

[1738] Terminal: Displays inventory information to the user and communicates it to the customer.

[1739] 7. Sharing customer service know-how

[1740] Server: Continuously collects and stores records of excellent employees' responses in a know-how database.

[1741] Users: When they want to learn about a specific situation or how to respond in the app, they can select a category or enter keywords.

[1742] Server: Searches for relevant information from the know-how database and provides it to the user.

[1743] Specific examples

[1744] Example 1: Handling complaints

[1745] If a customer complains that "the response to the previous product exchange was slow,"

[1746] User: Enters "product exchange" and performs a search.

[1747] Server: Searches past conversation records related to "product exchange" and provides three results to the user's device.

[1748] Users: Check past responses and use them as a reference for responding quickly.

[1749] Emotion Engine: It also analyzes customer emotions and advises users on appropriate actions before the issue escalates.

[1750] Example 2: Checking inventory

[1751] If a customer asks, "Do you have this T-shirt in a medium size?"

[1752] User: Scans the product barcode to request an inventory check.

[1753] Server: Searches the inventory database, finds that there are 5 units of the item in stock, and sends the result to the user.

[1754] User: Tell customer "In stock."

[1755] Example 3: Sharing customer service know-how

[1756] If new staff want to learn "effective upselling methods,"

[1757] User: Searches for "upsell" in the app.

[1758] Server: Extracts effective phrases and techniques related to "upselling" from a know-how database and provides them to users.

[1759] User: Review the information provided and apply it to actual customer service.

[1760] Emotion engine: Analyzes the user's emotions and suggests relaxation methods if stress is building up.

[1761] As described above, the embodiment of the present invention realizes efficient customer service and improved service quality in stores through the specific operating procedures and processing flows. In addition, by combining it with an emotion engine, it becomes possible to grasp the user's emotional state in real time and provide more appropriate service.

[1762] The processing flow will be explained below.

[1763] Specific processing flow of the system

[1764] 1. Connecting the user to the audio device

[1765] Step 1:

[1766] User: Power on the audio device.

[1767] Step 2:

[1768] Device: Turn on Bluetooth on your smartphone and launch the dedicated app.

[1769] Step 3:

[1770] On your device: The app performs a Bluetooth scan to discover nearby audio devices.

[1771] Step 4:

[1772] Device: Displays a list of detected audio devices and lets the user select a device.

[1773] Step 5:

[1774] User: Approves to pair the selected device.

[1775] Step 6:

[1776] On the device: Notify the user that pairing was successful.

[1777] Step 7:

[1778] Server: Receives the device ID from the mobile device and checks the device registration status.

[1779] Step 8:

[1780] Server: Once registration is confirmed, approve the connection and send a notification to the device that the connection is complete.

[1781] 2. Collection and transmission of voice data

[1782] Step 1:

[1783] User: Presses the start recording button in the app.

[1784] Step 2:

[1785] Device: The audio device collects the conversation and transmits the voice data to the mobile device in real time.

[1786] Step 3:

[1787] Device: Your mobile device temporarily stores the audio data.

[1788] Step 4:

[1789] Terminal: Uploads audio data to the server periodically in batches.

[1790] 3. Audio Data Processing

[1791] Step 1:

[1792] Server: Receives the audio data.

[1793] Step 2:

[1794] Server: The speech recognition engine converts the voice data into text.

[1795] Step 3:

[1796] Server: The converted text data is saved in a database.

[1797] 4. Emotional Data Processing

[1798] Step 1:

[1799] Emotion Engine: Receives voice data.

[1800] Step 2:

[1801] Emotion engine: Analyzes emotional states from voice data.

[1802] Step 3:

[1803] Emotion Engine: Notifies users of their analyzed emotional state in real time.

[1804] 5. Search for past claims data

[1805] Step 1:

[1806] User: When a complaint occurs, use the complaint search feature in the app.

[1807] Step 2:

[1808] User: Enter keywords related to the complaint in the search box.

[1809] Step 3:

[1810] Device: The app sends a search request to the server.

[1811] Step 4:

[1812] Server: Searches the database for relevant past conversation records.

[1813] Step 5:

[1814] Server: Returns search results to the user's device.

[1815] Step 6:

[1816] Device: Display search results to the user in a list format.

[1817] 6. Check inventory information

[1818] Step 1:

[1819] User: Receives a request from a customer to check inventory.

[1820] Step 2:

[1821] User: Use the in-app inventory check feature to enter the product name or barcode.

[1822] Step 3:

[1823] Terminal: Sends an inventory check request to the server.

[1824] Step 4:

[1825] Server: Retrieves product inventory information from the database.

[1826] Step 5:

[1827] Server: Returns inventory information to the user's device.

[1828] Step 6:

[1829] Terminal: Display inventory information to the user.

[1830] 7. Sharing customer service know-how

[1831] Step 1:

[1832] Server: Continuously collects and stores records of excellent employees' responses in a know-how database.

[1833] Step 2:

[1834] Users: When they want to learn about a specific situation or how to respond in the app, they can select a category or enter keywords.

[1835] Step 3:

[1836] Terminal: Sends a know-how search request to the server.

[1837] Step 4:

[1838] Server: Searches for relevant information from a know-how database.

[1839] Step 5:

[1840] Server: Sends search results to the user's device.

[1841] Step 6:

[1842] Terminal: Displays the provided know-how information to the user.

[1843] With the above processing flow, the system of the present invention can improve the efficiency of customer service, handle complaints, check inventory, and share customer service know-how. In addition, by combining it with an emotion engine, it is possible to grasp the user's emotional state in real time and respond more appropriately.

[1844] Example 2

[1845] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1846] Conventional customer service systems have difficulty efficiently recording conversations between customers and users (store staff) and processing them in real time. Furthermore, systems that can simultaneously support multiple tasks, such as handling complaints, checking inventory, and sharing customer service know-how, are limited. Furthermore, they lack the functionality to instantly grasp the emotional state of customers and optimize responses, resulting in problems such as stress for users and a decline in the quality of customer service. An effective system that can solve these issues is needed.

[1847] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1848] In this invention, the server includes means for connecting an audio device to a user's mobile terminal, means for recording conversations between a user and a customer via the audio device, means for transmitting the recorded voice data to an information processing device in real time, means for converting the voice data into text data in the information processing device, means for saving the converted text data in a storage device, means for searching for related past data from the storage device, means for providing the search results to the user, means for analyzing the emotional state of the customer from the voice data and notifying the user in real time, means for acquiring and providing inventory information in response to a user's request, and means for collecting and sharing records of responses from excellent employees. This makes it possible to improve the efficiency of customer service, speed up complaint responses, improve the accuracy of inventory checks, share customer service know-how, and also to grasp the emotional state of customers in real time and respond appropriately.

[1849] An "audio device" is a device that collects audio and transmits that data to an external device.

[1850] "Users" refer to store staff and service personnel who use this system to provide customer service.

[1851] "Mobile terminal" refers to a small, portable information processing device such as a smartphone or tablet.

[1852] "Customer" refers to a consumer who uses a store or service.

[1853] The term "mobile device" is synonymous with a portable terminal and refers to an information processing device that is carried and used by a user.

[1854] "Information processing device" refers to a computer system for processing voice data and text data.

[1855] "Recording means" refers to devices or software capable of collecting and storing audio data.

[1856] "Voice data" refers to data that represents voice information, including the content of conversations between users and customers, in digital form.

[1857] "Text data" refers to digital data that has been converted from voice data into text information.

[1858] "Storage device" refers to hardware and software for storing data for long periods of time.

[1859] "Means for storing data in a storage device" refers to the functions and processes for storing data in a storage device.

[1860] "Means for searching past data" refers to the function or process of searching for information that matches specific conditions from data stored in a storage device.

[1861] "Means of providing search results to users" refers to the function of displaying or providing the information obtained through a search in a format that is easy for users to view.

[1862] "Means for analyzing emotional state and notifying in real time" refers to the functions and processes for determining emotions from voice data and immediately conveying that information to the user.

[1863] "Means for obtaining and providing inventory information" refers to the function of retrieving inventory information from a database or other storage device and providing it to the user.

[1864] "Means for collecting and sharing records of excellent employee interactions" refers to the functions and processes for collecting records of excellent employee interactions with customers and sharing them with other employees.

[1865] As an embodiment of the present invention, a system is constructed that combines an audio device, a mobile device, a server, and an emotion engine to improve the efficiency of customer service in stores, handle complaints, check inventory, and share customer service know-how. The system aims to collect, analyze, save, and provide conversations between customers and store staff in real time.

[1866] Basic system configuration

[1867] User (store staff)

[1868] Users can use the system to record conversations with customers, manage complaints, check inventory, share customer service know-how, and use an emotion engine to understand the emotional state of their customers and themselves in real time.

[1869] Audio Devices

[1870] The audio device is a user-worn device that collects conversations with customers and connects to a mobile device via Bluetooth.

[1871] Mobile device (smartphone or tablet)

[1872] The mobile device receives voice data from the audio device and sends it to the server and emotion engine, and some voice data is temporarily stored.

[1873] server

[1874] The server converts the received voice data into text and stores it in a database. It also manages inventory information and provides information upon user request. It also has the capacity to collect and share records of excellent employees' interactions.

[1875] Emotion Engine

[1876] The emotion engine analyzes customer and user emotions from voice data and determines their emotional state in real time.

[1877] Specific examples

[1878] Example 1: Handling complaints

[1879] If a customer complains that "the response to the previous product exchange was slow," the system operates as follows:

[1880] User: Enters "product exchange" into the app and performs a search.

[1881] Server: Searches the database for conversation records related to past "product exchanges" and provides the results to the user's mobile device.

[1882] Users: Check past responses and use them as a reference for responding quickly.

[1883] Emotion Engine: Analyzes customer emotions and advises users on appropriate actions before the issue escalates.

[1884] Example 2: Checking inventory

[1885] When a customer asks, "Do you have this T-shirt in size medium?", the system works as follows:

[1886] User: Scans the product barcode to request an inventory check.

[1887] Server: Searches the inventory database, finds that there are 5 units of the item in stock, and sends the result to the user.

[1888] User: Tell customer "In stock."

[1889] Example 3: Sharing customer service know-how

[1890] If a new staff member wants to learn "effective upselling methods," the system works as follows:

[1891] User: Searches for "upsell" in the app.

[1892] Server: Extracts effective phrases and techniques related to "upselling" from a know-how database and provides them to users.

[1893] User: Review the information provided and apply it to actual customer service.

[1894] Emotion engine: Analyzes the user's emotions and suggests relaxation methods if stress is building up.

[1895] In this way, the present invention achieves more efficient customer service and improved service quality in stores through specific operational procedures and processing flows. In addition, by combining it with an emotion engine, it is possible to grasp the user's emotional state in real time and provide more appropriate responses.

[1896] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1897] Step 1:

[1898] Connecting users and audio devices

[1899] Input: The user turns on the audio device and launches a dedicated app on their mobile device.

[1900] Specific operation: The user operates the Bluetooth settings of the mobile device to pair with the audio device. The dedicated app on the mobile device obtains the audio device ID and sends it to the server.

[1901] Data processing: The server checks the received device ID in the database to determine whether it is a registered device. If successful, the server returns a connection acknowledgement message.

[1902] Output: A "Connection successful" message will appear on the mobile device screen.

[1903] Step 2:

[1904] Collection and transmission of voice data

[1905] Input: The user initiates a conversation and the audio device collects the audio.

[1906] What it does: The audio device collects the conversation with the customer and transmits it in real time to the mobile device, which temporarily stores the audio data.

[1907] Data processing: Once a certain amount of voice data has been accumulated, it is transferred to the server and emotion engine in a batch process.

[1908] Output: The server and emotion engine receive the audio data.

[1909] Step 3:

[1910] Audio data processing

[1911] Input: Audio data sent from a mobile device.

[1912] Specific operation: The server converts the voice data into text data using a speech recognition engine such as the Google Cloud Speech-to-Text API.

[1913] Data processing: The converted text data is added with time series and related information (customer ID, date and time, etc.) and stored in a database.

[1914] Output: The saved text data.

[1915] Step 4:

[1916] Emotional Data Processing

[1917] Input: Audio data sent from a mobile device.

[1918] What it does: The emotion engine analyzes voice data to identify the emotional state of customers and users.

[1919] Data processing: The emotion engine determines emotions from the tone, speed, and content of the voice, and analyzes the results in real time.

[1920] Output: Notifications about the emotional state of the customer and user are displayed in a dedicated app on the mobile device.

[1921] Step 5:

[1922] Handling complaints

[1923] Input: User searches for keywords related to the claim in the app.

[1924] How it works: A user enters a keyword, such as "product exchange," into the app, and the server searches the database to extract relevant past complaint data.

[1925] Data processing: The data extracted as search results is sorted in chronological order and provided to the user along with related information.

[1926] Output: The app displays the past complaint handling records.

[1927] Step 6:

[1928] Check stock information

[1929] Input: The user enters the product name or barcode into a dedicated app.

[1930] Specific operation: The user scans an item with a barcode reader or enters the item name. The server searches the inventory database to obtain the stock information for that item.

[1931] Data processing: Inventory information is sent to the user's device in real time.

[1932] Output: Inventory information is displayed on the user's mobile device, and the user communicates stock availability to the customer.

[1933] Step 7:

[1934] Sharing customer service know-how

[1935] Input: A user searches the app for keywords related to a specific interaction.

[1936] Specific operation: The user enters a keyword such as "upsell" into the app. The server searches the know-how database and extracts records of related customer service know-how.

[1937] Data processing: The extracted know-how information is organized by category and provided to users.

[1938] Output: Know-how information is displayed on the app, and users can apply it to actual customer service.

[1939] As described above, this system collects, processes, and provides data according to user requests through each processing step, thereby improving the efficiency of customer service and service quality.

[1940] (Application example 2)

[1941] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1942] Real-time information collection and analysis is essential to improve the efficiency of customer service in stores, the speed of complaint handling, the speed of inventory checks, and the customer service skills of employees. However, conventional systems lack the means to comprehensively solve these issues, making it particularly difficult to incorporate emotion analysis into customer service, efficiently check inventory, and instantly share customer service know-how.

[1943] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for connecting an audio device to the user's mobile device, means for recording conversations between the user and customers via the audio device, means for transmitting the recorded voice data to the server in real time, means for converting the voice data into text data in the server, means for storing the converted text data in a database, means for searching for related past data from the database, means for providing the search results to the user, means for having the user wear smart glasses and collecting the conversation and emotional state through the glasses, means for analyzing the emotional state from the voice data using an emotion engine and notifying the user in real time, and means for displaying inventory information, complaint data, and customer service know-how on the smart glasses. This enables efficient customer service, appropriate responses based on emotion analysis, rapid inventory confirmation, and real-time sharing of customer service know-how.

[1944] An "audio device" is a device that records conversations between users and customers and transmits them as audio data to a mobile device.

[1945] "Mobile Device" means a portable electronic device that receives recorded audio data and transmits the data to a server or emotion engine.

[1946] A "server" is a central processing unit that converts voice data into text data and stores it in a database.

[1947] An "emotion engine" is an analysis device that analyzes emotions from voice data and determines emotional states in real time.

[1948] "Smart glasses" are devices worn by users that collect speech and emotional states and display information.

[1949] A "database" is a collection of information in which voice data is converted into text data and stored.

[1950] "Inventory information" is data that indicates the quantity and condition of a particular product currently in stock in the store.

[1951] "Complaint data" refers to recorded data regarding past complaints and requests from customers.

[1952] "Customer service know-how" is knowledge that allows employees to share records and techniques of excellent customer service.

[1953] "Real-time" refers to data and information being processed immediately, without delay.

[1954] A "search tool" is a method or device for finding relevant information within a database.

[1955] "Notification means" refers to a method or device for notifying the user of information such as the analyzed emotional state.

[1956] A "display" is a method or apparatus for visually presenting information to smart glasses or other devices.

[1957] This invention is a system for improving the efficiency of customer service in stores, handling complaints quickly, checking inventory, and sharing customer service know-how. This system combines audio devices, smart glasses, mobile devices, a server, and an emotion engine.

[1958] Basic system configuration

[1959] User (store staff):

[1960] The user wears smart glasses and uses a mobile device to collect conversations with customers, handle complaints, check inventory, and learn customer service know-how. The emotion engine allows the user to understand the emotional state of both the customer and the user in real time.

[1961] Audio Device:

[1962] A microphone built into smart glasses that collects customer conversations and transmits them to a mobile device.

[1963] Smart Glasses:

[1964] This device collects conversations with customers and their emotional states and displays the information, displaying customer emotion analysis results, inventory information, complaint handling data, and customer service know-how in real time.

[1965] Mobile devices:

[1966] It works in conjunction with smart glasses to send voice data to a server in real time, and also receives converted text data, emotion analysis results, and past data, and sends them to the smart glasses.

[1967] server:

[1968] The system converts voice data into text data and stores it in a database. It also uses an emotion engine to analyze emotions from the voice data and notify users in real time. It also manages complaint histories and inventory information.

[1969] Emotion Engine:

[1970] It analyzes emotions from voice data and determines the user's emotional state in real time. This information is sent to the smart glasses, which then advise the user on how to respond appropriately.

[1971] Hardware and software used

[1972] Hardware:

[1973] Smart glasses (e.g., Google Glass): devices for collecting speech and emotional state and displaying information.

[1974] Mobile devices (e.g. smartphones and tablets): process the voice data and send it to the server.

[1975] software:

[1976] Speech recognition engine (e.g. Google Speech-to-Text API): Converts voice data into text.

[1977] Sentiment analysis engine (e.g., IBM Watson Tone Analyzer): Analyzes customer and user sentiment.

[1978] Database (e.g. MySQL): Manages claims, inventory, and know-how.

[1979] System operation example

[1980] 1. Stock Check Example:

[1981] A customer asks, "Do you have this shoe in size 27.5cm?"

[1982] Users use smart glasses to scan product names or barcodes.

[1983] The server searches the inventory database.

[1984] The smart glasses will display "in stock" and the user can then direct it to the customer.

[1985] Example prompt sentence:

[1986] "Check if this item is in stock"

[1987] Customer: "Do you have these shoes in size 27.5cm?"

[1988] User: Scan product name or barcode with smart glasses

[1989] System: "Checking inventory..."

[1990] System: Shows "In stock"

[1991] 2. Examples of complaint handling:

[1992] The customer complains that the previous replacement was too slow.

[1993] The user types "exchange" into the smart glasses and searches.

[1994] The server searches past "exchange" claim data.

[1995] Past response results are displayed on the smart glasses, allowing users to respond quickly based on them.

[1996] Example prompt sentence:

[1997] "Check past complaint handling"

[1998] Customer: "The last exchange was slow."

[1999] User: Searches for "exchange" on smart glasses

[2000] System: Display "View past complaint responses"

[2001] In this way, the system will enable more efficient customer service, appropriate responses based on sentiment analysis, quick inventory checks, and real-time sharing of customer service know-how.

[2002] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2003] Step 1:

[2004] The user wears the smart glasses and connects the audio device to the mobile device. Specifically, the user turns on the smart glasses, launches the app on the mobile device, and pairs with the audio device via Bluetooth. The input is the user's operation, and the output is the connection status between the devices.

[2005] Step 2:

[2006] When a user starts a customer interaction, the audio device collects the voice. The input is the conversation between the customer and the user, and the output is the collected voice data. Specifically, the microphone of the audio device records the conversation and transmits it to the mobile device in real time.

[2007] Step 3:

[2008] The mobile device temporarily stores the voice data and transfers it to the server and emotion engine in batch processing. Specifically, once a certain amount of voice data has been collected, it is automatically sent to the server and emotion engine. The input is the collected voice data, and the output is the transmitted voice data.

[2009] Step 4:

[2010] The server receives the voice data and converts it into text data using a speech recognition engine. Specifically, speech recognition software such as the Google Speech-to-Text API is used. The input is voice data and the output is text data.

[2011] Step 5:

[2012] The server stores the text data in a database. Specifically, database software such as MySQL is used to store the data in an appropriate format. The input is text data, and the output is the stored text data.

[2013] Step 6:

[2014] The emotion engine analyzes the emotional state from the voice data and notifies the user in real time. Specifically, emotion analysis software such as IBM Watson Tone Analyzer is used. The input is the voice data, and the output is the analysis result of the emotional state.

[2015] Step 7:

[2016] Users can search for past complaint data or inventory information as needed. Specifically, they enter specific keywords into a mobile device app and execute a search. The input is the search keyword, and the output is the search results for related data.

[2017] Step 8:

[2018] The server searches the database for relevant historical data and provides it to the user. Specifically, it searches the database using an SQL query and sends the results to the mobile device. The input is the search query, and the output is the search result data.

[2019] Step 9:

[2020] The server retrieves inventory information in real time and displays it on the smart glasses. Specifically, it retrieves the latest information from the inventory database and displays the results on the user's smart glasses. The input is an inventory check request, and the output is inventory information.

[2021] Step 10:

[2022] The server collects the customer service records of outstanding employees and stores their customer service know-how in a database. Specifically, it analyzes daily customer service data, extracts effective customer service techniques, and adds them to the database. The input is customer service data, and the output is the analyzed customer service know-how.

[2023] Step 11:

[2024] The user checks customer service know-how using the smart glasses and applies it to actual customer service. Specifically, the user queries know-how for a specific situation and responds based on the information displayed on the smart glasses. The input is a know-how query request, and the output is the displayed know-how information.

[2025] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[2026] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2027] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[2028] [Fourth embodiment]

[2029] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[2030] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[2031] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[2032] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[2033] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[2034] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[2035] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[2036] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[2037] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[2038] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[2039] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[2040] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[2041] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2042] As an embodiment of the present invention, a system is constructed in which an audio device, a mobile device, and a server are linked together in order to improve the efficiency of customer service in a store.

[2043] 1. Basic system configuration

[2044] User (store staff): Uses this system to record conversations with customers, manage complaints, check inventory, and share customer service know-how.

[2045] Audio device: A device worn by the user that captures conversations with customers.

[2046] Mobile device (smartphone or tablet): Responsible for receiving audio data from the audio device and sending it to the server.

[2047] Server: Converts received voice data into text and stores it in a database. It also manages inventory information and provides information upon user request. It also collects and shares records of outstanding employees' interactions.

[2048] 2. System Operation Flow

[2049] 1. Connecting the user to the audio device

[2050] Device: The user powers on the audio device and launches the app on their mobile device, which pairs with the audio device via Bluetooth.

[2051] Server: If pairing is successful, the server receives the device ID, checks the device registration status, and approves the connection.

[2052] 2. Collection and transmission of voice data

[2053] Device: When a user initiates a customer interaction, the audio device collects the voice and transmits the collected voice data to the mobile device in real time.

[2054] Device: The mobile device temporarily stores the collected voice data and transfers it to the server in batches.

[2055] 3. Audio Data Processing

[2056] Server: Receives voice data and converts it into text data using a speech recognition engine.

[2057] Server: The converted text data is saved in a database.

[2058] 4. Search for past claims data

[2059] Users: Use the search function in the app when handling a complaint by entering keywords related to previous complaints.

[2060] Server: Searches for relevant historical data in the database and provides it to the user.

[2061] 5. Check inventory information

[2062] User: When a customer requests an inventory check, they enter the product name and barcode through the app.

[2063] Server: Retrieves inventory information from the database and provides it to users in real time.

[2064] Terminal: Displays inventory information to the user and communicates it to the customer.

[2065] 6. Sharing customer service know-how

[2066] Server: Continuously collects and stores records of excellent employees' responses in a know-how database.

[2067] Users: When they want to learn about a specific situation or how to respond in the app, they can select a category or enter keywords.

[2068] Server: Searches for relevant information from the know-how database and provides it to the user.

[2069] Examples:

[2070] Example 1: Handling complaints

[2071] If a customer complains that "the response to the previous product exchange was slow,"

[2072] User: Enters "product exchange" and performs a search.

[2073] Server: Searches past conversation records related to "product exchange" and provides three results to the user's device.

[2074] Users: Check past responses and use them as a reference for responding quickly.

[2075] Example 2: Checking inventory

[2076] If a customer asks, "Do you have this T-shirt in a medium size?"

[2077] User: Scans the product barcode to request an inventory check.

[2078] Server: Searches the inventory database, finds that there are 5 units of the item in stock, and sends the result to the user.

[2079] User: Tell customer "In stock."

[2080] Example 3: Sharing customer service know-how

[2081] If new staff want to learn "effective upselling methods,"

[2082] User: Searches for "upsell" in the app.

[2083] Server: Extracts effective phrases and techniques related to "upselling" from a know-how database and provides them to users.

[2084] User: Review the information provided and apply it to actual customer service.

[2085] As described above, the embodiment of the present invention realizes efficient customer service and improved service quality in stores through the specific operation procedures and processing flows.

[2086] The processing flow will be explained below.

[2087] Specific processing flow of the system

[2088] 1. Connecting the user to the audio device

[2089] Step 1:

[2090] User: Power on the audio device.

[2091] Step 2:

[2092] Device: Turn on Bluetooth on your smartphone and launch the dedicated app.

[2093] Step 3:

[2094] On your device: The app performs a Bluetooth scan to discover nearby audio devices.

[2095] Step 4:

[2096] Device: Displays a list of detected audio devices and lets the user select a device.

[2097] Step 5:

[2098] User: Approves to pair the selected device.

[2099] Step 6:

[2100] On the device: Notify the user that pairing was successful.

[2101] Step 7:

[2102] Server: Receives the device ID from the mobile device and checks the device registration status.

[2103] Step 8:

[2104] Server: Once registration is confirmed, approve the connection and send a notification to the device that the connection is complete.

[2105] 2. Collection and transmission of voice data

[2106] Step 1:

[2107] User: Presses the start recording button in the app.

[2108] Step 2:

[2109] Device: The audio device collects the conversation and transmits the voice data to the mobile device in real time.

[2110] Step 3:

[2111] Device: Your mobile device temporarily stores the audio data.

[2112] Step 4:

[2113] Terminal: Uploads audio data to the server periodically in batches.

[2114] 3. Audio Data Processing

[2115] Step 1:

[2116] Server: Receives the audio data.

[2117] Step 2:

[2118] Server: The speech recognition engine converts the voice data into text.

[2119] Step 3:

[2120] Server: The converted text data is saved in a database.

[2121] 4. Search for past claims data

[2122] Step 1:

[2123] User: When a complaint occurs, use the complaint search feature in the app.

[2124] Step 2:

[2125] User: Enter keywords related to the complaint in the search box.

[2126] Step 3:

[2127] Device: The app sends a search request to the server.

[2128] Step 4:

[2129] Server: Searches the database for relevant past conversation records.

[2130] Step 5:

[2131] Server: Returns search results to the user's device.

[2132] Step 6:

[2133] Device: Display search results to the user in a list format.

[2134] 5. Check inventory information

[2135] Step 1:

[2136] User: Receives a request from a customer to check inventory.

[2137] Step 2:

[2138] User: Use the in-app inventory check feature to enter the product name or barcode.

[2139] Step 3:

[2140] Terminal: Sends an inventory check request to the server.

[2141] Step 4:

[2142] Server: Retrieves product inventory information from the database.

[2143] Step 5:

[2144] Server: Returns inventory information to the user's device.

[2145] Step 6:

[2146] Terminal: Display inventory information to the user.

[2147] 6. Sharing customer service know-how

[2148] Step 1:

[2149] Server: Continuously collects and stores records of excellent employees' responses in a know-how database.

[2150] Step 2:

[2151] Users: When they want to learn about a specific situation or how to respond in the app, they can select a category or enter keywords.

[2152] Step 3:

[2153] Terminal: Sends a know-how search request to the server.

[2154] Step 4:

[2155] Server: Searches for relevant information from a know-how database.

[2156] Step 5:

[2157] Server: Sends search results to the user's device.

[2158] Step 6:

[2159] Terminal: Displays the provided know-how information to the user.

[2160] With the above processing flow, the system of the present invention can improve the efficiency of customer service for users, and can realize the sharing of know-how on handling complaints, checking inventory, and customer service.

[2161] Example 1

[2162] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2163] Conventional customer service systems have had issues with efficiently collecting and managing conversation records, quickly obtaining inventory information, and sharing excellent customer service know-how. In particular, there is a demand for systems that can efficiently process customer conversation data in real time, quickly search for past complaint handling records, and provide inventory information in a timely manner.

[2164] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[2165] In this invention, the server includes means for collecting conversations via an audio device connected to the mobile terminal, means for temporarily storing the collected voice data and transmitting it to the server, and means for converting the voice data into text data and storing it in a database, thereby enabling real-time processing of conversation data with customers, rapid search of past complaint handling records, and timely provision of inventory information.

[2166] An "audio device" is an audio device that allows a user to collect conversations with customers.

[2167] "Mobile device" refers to the mobile device used by the user, such as a smartphone or tablet.

[2168] "Means of collection" refers to the function of using an audio device to record conversations between customers and users and capture them as data.

[2169] "Temporarily stored" means that the collected voice data is temporarily stored on the mobile device until it is batch processed.

[2170] "Batch processing" refers to the process of sending data to a server in bulk based on a set time or data size.

[2171] "Server" means a central processing unit for processing, storing, and providing information to users.

[2172] "Means for converting into text data" refers to the process of converting voice data into text information using a voice recognition engine.

[2173] A "database" is a data storage system for systematically storing and managing converted text data and related information.

[2174] "Searching means" refers to the functionality for locating specific information within a database.

[2175] "Means for providing search results" refers to the function of displaying the information obtained through a search on the user's mobile device.

[2176] "Means of obtaining inventory information in real time" refers to the process of instantly obtaining the latest inventory information from the database and providing it to the user.

[2177] "Excellent employee service records" refers to records of customer service provided by highly rated employees.

[2178] "Means for sharing" refers to the ability to share collected records of excellent employee responses with other users for learning and reference.

[2179] This invention is a system aimed at improving the efficiency of customer service in stores. By linking audio devices, mobile terminals, and servers, this system collects and converts conversation data with customers into text, and provides the functionality to store and search the data in a database.

[2180] Basic system configuration

[2181] User (store staff): Records conversations with customers and uses them to manage complaints, check inventory, and share customer service know-how.

[2182] Audio Device: A device for collecting audio. The user wears this device and records the conversation with the customer.

[2183] Mobile device (smartphone or tablet): A device that receives audio data from an audio device and sends it to a server.

[2184] Server: Converts voice data into text data and stores it in a database. It also provides users with related past data, inventory information, and customer service know-how.

[2185] Overall system operation

[2186] The user turns on the audio device and launches the dedicated app on the mobile device. The audio device connects to the mobile device via Bluetooth. The server receives the device ID, checks the registration status, and then approves the connection. When the user starts talking with a customer, the audio device collects voice data and transmits it to the mobile device in real time. The mobile device temporarily stores the collected voice data and transmits it to the server in batch processing.

[2187] Audio data processing

[2188] The server converts the received voice data into text data using a speech recognition engine (such as IBM Watson or Google Cloud Speech-to-Text). This text data is then stored in a database. The stored data is searchable and can be used as a record of past complaint handling, inventory information, or records of outstanding employee responses.

[2189] Specific examples

[2190] Handling complaints

[2191] If a user receives a complaint from a customer that "the previous product exchange was handled slowly":

[2192] User: Enters "product exchange" into the app and performs a search.

[2193] Server: Searches the database and provides records related to past "commodity exchanges" to the user's device.

[2194] Users: Use search results to act quickly.

[2195] Stock Check

[2196] When a customer asks, "Do you have this item?"

[2197] User: Scans the product barcode to request an inventory check.

[2198] Server: Searches the database and provides real-time inventory status to users.

[2199] User: Tell customer "In stock."

[2200] Sharing customer service know-how

[2201] If you're a new staff member and want to learn "effective upselling methods":

[2202] User: Searches for "upsell" in the app.

[2203] Server: Provides users with relevant phrases and techniques from a know-how database.

[2204] User: Review the information provided and put it into practice when serving customers.

[2205] As a result, this system will improve the efficiency of customer service and service quality in stores.

[2206] The flow of the identification process in the first embodiment will be described with reference to FIG.

[2207] Step 1:

[2208] Connecting your audio device to your mobile device

[2209] Input: The user turns on the audio device and launches a dedicated app on their mobile device.

[2210] Output: The audio device and mobile device are connected via Bluetooth.

[2211] Specific behavior:

[2212] User: Power on the audio device.

[2213] Device: Launch the dedicated app on your mobile device and pair it with your audio device via Bluetooth.

[2214] Server: If pairing is successful, it receives the device ID, checks the device registration status, and approves the connection.

[2215] Step 2:

[2216] User initiates customer interaction

[2217] Input: The user initiates a conversation with the customer.

[2218] Output: The conversation with the customer is collected as audio data.

[2219] Specific behavior:

[2220] User: Initiate a conversation with a customer.

[2221] Audio devices: Collect voice data from user and customer conversations.

[2222] Step 3:

[2223] Temporary storage and batch processing of audio data

[2224] Input: The collected voice data is sent to the mobile device.

[2225] Output: The audio data is temporarily stored and sent to the server in batch processing.

[2226] Specific behavior:

[2227] Audio device: Sends audio data to mobile devices in real time.

[2228] Terminal: Temporarily stores received audio data and starts batch processing when a certain time or data size is reached.

[2229] Terminal: Encrypts the audio data through batch processing and sends it to the server.

[2230] Step 4:

[2231] Converting audio data to text

[2232] Input: The server receives the audio data.

[2233] Output: The audio data is converted to text data.

[2234] Specific behavior:

[2235] Server: Converts the voice data into text data using a speech recognition engine (e.g., IBM Watson or Google Cloud Speech-to-Text).

[2236] Server: Validates the accuracy of the speech recognition results and makes corrections if necessary.

[2237] Step 5:

[2238] Text data storage in a database

[2239] Input: Text data is generated.

[2240] Output: The text data is saved in the database.

[2241] Specific behavior:

[2242] Server: The converted text data is saved in a database.

[2243] Server: Verify that the data was saved correctly and generate a success message.

[2244] Step 6:

[2245] Search past claims data

[2246] Input: The user types a keyword into the app.

[2247] Output: The relevant claim record is displayed on the user's terminal.

[2248] Specific behavior:

[2249] User: Enters a keyword, such as "exchange," into the app and performs a search.

[2250] Server: Searches the database, extracts relevant past complaint handling records, and provides them to the user.

[2251] Step 7:

[2252] Check stock information

[2253] Input: The user types a stock check request into the app.

[2254] Output: The inventory information is displayed on the user's terminal.

[2255] Specific behavior:

[2256] User: Enters product name or barcode into the app to request availability.

[2257] Server: Retrieves inventory information from the database and provides it to users in real time.

[2258] Device: Display inventory information on the user's device and communicate it to the customer.

[2259] Step 8:

[2260] Sharing customer service know-how

[2261] Input: The user types a specific keyword into the app.

[2262] Output: Related response know-how is displayed on the user's terminal.

[2263] Specific behavior:

[2264] User: Enters a keyword into the app, such as "upsell," and performs a search.

[2265] Server: Extracts relevant phrases and techniques from the know-how database and provides them to users.

[2266] User: Review the information provided and put it into practice in customer service.

[2267] (Application example 1)

[2268] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2269] Conventional factory management systems rely on manual processes for managing work orders, checking inventory, handling complaints, and sharing work know-how, which can be inefficient and prone to errors. Performing these tasks efficiently is particularly difficult in large factories. This has led to a demand for automating various tasks within factories, processing data in real time, and improving work efficiency.

[2270] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2271] In this invention, the server includes means for connecting an audio device to a user's mobile device, means for recording conversations between the user and customers via the audio device, means for transmitting the recorded audio data to the server in real time, means for converting the audio data to text data in the server, means for storing the converted text data in a database, means for searching related past data from the database, means for providing the search results to the user, means for inputting and managing work instructions by voice, means for checking inventory information in real time, means for recording and searching complaint responses, and means for classifying and sharing excellent work records, which enables improved work efficiency in factories, more accurate inventory management, faster complaint responses, and the sharing of work know-how.

[2272] "Audio Device" means a device that collects audio and is connected to a User's Mobile Device.

[2273] A "mobile device" is a portable information terminal such as a smartphone or tablet.

[2274] A "recording means" is a method or apparatus for storing audio data collected via an audio device.

[2275] "Means for transmitting in real time" refers to a method or device for instantly transmitting collected voice data to a server.

[2276] A "server" is a central processing unit that processes and stores information on a network.

[2277] The "means for converting into text data" refers to a method or device that has the function of converting voice data into text information.

[2278] A "database" is a system for centrally storing and managing information.

[2279] A "searching means" is a method or device for efficiently locating specific information within a database.

[2280] "Means of providing" refers to the method or device for displaying search results and data to users.

[2281] "Means for managing work instructions" refers to a method or device for organizing and saving work instructions entered by voice so that they can be referenced as needed.

[2282] "Means for checking inventory information" refers to a method or device that allows you to grasp the inventory status of a product in real time.

[2283] "Means for recording complaint handling" refers to a method or device for recording and saving customer complaints and requests as audio data.

[2284] "Means for classifying and sharing excellent work records" refers to methods and devices for collecting effective work methods and know-how, organizing and storing them, and sharing them with other users.

[2285] As an embodiment of the present invention, a method for constructing a factory management system will be described below.

[2286] 1. Basic system configuration

[2287] Users (factory workers): Use this system to input and manage work instructions by voice, check inventory, handle complaints, and share excellent work know-how.

[2288] Audio device: A device worn by the user that collects audio such as instructions and complaints within the factory.

[2289] Mobile device (smartphone or tablet): Temporarily stores audio data received from the audio device and sends it to the server.

[2290] Server: Converts received voice data into text and stores it in a database. It also manages inventory information and provides information upon user request. It also collects and shares excellent work records.

[2291] 2. System Operation Flow

[2292] 1. Connecting the user to the audio device

[2293] Device: The user powers on the audio device and launches the app on their mobile device, which pairs with the audio device via Bluetooth.

[2294] Server: If pairing is successful, the server receives the device ID, checks the device registration status, and approves the connection.

[2295] 2. Collection and transmission of voice data

[2296] Device: When the user begins to speak work instructions, the audio device collects the voice and transmits the collected voice data to the mobile device in real time.

[2297] Device: The mobile device temporarily stores the collected voice data and transfers it to the server in batches.

[2298] 3. Audio Data Processing

[2299] Server: Receives the voice data and converts it into text data using a speech recognition engine (e.g., Google Cloud Speech-to-Text).

[2300] Server: The converted text data is stored in a database (e.g. MySQL).

[2301] 4. Search for past claims data

[2302] Users: Use the search function in the app when handling a complaint by entering keywords related to previous complaints.

[2303] Server: Searches for relevant historical data in the database and provides it to the user.

[2304] 5. Check inventory information

[2305] User: Enter the product name or barcode by voice.

[2306] Server: Retrieves inventory information from the database and provides it to users in real time.

[2307] Terminal: Displays inventory information to the user and communicates it to the customer.

[2308] 6. Share good work records

[2309] Server: Continuously collects voice data of excellent work and stores it in a know-how database.

[2310] Users: When they want to learn about a specific task or know-how in the app, they select a category or enter keywords.

[2311] Server: Searches for relevant information from the know-how database and provides it to the user.

[2312] Specific examples

[2313] Example 1: Entering and managing work orders

[2314] User: Speak instructions such as "Assemble 3 units of part A."

[2315] Server: Converts the received voice data into text and adds the instruction "Assemble 3 units of part A" to the work list.

[2316] Example 2: Checking inventory

[2317] User: Asks aloud, "What is the inventory of part A, barcode 1234?"

[2318] Server: Searches the inventory database and sends the result, "There are 5 units of part A with barcode 1234 in stock."

[2319] Example 3: Recording complaints

[2320] User: Speaks a complaint: "Machine X is not working properly."

[2321] Server: Converts the received voice data into text and stores it in the claim database.

[2322] User: Search for past complaint history regarding "Machine X".

[2323] Prompt Sentence Examples

[2324] (Voice input): "What is the current inventory?"

[2325] (Generated text): "Checking current inventory. Please wait."

[2326] In this way, the efficiency of work and the quality of service in the factory management system are improved.

[2327] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2328] Step 1:

[2329] Connecting users and audio devices

[2330] User: Power on the audio device and launch the app on their smartphone.

[2331] Input: User interaction to open the audio device and launch the app.

[2332] Output: The audio device and smartphone are paired.

[2333] Device: The app pairs with your audio device via Bluetooth.

[2334] Server: If pairing is successful, the server receives the device ID, checks the device's registration status, and approves the connection.

[2335] Step 2:

[2336] Collection and transmission of voice data

[2337] User: Start a conversation with customers or other workers.

[2338] Input: The user's voice conversation.

[2339] Output: Audio device collects audio data.

[2340] Device: The audio device sends audio data to the smartphone in real time.

[2341] Input: Audio data from the audio device.

[2342] Output: Audio data temporarily stored on your smartphone.

[2343] Terminal: The temporarily saved audio data is transferred to the server in batch processing.

[2344] Step 3:

[2345] Audio data processing

[2346] Server: Receives the voice data and analyzes it using a voice recognition engine.

[2347] Input: Audio data sent from the smartphone.

[2348] Output: Parsed text data.

[2349] Server: Converts voice data into text data using a speech recognition engine (e.g., Google Cloud Speech-to-Text).

[2350] Input: Speech data to the speech recognition engine.

[2351] Output: The content converted to text data.

[2352] Server: The converted text data is stored in a database (e.g. MySQL).

[2353] Step 4:

[2354] Search past claims data

[2355] User: When handling a complaint, use the search function in the app and enter keywords.

[2356] Input: Claim response keywords.

[2357] Output: Search results for historical claims data.

[2358] Server: Searches for relevant historical data in a database.

[2359] Input: A query search to the database.

[2360] Output: Claim data as search results.

[2361] Server: Serves search results to users.

[2362] Step 5:

[2363] Check stock information

[2364] User: Enter the product name or barcode by voice.

[2365] Input: Product name or barcode.

[2366] Output: Inventory information retrieved from the database.

[2367] Server: Retrieves inventory information from the database and provides it to users in real time.

[2368] Input: Inventory information from the database.

[2369] Output: Inventory information provided to the user.

[2370] Terminal: Displays inventory information to the user and communicates it to the customer.

[2371] Step 6:

[2372] Sharing excellent work records

[2373] Server: Continuously collects voice data of excellent work and stores it in a know-how database.

[2374] Input: Excellent work audio data.

[2375] Output: Text data stored in the know-how database.

[2376] Users: When they want to learn about a specific task or know-how in the app, they select a category or enter keywords.

[2377] Input: Keywords related to know-how.

[2378] Output: Relevant know-how information.

[2379] Server: Searches for relevant information from the know-how database and provides it to the user.

[2380] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2381] As a form for implementing the present invention, a system is constructed that combines an audio device, a mobile device, a server, and an emotion engine to improve the efficiency of customer service in stores, handle complaints, check inventory, and share customer service know-how.

[2382] 1. Basic system configuration

[2383] User (store staff): This system records conversations with customers, manages complaints, checks inventory, and shares customer service know-how. It also uses an emotion engine to understand emotional states in real time.

[2384] Audio device: A device worn by the user that captures conversations with customers.

[2385] Mobile device (smartphone or tablet): Responsible for receiving voice data from the audio device and sending it to the server and emotion engine.

[2386] Server: Converts received voice data into text and stores it in a database. It also manages inventory information and provides information upon user request. It also collects and shares records of outstanding employees' interactions.

[2387] Emotion Engine: Analyzes emotions from user and customer voice data to determine their emotional state in real time.

[2388] 2. System Operation Flow

[2389] 1. Connecting the user to the audio device

[2390] Device: The user powers on the audio device and launches the app on their mobile device, which pairs with the audio device via Bluetooth.

[2391] Server: If pairing is successful, the server receives the device ID, checks the device registration status, and approves the connection.

[2392] 2. Collection and transmission of voice data

[2393] Device: When a user initiates a customer interaction, the audio device collects the voice and transmits the collected voice data to the mobile device in real time.

[2394] Terminal: The mobile device temporarily stores the voice data and transfers it to the server and emotion engine in batch processing.

[2395] 3. Audio Data Processing

[2396] Server: Receives the voice data and converts it into text using a speech recognition engine.

[2397] Server: The converted text data is saved in a database.

[2398] 4. Emotional Data Processing

[2399] Emotion Engine: Analyzes emotional state from voice data and notifies the user in real time.

[2400] 5. Search for past claims data

[2401] Users: Use the search function in the app when handling a complaint by entering keywords related to previous complaints.

[2402] Server: Searches for relevant historical data in the database and provides it to the user.

[2403] 6. Check inventory information

[2404] User: When a customer requests an inventory check, they enter the product name and barcode through the app.

[2405] Server: Retrieves inventory information from the database and provides it to users in real time.

[2406] Terminal: Displays inventory information to the user and communicates it to the customer.

[2407] 7. Sharing customer service know-how

[2408] Server: Continuously collects and stores records of excellent employees' responses in a know-how database.

[2409] Users: When they want to learn about a specific situation or how to respond in the app, they can select a category or enter keywords.

[2410] Server: Searches for relevant information from the know-how database and provides it to the user.

[2411] Specific examples

[2412] Example 1: Handling complaints

[2413] If a customer complains that "the response to the previous product exchange was slow,"

[2414] User: Enters "product exchange" and performs a search.

[2415] Server: Searches past conversation records related to "product exchange" and provides three results to the user's device.

[2416] Users: Check past responses and use them as a reference for responding quickly.

[2417] Emotion Engine: It also analyzes customer emotions and advises users on appropriate actions before the issue escalates.

[2418] Example 2: Checking inventory

[2419] If a customer asks, "Do you have this T-shirt in a medium size?"

[2420] User: Scans the product barcode to request an inventory check.

[2421] Server: Searches the inventory database, finds that there are 5 units of the item in stock, and sends the result to the user.

[2422] User: Tell customer "In stock."

[2423] Example 3: Sharing customer service know-how

[2424] If new staff want to learn "effective upselling methods,"

[2425] User: Searches for "upsell" in the app.

[2426] Server: Extracts effective phrases and techniques related to "upselling" from a know-how database and provides them to users.

[2427] User: Review the information provided and apply it to actual customer service.

[2428] Emotion engine: Analyzes the user's emotions and suggests relaxation methods if stress is building up.

[2429] As described above, the embodiment of the present invention realizes efficient customer service and improved service quality in stores through the specific operating procedures and processing flows. In addition, by combining it with an emotion engine, it becomes possible to grasp the user's emotional state in real time and provide more appropriate service.

[2430] The processing flow will be explained below.

[2431] Specific processing flow of the system

[2432] 1. Connecting the user to the audio device

[2433] Step 1:

[2434] User: Power on the audio device.

[2435] Step 2:

[2436] Device: Turn on Bluetooth on your smartphone and launch the dedicated app.

[2437] Step 3:

[2438] On your device: The app performs a Bluetooth scan to discover nearby audio devices.

[2439] Step 4:

[2440] Device: Displays a list of detected audio devices and lets the user select a device.

[2441] Step 5:

[2442] User: Approves to pair the selected device.

[2443] Step 6:

[2444] On the device: Notify the user that pairing was successful.

[2445] Step 7:

[2446] Server: Receives the device ID from the mobile device and checks the device registration status.

[2447] Step 8:

[2448] Server: Once registration is confirmed, approve the connection and send a notification to the device that the connection is complete.

[2449] 2. Collection and transmission of voice data

[2450] Step 1:

[2451] User: Presses the start recording button in the app.

[2452] Step 2:

[2453] Device: The audio device collects the conversation and transmits the voice data to the mobile device in real time.

[2454] Step 3:

[2455] Device: Your mobile device temporarily stores the audio data.

[2456] Step 4:

[2457] Terminal: Uploads audio data to the server periodically in batches.

[2458] 3. Audio Data Processing

[2459] Step 1:

[2460] Server: Receives the audio data.

[2461] Step 2:

[2462] Server: The speech recognition engine converts the voice data into text.

[2463] Step 3:

[2464] Server: The converted text data is saved in a database.

[2465] 4. Emotional Data Processing

[2466] Step 1:

[2467] Emotion Engine: Receives voice data.

[2468] Step 2:

[2469] Emotion engine: Analyzes emotional states from voice data.

[2470] Step 3:

[2471] Emotion Engine: Notifies users of their analyzed emotional state in real time.

[2472] 5. Search for past claims data

[2473] Step 1:

[2474] User: When a complaint occurs, use the complaint search feature in the app.

[2475] Step 2:

[2476] User: Enter keywords related to the complaint in the search box.

[2477] Step 3:

[2478] Device: The app sends a search request to the server.

[2479] Step 4:

[2480] Server: Searches the database for relevant past conversation records.

[2481] Step 5:

[2482] Server: Returns search results to the user's device.

[2483] Step 6:

[2484] Device: Display search results to the user in a list format.

[2485] 6. Check inventory information

[2486] Step 1:

[2487] User: Receives a request from a customer to check inventory.

[2488] Step 2:

[2489] User: Use the in-app inventory check feature to enter the product name or barcode.

[2490] Step 3:

[2491] Terminal: Sends an inventory check request to the server.

[2492] Step 4:

[2493] Server: Retrieves product inventory information from the database.

[2494] Step 5:

[2495] Server: Returns inventory information to the user's device.

[2496] Step 6:

[2497] Terminal: Display inventory information to the user.

[2498] 7. Sharing customer service know-how

[2499] Step 1:

[2500] Server: Continuously collects and stores records of excellent employees' responses in a know-how database.

[2501] Step 2:

[2502] Users: When they want to learn about a specific situation or how to respond in the app, they can select a category or enter keywords.

[2503] Step 3:

[2504] Terminal: Sends a know-how search request to the server.

[2505] Step 4:

[2506] Server: Searches for relevant information from a know-how database.

[2507] Step 5:

[2508] Server: Sends search results to the user's device.

[2509] Step 6:

[2510] Terminal: Displays the provided know-how information to the user.

[2511] With the above processing flow, the system of the present invention can improve the efficiency of customer service, handle complaints, check inventory, and share customer service know-how. In addition, by combining it with an emotion engine, it is possible to grasp the user's emotional state in real time and respond more appropriately.

[2512] Example 2

[2513] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2514] Conventional customer service systems have difficulty efficiently recording conversations between customers and users (store staff) and processing them in real time. Furthermore, systems that can simultaneously support multiple tasks, such as handling complaints, checking inventory, and sharing customer service know-how, are limited. Furthermore, they lack the functionality to instantly grasp the emotional state of customers and optimize responses, resulting in problems such as stress for users and a decline in the quality of customer service. An effective system that can solve these issues is needed.

[2515] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2516] In this invention, the server includes means for connecting an audio device to a user's mobile terminal, means for recording conversations between a user and a customer via the audio device, means for transmitting the recorded voice data to an information processing device in real time, means for converting the voice data into text data in the information processing device, means for saving the converted text data in a storage device, means for searching for related past data from the storage device, means for providing the search results to the user, means for analyzing the emotional state of the customer from the voice data and notifying the user in real time, means for acquiring and providing inventory information in response to a user's request, and means for collecting and sharing records of responses from excellent employees. This makes it possible to improve the efficiency of customer service, speed up complaint responses, improve the accuracy of inventory checks, share customer service know-how, and also to grasp the emotional state of customers in real time and respond appropriately.

[2517] An "audio device" is a device that collects audio and transmits that data to an external device.

[2518] "Users" refer to store staff and service personnel who use this system to provide customer service.

[2519] "Mobile terminal" refers to a small, portable information processing device such as a smartphone or tablet.

[2520] "Customer" refers to a consumer who uses a store or service.

[2521] The term "mobile device" is synonymous with a portable terminal and refers to an information processing device that is carried and used by a user.

[2522] "Information processing device" refers to a computer system for processing voice data and text data.

[2523] "Recording means" refers to devices or software capable of collecting and storing audio data.

[2524] "Voice data" refers to data that represents voice information, including the content of conversations between users and customers, in digital form.

[2525] "Text data" refers to digital data that has been converted from voice data into text information.

[2526] "Storage device" refers to hardware and software for storing data for long periods of time.

[2527] "Means for storing data in a storage device" refers to the functions and processes for storing data in a storage device.

[2528] "Means for searching past data" refers to the function or process of searching for information that matches specific conditions from data stored in a storage device.

[2529] "Means of providing search results to users" refers to the function of displaying or providing the information obtained through a search in a format that is easy for users to view.

[2530] "Means for analyzing emotional state and notifying in real time" refers to the functions and processes for determining emotions from voice data and immediately conveying that information to the user.

[2531] "Means for obtaining and providing inventory information" refers to the function of retrieving inventory information from a database or other storage device and providing it to the user.

[2532] "Means for collecting and sharing records of excellent employee interactions" refers to the functions and processes for collecting records of excellent employee interactions with customers and sharing them with other employees.

[2533] As an embodiment of the present invention, a system is constructed that combines an audio device, a mobile device, a server, and an emotion engine to improve the efficiency of customer service in stores, handle complaints, check inventory, and share customer service know-how. The system aims to collect, analyze, save, and provide conversations between customers and store staff in real time.

[2534] Basic system configuration

[2535] User (store staff)

[2536] Users can use the system to record conversations with customers, manage complaints, check inventory, share customer service know-how, and use an emotion engine to understand the emotional state of their customers and themselves in real time.

[2537] Audio Devices

[2538] The audio device is a user-worn device that collects conversations with customers and connects to a mobile device via Bluetooth.

[2539] Mobile device (smartphone or tablet)

[2540] The mobile device receives voice data from the audio device and sends it to the server and emotion engine, and some voice data is temporarily stored.

[2541] server

[2542] The server converts the received voice data into text and stores it in a database. It also manages inventory information and provides information upon user request. It also has the capacity to collect and share records of excellent employees' interactions.

[2543] Emotion Engine

[2544] The emotion engine analyzes customer and user emotions from voice data and determines their emotional state in real time.

[2545] Specific examples

[2546] Example 1: Handling complaints

[2547] If a customer complains that "the response to the previous product exchange was slow," the system operates as follows:

[2548] User: Enters "product exchange" into the app and performs a search.

[2549] Server: Searches the database for conversation records related to past "product exchanges" and provides the results to the user's mobile device.

[2550] Users: Check past responses and use them as a reference for responding quickly.

[2551] Emotion Engine: Analyzes customer emotions and advises users on appropriate actions before the issue escalates.

[2552] Example 2: Checking inventory

[2553] When a customer asks, "Do you have this T-shirt in size medium?", the system works as follows:

[2554] User: Scans the product barcode to request an inventory check.

[2555] Server: Searches the inventory database, finds that there are 5 units of the item in stock, and sends the result to the user.

[2556] User: Tell customer "In stock."

[2557] Example 3: Sharing customer service know-how

[2558] If a new staff member wants to learn "effective upselling methods," the system works as follows:

[2559] User: Searches for "upsell" in the app.

[2560] Server: Extracts effective phrases and techniques related to "upselling" from a know-how database and provides them to users.

[2561] User: Review the information provided and apply it to actual customer service.

[2562] Emotion engine: Analyzes the user's emotions and suggests relaxation methods if stress is building up.

[2563] In this way, the present invention achieves more efficient customer service and improved service quality in stores through specific operational procedures and processing flows. In addition, by combining it with an emotion engine, it is possible to grasp the user's emotional state in real time and provide more appropriate responses.

[2564] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2565] Step 1:

[2566] Connecting users and audio devices

[2567] Input: The user turns on the audio device and launches a dedicated app on their mobile device.

[2568] Specific operation: The user operates the Bluetooth settings of the mobile device to pair with the audio device. The dedicated app on the mobile device obtains the audio device ID and sends it to the server.

[2569] Data processing: The server checks the received device ID in the database to determine whether it is a registered device. If successful, the server returns a connection acknowledgement message.

[2570] Output: A "Connection successful" message will appear on the mobile device screen.

[2571] Step 2:

[2572] Collection and transmission of voice data

[2573] Input: The user initiates a conversation and the audio device collects the audio.

[2574] What it does: The audio device collects the conversation with the customer and transmits it in real time to the mobile device, which temporarily stores the audio data.

[2575] Data processing: Once a certain amount of voice data has been accumulated, it is transferred to the server and emotion engine in a batch process.

[2576] Output: The server and emotion engine receive the audio data.

[2577] Step 3:

[2578] Audio data processing

[2579] Input: Audio data sent from a mobile device.

[2580] Specific operation: The server converts the voice data into text data using a speech recognition engine such as the Google Cloud Speech-to-Text API.

[2581] Data processing: The converted text data is added with time series and related information (customer ID, date and time, etc.) and stored in a database.

[2582] Output: The saved text data.

[2583] Step 4:

[2584] Emotional Data Processing

[2585] Input: Audio data sent from a mobile device.

[2586] What it does: The emotion engine analyzes voice data to identify the emotional state of customers and users.

[2587] Data processing: The emotion engine determines emotions from the tone, speed, and content of the voice, and analyzes the results in real time.

[2588] Output: Notifications about the emotional state of the customer and user are displayed in a dedicated app on the mobile device.

[2589] Step 5:

[2590] Handling complaints

[2591] Input: User searches for keywords related to the claim in the app.

[2592] How it works: A user enters a keyword, such as "product exchange," into the app, and the server searches the database to extract relevant past complaint data.

[2593] Data processing: The data extracted as search results is sorted in chronological order and provided to the user along with related information.

[2594] Output: The app displays the past complaint handling records.

[2595] Step 6:

[2596] Check stock information

[2597] Input: The user enters the product name or barcode into a dedicated app.

[2598] Specific operation: The user scans an item with a barcode reader or enters the item name. The server searches the inventory database to obtain the stock information for that item.

[2599] Data processing: Inventory information is sent to the user's device in real time.

[2600] Output: Inventory information is displayed on the user's mobile device, and the user communicates stock availability to the customer.

[2601] Step 7:

[2602] Sharing customer service know-how

[2603] Input: A user searches the app for keywords related to a specific interaction.

[2604] Specific operation: The user enters a keyword such as "upsell" into the app. The server searches the know-how database and extracts records of related customer service know-how.

[2605] Data processing: The extracted know-how information is organized by category and provided to users.

[2606] Output: Know-how information is displayed on the app, and users can apply it to actual customer service.

[2607] As described above, this system collects, processes, and provides data according to user requests through each processing step, thereby improving the efficiency of customer service and service quality.

[2608] (Application example 2)

[2609] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2610] Real-time information collection and analysis is essential to improve the efficiency of customer service in stores, the speed of complaint handling, the speed of inventory checks, and the customer service skills of employees. However, conventional systems lack the means to comprehensively solve these issues, making it particularly difficult to incorporate emotion analysis into customer service, efficiently check inventory, and instantly share customer service know-how.

[2611] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for connecting an audio device to the user's mobile device, means for recording conversations between the user and customers via the audio device, means for transmitting the recorded voice data to the server in real time, means for converting the voice data into text data in the server, means for storing the converted text data in a database, means for searching for related past data from the database, means for providing the search results to the user, means for having the user wear smart glasses and collecting the conversation and emotional state through the glasses, means for analyzing the emotional state from the voice data using an emotion engine and notifying the user in real time, and means for displaying inventory information, complaint data, and customer service know-how on the smart glasses. This enables efficient customer service, appropriate responses based on emotion analysis, rapid inventory confirmation, and real-time sharing of customer service know-how.

[2612] An "audio device" is a device that records conversations between users and customers and transmits them as audio data to a mobile device.

[2613] "Mobile Device" means a portable electronic device that receives recorded audio data and transmits the data to a server or emotion engine.

[2614] A "server" is a central processing unit that converts voice data into text data and stores it in a database.

[2615] An "emotion engine" is an analysis device that analyzes emotions from voice data and determines emotional states in real time.

[2616] "Smart glasses" are devices worn by users that collect speech and emotional states and display information.

[2617] A "database" is a collection of information in which voice data is converted into text data and stored.

[2618] "Inventory information" is data that indicates the quantity and condition of a particular product currently in stock in the store.

[2619] "Complaint data" refers to recorded data regarding past complaints and requests from customers.

[2620] "Customer service know-how" is knowledge that allows employees to share records and techniques of excellent customer service.

[2621] "Real-time" refers to data and information being processed immediately, without delay.

[2622] A "search tool" is a method or device for finding relevant information within a database.

[2623] "Notification means" refers to a method or device for notifying the user of information such as the analyzed emotional state.

[2624] A "display" is a method or apparatus for visually presenting information to smart glasses or other devices.

[2625] This invention is a system for improving the efficiency of customer service in stores, handling complaints quickly, checking inventory, and sharing customer service know-how. This system combines audio devices, smart glasses, mobile devices, a server, and an emotion engine.

[2626] Basic system configuration

[2627] User (store staff):

[2628] The user wears smart glasses and uses a mobile device to collect conversations with customers, handle complaints, check inventory, and learn customer service know-how. The emotion engine allows the user to understand the emotional state of both the customer and the user in real time.

[2629] Audio Device:

[2630] A microphone built into smart glasses that collects customer conversations and transmits them to a mobile device.

[2631] Smart Glasses:

[2632] This device collects conversations with customers and their emotional states and displays the information, displaying customer emotion analysis results, inventory information, complaint handling data, and customer service know-how in real time.

[2633] Mobile devices:

[2634] It works in conjunction with smart glasses to send voice data to a server in real time, and also receives converted text data, emotion analysis results, and past data, and sends them to the smart glasses.

[2635] server:

[2636] The system converts voice data into text data and stores it in a database. It also uses an emotion engine to analyze emotions from the voice data and notify users in real time. It also manages complaint histories and inventory information.

[2637] Emotion Engine:

[2638] It analyzes emotions from voice data and determines the user's emotional state in real time. This information is sent to the smart glasses, which then advise the user on how to respond appropriately.

[2639] Hardware and software used

[2640] Hardware:

[2641] Smart glasses (e.g., Google Glass): devices for collecting speech and emotional state and displaying information.

[2642] Mobile devices (e.g. smartphones and tablets): process the voice data and send it to the server.

[2643] software:

[2644] Speech recognition engine (e.g. Google Speech-to-Text API): Converts voice data into text.

[2645] Sentiment analysis engine (e.g., IBM Watson Tone Analyzer): Analyzes customer and user sentiment.

[2646] Database (e.g. MySQL): Manages claims, inventory, and know-how.

[2647] System operation example

[2648] 1. Stock Check Example:

[2649] A customer asks, "Do you have this shoe in size 27.5cm?"

[2650] Users use smart glasses to scan product names or barcodes.

[2651] The server searches the inventory database.

[2652] The smart glasses will display "in stock" and the user can then direct it to the customer.

[2653] Example prompt sentence:

[2654] "Check if this item is in stock"

[2655] Customer: "Do you have these shoes in size 27.5cm?"

[2656] User: Scan product name or barcode with smart glasses

[2657] System: "Checking inventory..."

[2658] System: Shows "In stock"

[2659] 2. Examples of complaint handling:

[2660] The customer complains that the previous replacement was too slow.

[2661] The user types "exchange" into the smart glasses and searches.

[2662] The server searches past "exchange" claim data.

[2663] Past response results are displayed on the smart glasses, allowing users to respond quickly based on them.

[2664] Example prompt sentence:

[2665] "Check past complaint handling"

[2666] Customer: "The last exchange was slow."

[2667] User: Searches for "exchange" on smart glasses

[2668] System: Display "View past complaint responses"

[2669] In this way, the system will enable more efficient customer service, appropriate responses based on sentiment analysis, quick inventory checks, and real-time sharing of customer service know-how.

[2670] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2671] Step 1:

[2672] The user wears the smart glasses and connects the audio device to the mobile device. Specifically, the user turns on the smart glasses, launches the app on the mobile device, and pairs with the audio device via Bluetooth. The input is the user's operation, and the output is the connection status between the devices.

[2673] Step 2:

[2674] When a user starts a customer interaction, the audio device collects the voice. The input is the conversation between the customer and the user, and the output is the collected voice data. Specifically, the microphone of the audio device records the conversation and transmits it to the mobile device in real time.

[2675] Step 3:

[2676] The mobile device temporarily stores the voice data and transfers it to the server and emotion engine in batch processing. Specifically, once a certain amount of voice data has been collected, it is automatically sent to the server and emotion engine. The input is the collected voice data, and the output is the transmitted voice data.

[2677] Step 4:

[2678] The server receives the voice data and converts it into text data using a speech recognition engine. Specifically, speech recognition software such as the Google Speech-to-Text API is used. The input is voice data and the output is text data.

[2679] Step 5:

[2680] The server stores the text data in a database. Specifically, database software such as MySQL is used to store the data in an appropriate format. The input is text data, and the output is the stored text data.

[2681] Step 6:

[2682] The emotion engine analyzes the emotional state from the voice data and notifies the user in real time. Specifically, emotion analysis software such as IBM Watson Tone Analyzer is used. The input is the voice data, and the output is the analysis result of the emotional state.

[2683] Step 7:

[2684] Users can search for past complaint data or inventory information as needed. Specifically, they enter specific keywords into a mobile device app and execute a search. The input is the search keyword, and the output is the search results for related data.

[2685] Step 8:

[2686] The server searches the database for relevant historical data and provides it to the user. Specifically, it searches the database using an SQL query and sends the results to the mobile device. The input is the search query, and the output is the search result data.

[2687] Step 9:

[2688] The server retrieves inventory information in real time and displays it on the smart glasses. Specifically, it retrieves the latest information from the inventory database and displays the results on the user's smart glasses. The input is an inventory check request, and the output is inventory information.

[2689] Step 10:

[2690] The server collects the customer service records of outstanding employees and stores their customer service know-how in a database. Specifically, it analyzes daily customer service data, extracts effective customer service techniques, and adds them to the database. The input is customer service data, and the output is the analyzed customer service know-how.

[2691] Step 11:

[2692] The user checks customer service know-how using the smart glasses and applies it to actual customer service. Specifically, the user queries know-how for a specific situation and responds based on the information displayed on the smart glasses. The input is a know-how query request, and the output is the displayed know-how information.

[2693] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2694] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2695] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2696] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2697] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2698] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2699] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2700] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2701] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2702] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2703] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2704] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2705] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2706] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2707] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[2708] The hardware resource for executing a specific process can be any of the following types of processors: A processor, for example, is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[2709] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[2710] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the func...

Claims

1. a means for connecting the audio device with a user's mobile device; means for recording a conversation between a user and a customer via said audio device; means for transmitting the recorded voice data to a server in real time; means for converting voice data into text data in the server; A means for storing the converted text data in a database; means for retrieving relevant historical data from said database; A system that includes a means for providing search results to a user.

2. 10. The system according to claim 1, further comprising means for obtaining and providing inventory information in real time in response to a user request.

3. 10. The system of claim 1, further comprising means for collecting and sharing records of excellent employees' responses.

Images