System

Abstract

A system is provided.SOLUTION: A system comprising: means for collecting conversation data; means for transcribing the collected conversation data; means for summarizing the transcribed text data; means for extracting important information from the summarized text data; means for automatically entering the extracted information into a business management tool; means for analyzing and visualizing business activity data; and means for notifying a user of updated information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In sales activities, there is a lot of work involved in manually recording conversations with customers and entering them into sales management tools, which is time-consuming. In addition, when dealing with multiple customers at the same time, there is a risk of missing or making mistakes in entering information. This reduces the work efficiency of sales representatives, and often leads to them leaving work late. Therefore, in order to allow sales representatives to concentrate on conversations with customers, there is a need for a system that automatically processes conversation content and automatically enters it into sales management tools. [Means for solving the problem]

[0005] The present invention is a system that includes means for collecting conversation data, means for transcribing the collected conversation data, means for summarizing the transcribed text data, means for extracting important information from the summarized text data, means for automatically inputting the extracted information into a sales management tool, means for analyzing and visualizing sales activity data, and means for notifying users of updated information. This allows sales representatives to concentrate on the conversation while automating the recording of conversation content and inputting it into the sales management tool, thereby improving work efficiency.

[0006] "Conversation data" is information that includes the content of voice or text communications between a sales representative and a customer.

[0007] "Collection means" refers to devices or software used to capture and store conversation data.

[0008] "Transcribing means" refers to technology or equipment for converting audio data into text data.

[0009] "Summarizing means" refers to a technique or device that extracts the main points from long text data and presents them in a shortened form.

[0010] "Extraction means" refers to a technique or device that identifies and extracts predefined key information (e.g., customer names, dates, action items, etc.) from the summarized text data.

[0011] "Sales management tools" refers to software systems (e.g., CRM systems) that are used to efficiently manage and track sales activities.

[0012] "Means for analysis and visualization" refers to techniques or devices that statistically analyze collected data and display it in an easy-to-understand format (e.g., graphs, charts).

[0013] "Means for notifying" refers to a technique or device for notifying new or updated information from the system to the user terminal.

[0014] A "system" refers to a collection of devices and software that combine multiple elements to achieve a specific function. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] Step 1:

[0037] User

[0038] Users initiate conversations with customers, whether by phone, online conference, or face-to-face.

[0039] Step 2:

[0040] Terminal

[0041] A dedicated application installed on the user's device captures the audio data of the conversation in real time and sends it to the server.

[0042] Step 3:

[0043] server

[0044] The server sends the received audio data to a transcription engine, which analyzes the audio data and generates corresponding text data.

[0045] Step 4:

[0046] server

[0047] The server passes the generated text data to a natural language processing (NLP) module to generate a summary of the conversation.

[0048] Step 5:

[0049] server

[0050] The server extracts important information (e.g., customer names, dates, action items) from the summarized text using entity recognition and relationship extraction algorithms.

[0051] Step 6:

[0052] server

[0053] The server automatically inputs the extracted information via the sales management tool's API, comparing it with existing customer data and updating any new or changed information.

[0054] Step 7:

[0055] server

[0056] The server analyzes all collected data and generates a dashboard to visualize sales activities.

[0057] Step 8:

[0058] server

[0059] The server prepares the generated dashboards and reports and prepares notifications for the users.

[0060] Step 9:

[0061] Terminal

[0062] The user terminal receives notifications from the server and displays automatically generated sales activity reports and important update information.

[0063] Step 10:

[0064] User

[0065] Users can view new information and reports through their devices and plan their next actions.

[0066] Example 1

[0067] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0068] There is a need to efficiently extract important information from voice data and automatically input it into data management tools such as sales management, thereby reducing the time and effort required for manual input and improving the efficiency and accuracy of sales activities. Furthermore, there is a need to enable sales representatives to make quick decisions through information visualization and real-time information notifications.

[0069] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0070] In this invention, the server includes means for collecting voice data, means for converting the voice data into text data, means for summarizing the voice data, means for extracting important information, means for automatically inputting the voice data into a data management tool, means for comparing the voice data with a database and updating the data with new information, means for analyzing and visualizing the data, and means for notifying the user. This makes it possible to quickly and accurately extract important information from the voice data and automatically input the information into the data management tool. Furthermore, data visualization and real-time notifications improve the efficiency of overall sales activities.

[0071] "Audio data" means digital or analog information relating to audio.

[0072] A "collection means" is a device or method for acquiring audio data and storing or transmitting it in a form that can be further processed.

[0073] "Means for converting into text data" refers to a device or software for converting voice data into text format.

[0074] A "summarizing means" is a device or method for extracting the main points or important information from long text data and creating a shortened version.

[0075] "Significant information" is a data element that is necessary or valuable in a particular context, such as a customer's name or a meeting date.

[0076] A "data management tool" is software or a system for storing, retrieving, organizing, and manipulating data. Examples include sales management tools.

[0077] An "automated input means" is a device or method for inputting extracted information into a data management tool without manual intervention.

[0078] "Means for checking against the database and updating with new information" means a device or method for comparing information in an existing database with newly acquired information and updating the database as necessary.

[0079] A "data analysis and visualization tool" is a device or method for analyzing and visually transforming data in order to display it in an understandable format (e.g., graphs or charts).

[0080] "Notification means" refers to a device or method for notifying a user of new or updated information.

[0081] MODE FOR CARRYING OUT THE INVENTION

[0082] This invention relates to a system that automatically records and processes conversations with customers, primarily during sales activities, and inputs them into a sales management tool. The system of the present invention is equipped with a series of means for collecting conversation data, transcribing it, summarizing it, extracting important information, and automatically inputting it into the sales management tool. It also includes a function for analyzing and visualizing sales activity data and notifying users of updated information.

[0083] System program processing

[0084] The following is a detailed description of how this system works.

[0085] User

[0086] A user initiates a conversation with a customer. The conversation can take place over the phone, via online conference, or in person. For example, if a user initiates a conversation using an online conference tool (e.g., a general video conference tool), the content of the conversation is captured in real time by a dedicated application installed on the user's device.

[0087] Terminal

[0088] The user device sends the captured audio data to the server. Specifically, a dedicated application compresses the audio data and sends it to the server using the HTTPS protocol. For example, the audio data is chunked every 10 seconds and sent continuously with minimal delay.

[0089] server

[0090] The server passes the received voice data to a transcription engine (e.g., a general cloud-based speech recognition API) and converts the voice data into text data. For example, the speech "Please schedule the next meeting for Monday" is converted into text "Please schedule the next meeting for Monday."

[0091] The server then passes the transcribed text data to a natural language processing (NLP) module (e.g., a popular NLP framework) for summarization. For example, a one-hour conversation might be summarized as "agreements on the next meeting date and time and customer needs."

[0092] The server then uses entity recognition and relationship extraction algorithms to extract important information from the summarized text (such as customer names, dates, and action items). For example, from the text "The next meeting with Taro Yamada is scheduled for Monday," the server extracts information such as "Taro Yamada," "next meeting," and "Monday."

[0093] The extracted information is automatically entered into the sales management tool (e.g., general sales management software) via its API. For example, information such as the customer name "Yamada Taro" and the next meeting date "Monday" is added to the database.

[0094] The server also analyzes the sales activity data and generates a dashboard (e.g., a common data visualization tool) to visualize it, for example, by displaying the number of customer meetings last week or a schedule for next week in graphs and tables.

[0095] Finally, the server notifies the user of the generated dashboard or report. For example, when the user opens a dedicated application, the user is notified to display the new information or report.

[0096] Specific examples

[0097] As a concrete example, consider the case where a user is holding an online conference with a client. When the user starts a conversation using a general video conferencing tool, a dedicated application captures the audio data in real time. For example, when the user says, "Please schedule the next meeting for Monday," the audio is captured.

[0098] The dedicated application sends the captured audio data to a server using the HTTPS protocol. The audio data is divided into chunks every 10 seconds and received by the server.

[0099] The server passes the audio data to a transcription engine, which converts it into text: "Please schedule our next meeting for Monday." The server then passes the text data to an NLP module, which generates a summary: "Next meeting: Monday." From that summary, it extracts key information, such as "meeting" and "Monday."

[0100] The server automatically inputs the extracted information using the sales management tool's API and updates the database. The server also generates a dashboard based on the sales activity data and displays a graph of the next week's meeting schedule.

[0101] Finally, the server notifies the user that a report of next week's meeting schedule is available, and the user opens the dedicated application to check the latest information.

[0102] Prompt Sentence Examples

[0103] Below is an example of a prompt sentence to input to the generative AI model.

[0104] After transcribing and summarizing your customer conversation, extract the following information:

[0105] 1. Customer name

[0106] 2. Meeting date and time

[0107] 3. Action Items

[0108] example:

[0109] The client's name is Taro Yamada, and our next meeting can be scheduled for next Monday."

[0110] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0111] Step 1: Capture conversation data

[0112] User

[0113] A user starts a conversation with a customer by telephone, online conference, or face-to-face. For example, when a user starts a conversation using a video conferencing tool, the conversation content is captured in real time by a dedicated application installed on the user's terminal. The input is voice data, and the output is the captured digitized voice data.

[0114] Step 2: Sending audio data

[0115] Terminal

[0116] The user device sends the captured audio data to the server. Specifically, a dedicated application compresses the audio data and sends it to the server using the HTTPS protocol. For example, the audio data is divided into chunks every 10 seconds and sent continuously with minimal delay. The input is the captured audio data, and the output is the audio data sent to the server.

[0117] Step 3: Transcribe the audio data

[0118] server

[0119] The server passes the received voice data to a transcription engine (for example, a general cloud-based voice recognition API) and converts the voice data into text data. For example, voice data such as "Please schedule the next meeting for Monday" is converted into text data such as "Please schedule the next meeting for Monday." The input is the voice data sent to the server, and the output is the transcribed text data.

[0120] Step 4: Summarizing the text data

[0121] server

[0122] The server passes the transcribed text data to a natural language processing (NLP) module (e.g., a general NLP framework) for summarization. For example, a one-hour conversation is summarized as "agreement on the date and time of the next meeting and the customer's requests." The input is the transcribed text data, and the output is the summarized text data.

[0123] Step 5: Extracting important information

[0124] server

[0125] The server uses entity recognition and relationship extraction algorithms to extract important information (such as customer names, dates, and action items) from the summarized text. For example, from the text "The next meeting with Yamada Taro is scheduled for Monday," the server extracts information such as "Yamada Taro," "next meeting," and "Monday." The input is the summarized text data, and the output is the extracted important information.

[0126] Step 6: Enter information into your sales management tool

[0127] server

[0128] The server automatically inputs the extracted important information through the API of a sales management tool (e.g., general sales management software). For example, information such as "Customer name: Yamada Taro, next meeting: Monday" is added to the database. The input is the extracted important information, and the output is the information entered into the sales management tool.

[0129] Step 7: Analyze and visualize the data

[0130] server

[0131] The server analyzes the sales activity data and generates a dashboard (e.g., a common data visualization tool) to visualize it. For example, it displays the number of customer meetings last week or the schedule for next week in graphs and tables. The input is the sales activity data, and the output is the generated dashboard.

[0132] Step 8: Report Notification

[0133] server

[0134] The server notifies the user terminal of the generated dashboard or report. For example, when the user opens a dedicated application, the server notifies the user to display new information or reports. The input is the generated dashboard or report, and the output is the notified information.

[0135] (Application example 1)

[0136] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0137] When interacting with customers in a virtual store, it is necessary to efficiently respond to customer questions and requests, and to automatically record, process, and manage the content of the conversation. Currently, these tasks are often done manually, which is time-consuming and labor-intensive, so there is an urgent need to create an efficient system.

[0138] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0139] In this invention, the server includes means for collecting conversation data, means for transcribing the collected conversation data, means for summarizing the transcribed text data, means for extracting important information from the summarized text data, means for automatically inputting the extracted information into a customer management system, means for analyzing and visualizing customer interaction data, and means for notifying users of updated information. This makes it possible to automate everything from recording to managing customer interactions, thereby improving the efficiency and accuracy of customer interactions.

[0140] "Conversation Data" means records of voice or text communications with customers.

[0141] "Means of collection" refers to the equipment or software that captures the voice and text data necessary to interact with customers.

[0142] "Transcription method" refers to the process or technique for converting audio data into text data.

[0143] "Summarization methods" refers to techniques and processes for extracting key information from transcribed text data and summarizing it concisely.

[0144] "Key information extraction methods" are techniques and methods for identifying and extracting specific entities or action items from summarized text data.

[0145] A "customer management system" refers to software or a platform for centrally managing customer information and transaction history.

[0146] "Means for automatic input" refers to methods or technologies for mechanically registering extracted information into a customer management system rather than manually.

[0147] "Analysis and visualization means" refers to techniques and methods for analyzing collected data and displaying it in a visual format such as graphs or charts.

[0148] "Means for notifying users of updated information" refers to techniques and methods for notifying users when data is updated or new information is obtained.

[0149] A "virtual store" is a virtual shopping environment or service provided over the Internet.

[0150] A "virtual assistant" is a character or bot equipped with artificial intelligence that interacts with customers and provides services within a virtual store.

[0151] This invention relates to a system for automating and efficiently managing customer interactions in a virtual store. The system of the present invention is equipped with a series of means for collecting and recording customer conversations, transcribing them, summarizing them, extracting important information, and automatically inputting them into a customer management system. It also includes a function for analyzing and visualizing the collected customer interaction data and notifying users of updated information.

[0152] System program processing

[0153] This system is realized using the following hardware and software.

[0154] Hardware:

[0155] User devices for customer service (e.g., smartphones and desktop PCs)

[0156] Microphone (for voice input)

[0157] Server (for data processing and management)

[0158] software:

[0159] speech_recognition library: Captures and transcribes speech data.

[0160] The transformers library: performs natural language processing (NLP).

[0161] Virtual CRM API module: Automatically registers extracted important information into a customer management system.

[0162] The server receives customer conversation data from the user's device and converts the speech to text in real time using a transcription engine. This text data is temporarily stored and then summarized using a natural language processing (NLP) module. From the summarized text, the server uses entity recognition and relationship extraction algorithms to extract important information (e.g., customer name, date, action items, etc.).

[0163] The extracted information is automatically entered into the customer management system via API. At this time, it is compared with the existing database, and if there is new information or changes, the database is updated. The server also analyzes the customer interaction data and visualizes it as a dashboard. This dashboard is provided to the user as a report, and notifications are sent whenever the information is updated.

[0164] Specific examples

[0165] As a concrete example, consider the case where a customer asks a question to a virtual assistant in a virtual store. The customer's question is captured using the microphone of the user terminal. For example, the question might be, "Please tell me the features of this product."

[0166] User

[0167] When a customer initiates a conversation with the virtual assistant, the conversation is captured in real time by the user terminal.

[0168] Terminal

[0169] The captured audio data is sent from the user device to the server, where it is converted into text using a transcription library.

[0170] server

[0171] The converted text data is summarized using a natural language processing module. For example, a summary such as "This product is lightweight and durable" is generated. The NLP module extracts important information (product features, durability, lightweight) and automatically inputs it into a customer management system.

[0172] In this way, users can improve the efficiency of customer service within the virtual store and automate a series of tasks from recording to management.

[0173] Example prompt sentence:

[0174] "Hello, what products are you looking for today?"

[0175] "What's your favorite thing about the items you purchased today?"

[0176] "Please tell me the features of this product."

[0177] This system makes it possible to improve the efficiency and accuracy of customer service within the virtual store.

[0178] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0179] Step 1:

[0180] Input: Customer's voice or text question

[0181] How it works: A user initiates a conversation with a customer through a virtual assistant in a virtual store. For example, the customer uses the microphone to ask, "What are the features of this product?"

[0182] Output: Raw captured audio or text data

[0183] Step 2:

[0184] Input: Raw audio data

[0185] How it works: The device uses a microphone to capture the customer's speech, and the captured voice data is sent directly to the device for further processing.

[0186] Output: Audio data stored on the device

[0187] Step 3:

[0188] Input: Audio data stored on the device

[0189] How it works: The device transcribes audio data using the speech_recognition library, specifically the Google Speech Recognition API, which converts audio data into text.

[0190] Output: Converted text data

[0191] Step 4:

[0192] Input: Converted text data

[0193] How it works: The server uses the transformers library to summarize text data. Specifically, it uses a generative AI model (e.g., the BART model) to convert input text into a summary sentence.

[0194] Output: Summarized text data

[0195] Step 5:

[0196] Input: Summarized text data

[0197] How it works: The server uses a natural language processing (NLP) module to extract key information, specifically applying entity recognition and relationship extraction algorithms to identify information like customer names, dates, action items, etc.

[0198] Output: Extracted key information (e.g. customer name, date, action items)

[0199] Step 6:

[0200] Input: Extracted important information

[0201] How it works: The server automatically inputs the extracted information through the customer management system's API, cross-checks it against existing data, and updates the database with any new or changed information.

[0202] Output: Updated customer relationship management system database

[0203] Step 7:

[0204] Input: Updated customer management system database

[0205] How it works: The server analyzes customer interaction data and generates a dashboard to visually display it. It visualizes data trends and statistics and provides them to the user as reports.

[0206] Output: Generated dashboards and reports

[0207] Step 8:

[0208] Input: Generated dashboards and reports

[0209] How it works: The server sends updated information to the user's device, allowing the user to check new information and analysis results through the notifications they receive.

[0210] Output: Notification sent to the user

[0211] The above are the specific processing steps of the system of the present invention, which automate customer service in the virtual store, significantly improving efficiency and accuracy.

[0212] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0213] Step 1:

[0214] User

[0215] The user initiates a conversation with the customer. This conversation can take place over the phone, an online conference, or in person. The user conducts the conversation using a specialized app.

[0216] Step 2:

[0217] Terminal

[0218] An application installed on the device captures the audio data of the conversation in real time and sends it to a server.

[0219] Step 3:

[0220] Terminal

[0221] At the same time, the emotion engine recognizes emotions from the user's voice in real time, generates emotion data, and sends it to the server.

[0222] Step 4:

[0223] server

[0224] The server passes the received voice data to a transcription engine, which converts the voice data into text data. For example, a conversation like "The client's name is Taro Yamada, and our next meeting is scheduled for next Monday" is generated as text data.

[0225] Step 5:

[0226] server

[0227] The server passes the generated text data to a natural language processing (NLP) module to generate a summary of the conversation, in this case "The next meeting with Taro Yamada is on Monday."

[0228] Step 6:

[0229] server

[0230] The server uses entity recognition and relation extraction algorithms to extract important information (such as customer name, date, and action item) from the summarized text. For example, the customer name is "Yamada Taro," the date is "next Monday," and the action item is "schedule a meeting."

[0231] Step 7:

[0232] server

[0233] The server receives the emotion data sent by the user and adds the emotion information to the extracted important information. For example, if the user indicates "joy," the emotion data is added as "joy."

[0234] Step 8:

[0235] server

[0236] The server automatically inputs this information via the sales management tool's API, matching it with existing customer data and updating the database as necessary.

[0237] Step 9:

[0238] server

[0239] The server analyzes sales activities based on all data and generates a dashboard containing sentiment data, which displays detailed information about sales activity performance and customer reactions.

[0240] Step 10:

[0241] server

[0242] The server sends the generated dashboard and report to the user's terminal, allowing the user to check the latest information.

[0243] Step 11:

[0244] Terminal

[0245] The user's device receives notifications from the server and displays dashboards and reports, allowing the user to plan their next actions.

[0246] Through these steps, the system of the present invention automates data recording and input work, significantly improving work efficiency while providing an environment in which salespeople can concentrate on conversation. Furthermore, by utilizing emotion data, it is expected to improve the quality of sales activities and increase customer satisfaction.

[0247] Example 2

[0248] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0249] In modern sales activities, it is important for sales representatives to record and manage detailed conversations with customers. However, this recording process is often done manually, prone to human error, and requires time and effort. It is also difficult to properly evaluate customer emotions and important information expressed during conversations and reflect them in sales management tools. This reduces the efficiency and quality of sales activities and affects customer satisfaction.

[0250] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for collecting conversation data, a means for transcribing, a means for summarizing, a means for extracting important information, a means for automatically inputting the data into a sales management tool, a means for recognizing emotions, and a means for evaluating the data. This enables accurate recording and management of conversation data, automates the evaluation of importance based on emotion data, and improves the efficiency and quality of sales activities.

[0251] "Conversation data" refers to the content of voice communication between a user and a customer.

[0252] "Transcription" is the process of analyzing audio data and converting its content into text data.

[0253] "Summarization" is the act of concisely summarizing transcribed text data and extracting the main points.

[0254] "Important information" refers to specific elements or facts extracted from conversation data, and is data that is necessary and useful in sales activities.

[0255] A "sales management tool" is software or an application for managing customer information, sales activity progress, results, etc.

[0256] "Auto-fill" is the process by which data is automatically entered by the system without manual input.

[0257] An "emotion engine" is a technology that recognizes a user's emotions from voice data or text data and extracts them as data.

[0258] "Emotion data" is data that represents the emotional state of the user as recognized by the emotion engine.

[0259] "Evaluation" is the act of assigning importance or priority to a conversation based on emotional data and other information.

[0260] "Matching" is the process of comparing new data with information in an existing database to identify matches.

[0261] "Update" is an operation that keeps existing data up to date with new information.

[0262] "Visualization" is the act of displaying analyzed data in a visual format such as a graph or chart to make it easier to understand.

[0263] This invention is a system that automatically records and manages conversations with customers during sales activities. This system has the functions of collecting conversation data, transcribing it, summarizing it, extracting important information, and automatically inputting it into a sales management tool. It also has the functions of analyzing and visualizing sales activity data and notifying users of updated information. Furthermore, it includes a function that combines an emotion engine that recognizes user emotions, uses emotion data to evaluate the importance of the conversation, and reflects this in the sales management tool.

[0264] System configuration

[0265] User terminal

[0266] The user terminal includes hardware for capturing voice data and dedicated application software for processing the data in real time. As the voice data is collected, an emotion engine recognizes the user's emotions and adds them to the data.

[0267] server

[0268] The server is equipped with a high-performance speech recognition engine, a natural language processing (NLP) module, a database management system, and an interface (API) with sales management tools.

[0269] 1. Capture and transmit audio data

[0270] When a user starts a conversation with a customer, a dedicated application installed on the user's terminal captures the voice data in real time and adds emotional data using an emotion engine.

[0271] This data is immediately sent to the server.

[0272] 2. Transcription and Summarization

[0273] The server passes the received audio data to a transcription engine, which converts the audio into text data. For example, a speech that says, "Hello, Yamada-san. I'd like to talk about our next meeting." is converted into text data.

[0274] The transcribed text data is sent to an NLP module, where the main points are summarized.

[0275] 3. Extracting important information

[0276] From the summarized text, the server uses entity recognition and relationship extraction algorithms to extract key information, such as customer names, dates, and action items.

[0277] 4. Evaluating Emotional Data

[0278] Emotion data sent from the user terminal is received and evaluated by the emotion engine, which determines the importance of the conversation data.

[0279] 5. Input and update to sales management tools

[0280] The server automatically inputs the extracted information and evaluation results through the sales management tool's API, comparing them with the existing database and updating any new information.

[0281] 6. Data Analysis and Visualization

[0282] The server analyzes sales activity data and generates and visualizes a dashboard that includes sentiment data, allowing users to get a detailed overview of sales activities.

[0283] 7. Notification

[0284] Finally, the generated dashboards and reports are sent to the user's device, allowing the user to check the latest information in real time.

[0285] Specific examples

[0286] As a specific example, consider a case where a user is having a telephone conversation with a customer.

[0287] User

[0288] When a user starts a call with a customer, the conversation is captured in real time by an application installed on the user's device. If the user shows strong emotions (e.g., joy, anger, etc.), the emotion engine recognizes the emotion and sends it to the server as emotion data.

[0289] Terminal

[0290] The captured voice data and emotional data are sent from the user's device to the server, allowing the content of the conversation and the user's emotional state to be instantly transmitted to the server.

[0291] server

[0292] The server immediately passes the received voice data to a transcription engine. For example, if a user says, "The next meeting is very important," the transcription engine converts this into text data. Next, a natural language processing module converts this text data into a summary such as "The next meeting is important" and extracts important information. The importance is evaluated taking into account emotional data, and the results are automatically entered into a sales management tool. The information is then compared with an existing customer database, and any new information is updated.

[0293] Prompt Sentence Examples

[0294] Below are some example prompts to input to a generative AI model:

[0295] "Please tell me the procedure for a system that automatically records conversations with customers and inputs emotional data into a sales management tool."

[0296] "Please explain each step of the process when a user starts a conversation with a customer."

[0297] "Can you tell me more about how the emotion engine works and how to feed data into a sales management tool?"

[0298] As a result, the system of the present invention provides an environment where salespeople can concentrate on the conversation, significantly improving work efficiency and quality. Furthermore, by utilizing emotion data, it is expected that customer satisfaction will be improved.

[0299] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0300] Step 1:

[0301] A user initiates a conversation with a customer. The conversation may take place over the phone, an online conference, or in person. A dedicated application installed on the user's device captures the conversation data in real time. The input is voice data. The output is voice data that is sent to the server in real time. Specifically, when the user says "hello," the voice is captured by the application.

[0302] Step 2:

[0303] The device sends the captured voice data along with the emotion data generated by the emotion engine to the server. The input is the captured voice data and emotion data, and the output is the data sent to the server. In concrete terms, if a user expresses high emotion by saying, "The next meeting is very important," that emotion is also sent as data to the server.

[0304] Step 3:

[0305] The server sends the received voice data to the transcription engine. The input is voice data and the output is text data. Specifically, the server converts the voice data "Hello, Yamada-san. I'd like to talk about our next meeting." into text data "Hello, Yamada-san. I'd like to talk about our next meeting."

[0306] Step 4:

[0307] The server sends the transcribed text data to a natural language processing (NLP) module for summarization. The input is the transcribed text data, and the output is the summarized text data. Specifically, the server converts the text data "I'd like to talk about our next meeting" into the summary "Meeting setup."

[0308] Step 5:

[0309] The server extracts important information from the summarized text data using entity recognition and relation extraction algorithms. The input is the summarized text data, and the output is important information (e.g., customer name, date, action item, etc.). Specifically, the server converts the summary "Meeting setup" into "Customer name: Yamada, action item: Meeting setup".

[0310] Step 6:

[0311] The server analyzes the emotion data sent from the user terminal and evaluates the importance of the conversation data. The input is emotion data, and the output is the evaluation result. Specifically, the server evaluates emotion data that is "important" as "high priority."

[0312] Step 7:

[0313] The server automatically inputs the extracted information and evaluation results through the API of the sales management tool. The input is the extracted information and evaluation results, and the output is the updated data in the sales management tool. Specifically, the server automatically reflects "Meeting arrangement with Yamada" in the sales management tool.

[0314] Step 8:

[0315] The server analyzes sales activity data and generates and visualizes a dashboard. The input is sales activity data, and the output is the visualized dashboard. Specifically, the server visualizes the data "Setting up a meeting with Yamada" in graphs and charts.

[0316] Step 9:

[0317] The server notifies the user device of the generated dashboard or report. The input is the visualized data, and the output is a notification to the user device. Specifically, the server notifies the user device of the "next meeting setting" information as an alert.

[0318] (Application example 2)

[0319] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0320] In conventional sales activities, accurately recording conversations with customers, converting them into data, and entering them into sales management tools is labor-intensive and prone to typographical errors and overlooking important information. Furthermore, because emotional data is processed without consideration, it is difficult to create sales strategies that reflect the customer's true feelings and emotions. As a result, there are limitations to improving the efficiency of sales activities and customer satisfaction. A system that solves these problems is needed.

[0321] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting conversation data, means for transcribing the collected conversation data, means for summarizing the transcribed text data, means for extracting important information from the summarized text data, means for automatically inputting the extracted information into a sales management tool, means for analyzing and visualizing sales activity data, means for notifying the user of update information, means for recognizing emotional data of the conversation, means for evaluating the importance of the conversation data based on the emotional data, and means for reflecting the evaluation results in the sales management tool. This makes it possible to accurately record the content of conversations with customers and their emotional data at the time, and automatically input them into the sales management tool. This makes it possible to improve the efficiency of sales activities and customer satisfaction.

[0322] "Conversation data" is information relating to the verbal exchanges that take place between a customer and a user.

[0323] "Means for collection" refers to devices or software for acquiring conversation data with customers and inputting it into the system.

[0324] "Transcribing means" means speech recognition technology or software for converting collected audio data into text data.

[0325] A "summarization tool" is a natural language processing technique or software that extracts key points from transcribed text data and summarizes them in a concise format.

[0326] A "means for extracting key information" is technology or software that identifies key elements from the summarized text data, such as customer names, dates, and action items to track.

[0327] "Means for automatically inputting information into a sales management tool" refers to a device or software that automatically reflects extracted important information in a sales management system (such as CRM).

[0328] "Means for analysis" refers to technology or software used to analyze collected sales activity data and evaluate its performance and trends.

[0329] "Visualization means" refers to technology or software for displaying analyzed data in a visually easy-to-understand format, such as a graph or chart.

[0330] "Means for notifying updated information" refers to technology or software that automatically notifies users of the latest sales activity data and analysis results.

[0331] "Means for recognizing emotional data" refers to technology or software for detecting emotional states from collected conversation data.

[0332] The "means for assessing importance" is a technology or software for assessing the content and importance of a conversation based on emotional data.

[0333] "Database matching means" means technology or software used to compare newly collected information with existing customer databases to identify matches.

[0334] "Means for updating information" refers to technology or software for updating the database in the sales management tool with the latest information based on the collated data.

[0335] This invention relates to a system that automatically records and processes conversations between salespeople and customers, primarily during sales activities in brick-and-mortar stores, and inputs the content into a sales management tool. The system of the present invention is equipped with a series of means for collecting conversation data, transcribing it, summarizing it, extracting important information, and automatically inputting it into the sales management tool. It can also recognize emotional data in the conversation, evaluate the importance of the conversation based on that information, and reflect the evaluation results in the sales management tool.

[0336] User

[0337] A salesperson (user) starts a conversation with a customer. This conversation is usually conducted face-to-face. A dedicated application is installed on the user's device, and this application captures conversation data in real time and sends it to a server.

[0338] Terminal

[0339] The user device transmits the captured voice data to the server, which instantly transmits the conversation data to the server. The device is equipped with an emotion engine that recognizes emotions from the voice data in real time.

[0340] server

[0341] The server sends the received audio data to a transcription engine using the Google Cloud Speech-to-Text API. The server converts the audio data into text and then summarizes it using a natural language processing (NLP) module. The transcription and summary data are temporarily stored. The server then uses entity recognition and relationship extraction algorithms to extract important information from the summary (e.g., customer name, date, action items, etc.).

[0342] Next, the server receives the emotion data sent from the user's device and evaluates the importance of the conversation data based on the emotions recognized by the emotion engine (e.g., Sentiment Analysis API). Based on this evaluation result, the extracted information is automatically entered into the sales management tool (e.g., Salesforce) via its API. The sales management tool compares the data with the existing database and updates it if any new information or changes are found.

[0343] The server also analyzes sales activity data and generates and visualizes dashboards. The generated dashboards also include sentiment data, enabling a more detailed understanding of the overall picture of sales activities. Finally, the server notifies the user of the generated dashboards and reports on their devices, allowing the user to access the latest information.

[0344] Specific examples

[0345] As a concrete example, consider a scenario in which a salesperson in a physical store talks with a customer about a new product. The salesperson asks, "What do you think about the features of this new product?" and the customer replies, "I think it's very attractive." This conversation is captured in real time using the smartphone's microphone. The captured voice data is immediately sent to a server, where a transcription engine converts it into text data such as, "What do you think about the features of this new product? I think it's very attractive."

[0346] The natural language processing module summarizes this as "the new product's features are attractive," and the entity recognition algorithm extracts the important information "new product" and "attractive." The emotion engine also recognizes that the customer is expressing "positive emotion" and reflects this in the evaluation results. All data is entered into the sales management tool by the server, where it is compared and updated with existing data. The analysis results of this conversation are visualized and displayed on a dashboard such as Salesforce.

[0347] Example prompts to input to the generative AI model

[0348] "Generate a program that transcribes customer voice data and analyzes sentiment data."

[0349] "Please provide sample code for a system that automatically records customer conversations and inputs key information into a sales management tool."

[0350] As described above, the system of the present invention allows salespeople to concentrate on talking to customers and greatly improves work efficiency by automating data recording and input. In addition, by using emotional data, it is possible to grasp customer reactions in real time, providing a valuable clue for appropriate responses.

[0351] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0352] Step 1:

[0353] The user starts a conversation with the customer. The conversation data is captured in real time by an application installed on the user's terminal. The input is the voice conversation between the customer and the user, and the output is the captured voice data.

[0354] Step 2:

[0355] The terminal transmits the captured voice data to the server. The input is the captured voice data, and the output is the transmission of the voice data to the server. This step ensures that the conversation data is immediately transmitted to the server.

[0356] Step 3:

[0357] The server sends the received audio data to a transcription engine, which converts the audio into text using the Google Cloud Speech-to-Text API. The input is audio data, and the output is transcribed text data.

[0358] Step 4:

[0359] The transcribed text data is summarized by the server's natural language processing (NLP) module. The summarization module extracts the main content and summarizes it in a concise format. The input is the transcribed text data, and the output is the summarized text data.

[0360] Step 5:

[0361] The server extracts key information from the summarized text data, using entity recognition and relationship extraction algorithms to identify key information such as customer names, dates, action items, etc. The input is the summarized text data, and the output is the extracted key information.

[0362] Step 6:

[0363] The server receives emotion data sent from the user device and evaluates the importance of the conversation data based on the emotions recognized by the emotion engine (e.g., Sentiment Analysis API). The input is emotion data and text data, and the output is the importance evaluation result.

[0364] Step 7:

[0365] Based on the evaluation results, the server automatically inputs the extracted information through the API of the sales management tool. The input is important information and the importance evaluation results, and the output is data input into the sales management tool. This is compared with the existing database, and any new information or changes are updated.

[0366] Step 8:

[0367] The server analyzes the sales activity data and generates a dashboard. The analysis results, including sentiment data, are displayed. The input is the sales activity data to be analyzed, and the output is the generated dashboard.

[0368] Step 9:

[0369] Finally, the server notifies the user of the generated dashboard or report. The user receives the notification and can access the latest information. The input is the generated dashboard or report, and the output is the notification sent to the user's device.

[0370] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0371] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0372] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0373] [Second embodiment]

[0374] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0375] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0376] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0377] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0378] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0379] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0380] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0381] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0382] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0383] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0384] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0385] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0386] Step 1:

[0387] User

[0388] Users initiate conversations with customers, whether by phone, online conference, or face-to-face.

[0389] Step 2:

[0390] Terminal

[0391] A dedicated application installed on the user's device captures the audio data of the conversation in real time and sends it to the server.

[0392] Step 3:

[0393] server

[0394] The server sends the received audio data to a transcription engine, which analyzes the audio data and generates corresponding text data.

[0395] Step 4:

[0396] server

[0397] The server passes the generated text data to a natural language processing (NLP) module to generate a summary of the conversation.

[0398] Step 5:

[0399] server

[0400] The server extracts important information (e.g., customer names, dates, action items) from the summarized text using entity recognition and relationship extraction algorithms.

[0401] Step 6:

[0402] server

[0403] The server automatically inputs the extracted information via the sales management tool's API, comparing it with existing customer data and updating any new or changed information.

[0404] Step 7:

[0405] server

[0406] The server analyzes all collected data and generates a dashboard to visualize sales activities.

[0407] Step 8:

[0408] server

[0409] The server prepares the generated dashboards and reports and prepares notifications for the users.

[0410] Step 9:

[0411] Terminal

[0412] The user terminal receives notifications from the server and displays automatically generated sales activity reports and important update information.

[0413] Step 10:

[0414] User

[0415] Users can view new information and reports through their devices and plan their next actions.

[0416] Example 1

[0417] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0418] There is a need to efficiently extract important information from voice data and automatically input it into data management tools such as sales management, thereby reducing the time and effort required for manual input and improving the efficiency and accuracy of sales activities. Furthermore, there is a need to enable sales representatives to make quick decisions through information visualization and real-time information notifications.

[0419] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0420] In this invention, the server includes means for collecting voice data, means for converting the voice data into text data, means for summarizing the voice data, means for extracting important information, means for automatically inputting the voice data into a data management tool, means for comparing the voice data with a database and updating the data with new information, means for analyzing and visualizing the data, and means for notifying the user. This makes it possible to quickly and accurately extract important information from the voice data and automatically input the information into the data management tool. Furthermore, data visualization and real-time notifications improve the efficiency of overall sales activities.

[0421] "Audio data" means digital or analog information relating to audio.

[0422] A "collection means" is a device or method for acquiring audio data and storing or transmitting it in a form that can be further processed.

[0423] "Means for converting into text data" refers to a device or software for converting voice data into text format.

[0424] A "summarizing means" is a device or method for extracting the main points or important information from long text data and creating a shortened version.

[0425] "Significant information" is a data element that is necessary or valuable in a particular context, such as a customer's name or a meeting date.

[0426] A "data management tool" is software or a system for storing, retrieving, organizing, and manipulating data. Examples include sales management tools.

[0427] An "automated input means" is a device or method for inputting extracted information into a data management tool without manual intervention.

[0428] "Means for checking against the database and updating with new information" means a device or method for comparing information in an existing database with newly acquired information and updating the database as necessary.

[0429] A "data analysis and visualization tool" is a device or method for analyzing and visually transforming data in order to display it in an understandable format (e.g., graphs or charts).

[0430] "Notification means" refers to a device or method for notifying a user of new or updated information.

[0431] MODE FOR CARRYING OUT THE INVENTION

[0432] This invention relates to a system that automatically records and processes conversations with customers, primarily during sales activities, and inputs them into a sales management tool. The system of the present invention is equipped with a series of means for collecting conversation data, transcribing it, summarizing it, extracting important information, and automatically inputting it into the sales management tool. It also includes a function for analyzing and visualizing sales activity data and notifying users of updated information.

[0433] System program processing

[0434] The following is a detailed description of how this system works.

[0435] User

[0436] A user initiates a conversation with a customer. The conversation can take place over the phone, via online conference, or in person. For example, if a user initiates a conversation using an online conference tool (e.g., a general video conference tool), the content of the conversation is captured in real time by a dedicated application installed on the user's device.

[0437] Terminal

[0438] The user device sends the captured audio data to the server. Specifically, a dedicated application compresses the audio data and sends it to the server using the HTTPS protocol. For example, the audio data is chunked every 10 seconds and sent continuously with minimal delay.

[0439] server

[0440] The server passes the received voice data to a transcription engine (e.g., a general cloud-based speech recognition API) and converts the voice data into text data. For example, the speech "Please schedule the next meeting for Monday" is converted into text "Please schedule the next meeting for Monday."

[0441] The server then passes the transcribed text data to a natural language processing (NLP) module (e.g., a popular NLP framework) for summarization. For example, a one-hour conversation might be summarized as "agreements on the next meeting date and time and customer needs."

[0442] The server then uses entity recognition and relationship extraction algorithms to extract important information from the summarized text (such as customer names, dates, and action items). For example, from the text "The next meeting with Taro Yamada is scheduled for Monday," the server extracts information such as "Taro Yamada," "next meeting," and "Monday."

[0443] The extracted information is automatically entered into the sales management tool (e.g., general sales management software) via its API. For example, information such as the customer name "Yamada Taro" and the next meeting date "Monday" is added to the database.

[0444] The server also analyzes the sales activity data and generates a dashboard (e.g., a common data visualization tool) to visualize it, for example, by displaying the number of customer meetings last week or a schedule for next week in graphs and tables.

[0445] Finally, the server notifies the user of the generated dashboard or report. For example, when the user opens a dedicated application, the user is notified to display the new information or report.

[0446] Specific examples

[0447] As a concrete example, consider the case where a user is holding an online conference with a client. When the user starts a conversation using a general video conferencing tool, a dedicated application captures the audio data in real time. For example, when the user says, "Please schedule the next meeting for Monday," the audio is captured.

[0448] The dedicated application sends the captured audio data to a server using the HTTPS protocol. The audio data is divided into chunks every 10 seconds and received by the server.

[0449] The server passes the audio data to a transcription engine, which converts it into text: "Please schedule our next meeting for Monday." The server then passes the text data to an NLP module, which generates a summary: "Next meeting: Monday." From that summary, it extracts key information, such as "meeting" and "Monday."

[0450] The server automatically inputs the extracted information using the sales management tool's API and updates the database. The server also generates a dashboard based on the sales activity data and displays a graph of the next week's meeting schedule.

[0451] Finally, the server notifies the user that a report of next week's meeting schedule is available, and the user opens the dedicated application to check the latest information.

[0452] Prompt Sentence Examples

[0453] Below is an example of a prompt sentence to input to the generative AI model.

[0454] After transcribing and summarizing your customer conversation, extract the following information:

[0455] 1. Customer name

[0456] 2. Meeting date and time

[0457] 3. Action Items

[0458] example:

[0459] The client's name is Taro Yamada, and our next meeting can be scheduled for next Monday."

[0460] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0461] Step 1: Capture conversation data

[0462] User

[0463] A user starts a conversation with a customer by telephone, online conference, or face-to-face. For example, when a user starts a conversation using a video conferencing tool, the conversation content is captured in real time by a dedicated application installed on the user's terminal. The input is voice data, and the output is the captured digitized voice data.

[0464] Step 2: Sending audio data

[0465] Terminal

[0466] The user device sends the captured audio data to the server. Specifically, a dedicated application compresses the audio data and sends it to the server using the HTTPS protocol. For example, the audio data is divided into chunks every 10 seconds and sent continuously with minimal delay. The input is the captured audio data, and the output is the audio data sent to the server.

[0467] Step 3: Transcribe the audio data

[0468] server

[0469] The server passes the received voice data to a transcription engine (for example, a general cloud-based voice recognition API) and converts the voice data into text data. For example, voice data such as "Please schedule the next meeting for Monday" is converted into text data such as "Please schedule the next meeting for Monday." The input is the voice data sent to the server, and the output is the transcribed text data.

[0470] Step 4: Summarizing the text data

[0471] server

[0472] The server passes the transcribed text data to a natural language processing (NLP) module (e.g., a general NLP framework) for summarization. For example, a one-hour conversation is summarized as "agreement on the date and time of the next meeting and the customer's requests." The input is the transcribed text data, and the output is the summarized text data.

[0473] Step 5: Extracting important information

[0474] server

[0475] The server uses entity recognition and relationship extraction algorithms to extract important information (such as customer names, dates, and action items) from the summarized text. For example, from the text "The next meeting with Yamada Taro is scheduled for Monday," the server extracts information such as "Yamada Taro," "next meeting," and "Monday." The input is the summarized text data, and the output is the extracted important information.

[0476] Step 6: Enter information into your sales management tool

[0477] server

[0478] The server automatically inputs the extracted important information through the API of a sales management tool (e.g., general sales management software). For example, information such as "Customer name: Yamada Taro, next meeting: Monday" is added to the database. The input is the extracted important information, and the output is the information entered into the sales management tool.

[0479] Step 7: Analyze and visualize the data

[0480] server

[0481] The server analyzes the sales activity data and generates a dashboard (e.g., a common data visualization tool) to visualize it. For example, it displays the number of customer meetings last week or the schedule for next week in graphs and tables. The input is the sales activity data, and the output is the generated dashboard.

[0482] Step 8: Report Notification

[0483] server

[0484] The server notifies the user terminal of the generated dashboard or report. For example, when the user opens a dedicated application, the server notifies the user to display new information or reports. The input is the generated dashboard or report, and the output is the notified information.

[0485] (Application example 1)

[0486] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0487] When interacting with customers in a virtual store, it is necessary to efficiently respond to customer questions and requests, and to automatically record, process, and manage the content of the conversation. Currently, these tasks are often done manually, which is time-consuming and labor-intensive, so there is an urgent need to create an efficient system.

[0488] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0489] In this invention, the server includes means for collecting conversation data, means for transcribing the collected conversation data, means for summarizing the transcribed text data, means for extracting important information from the summarized text data, means for automatically inputting the extracted information into a customer management system, means for analyzing and visualizing customer interaction data, and means for notifying users of updated information. This makes it possible to automate everything from recording to managing customer interactions, thereby improving the efficiency and accuracy of customer interactions.

[0490] "Conversation Data" means records of voice or text communications with customers.

[0491] "Means of collection" refers to the equipment or software that captures the voice and text data necessary to interact with customers.

[0492] "Transcription method" refers to the process or technique for converting audio data into text data.

[0493] "Summarization methods" refers to techniques and processes for extracting key information from transcribed text data and summarizing it concisely.

[0494] "Key information extraction methods" are techniques and methods for identifying and extracting specific entities or action items from summarized text data.

[0495] A "customer management system" refers to software or a platform for centrally managing customer information and transaction history.

[0496] "Means for automatic input" refers to methods or technologies for mechanically registering extracted information into a customer management system rather than manually.

[0497] "Analysis and visualization means" refers to techniques and methods for analyzing collected data and displaying it in a visual format such as graphs or charts.

[0498] "Means for notifying users of updated information" refers to techniques and methods for notifying users when data is updated or new information is obtained.

[0499] A "virtual store" is a virtual shopping environment or service provided over the Internet.

[0500] A "virtual assistant" is a character or bot equipped with artificial intelligence that interacts with customers and provides services within a virtual store.

[0501] This invention relates to a system for automating and efficiently managing customer interactions in a virtual store. The system of the present invention is equipped with a series of means for collecting and recording customer conversations, transcribing them, summarizing them, extracting important information, and automatically inputting them into a customer management system. It also includes a function for analyzing and visualizing the collected customer interaction data and notifying users of updated information.

[0502] System program processing

[0503] This system is realized using the following hardware and software.

[0504] Hardware:

[0505] User devices for customer service (e.g., smartphones and desktop PCs)

[0506] Microphone (for voice input)

[0507] Server (for data processing and management)

[0508] software:

[0509] speech_recognition library: Captures and transcribes speech data.

[0510] The transformers library: performs natural language processing (NLP).

[0511] Virtual CRM API module: Automatically registers extracted important information into a customer management system.

[0512] The server receives customer conversation data from the user's device and converts the speech to text in real time using a transcription engine. This text data is temporarily stored and then summarized using a natural language processing (NLP) module. From the summarized text, the server uses entity recognition and relationship extraction algorithms to extract important information (e.g., customer name, date, action items, etc.).

[0513] The extracted information is automatically entered into the customer management system via API. At this time, it is compared with the existing database, and if there is new information or changes, the database is updated. The server also analyzes the customer interaction data and visualizes it as a dashboard. This dashboard is provided to the user as a report, and notifications are sent whenever the information is updated.

[0514] Specific examples

[0515] As a concrete example, consider the case where a customer asks a question to a virtual assistant in a virtual store. The customer's question is captured using the microphone of the user terminal. For example, the question might be, "Please tell me the features of this product."

[0516] User

[0517] When a customer initiates a conversation with the virtual assistant, the conversation is captured in real time by the user terminal.

[0518] Terminal

[0519] The captured audio data is sent from the user device to the server, where it is converted into text using a transcription library.

[0520] server

[0521] The converted text data is summarized using a natural language processing module. For example, a summary such as "This product is lightweight and durable" is generated. The NLP module extracts important information (product features, durability, lightweight) and automatically inputs it into a customer management system.

[0522] In this way, users can improve the efficiency of customer service within the virtual store and automate a series of tasks from recording to management.

[0523] Example prompt sentence:

[0524] "Hello, what products are you looking for today?"

[0525] "What's your favorite thing about the items you purchased today?"

[0526] "Please tell me the features of this product."

[0527] This system makes it possible to improve the efficiency and accuracy of customer service within the virtual store.

[0528] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0529] Step 1:

[0530] Input: Customer's voice or text question

[0531] How it works: A user initiates a conversation with a customer through a virtual assistant in a virtual store. For example, the customer uses the microphone to ask, "What are the features of this product?"

[0532] Output: Raw captured audio or text data

[0533] Step 2:

[0534] Input: Raw audio data

[0535] How it works: The device uses a microphone to capture the customer's speech, and the captured voice data is sent directly to the device for further processing.

[0536] Output: Audio data stored on the device

[0537] Step 3:

[0538] Input: Audio data stored on the device

[0539] How it works: The device transcribes audio data using the speech_recognition library, specifically the Google Speech Recognition API, which converts audio data into text.

[0540] Output: Converted text data

[0541] Step 4:

[0542] Input: Converted text data

[0543] How it works: The server uses the transformers library to summarize text data. Specifically, it uses a generative AI model (e.g., the BART model) to convert input text into a summary sentence.

[0544] Output: Summarized text data

[0545] Step 5:

[0546] Input: Summarized text data

[0547] How it works: The server uses a natural language processing (NLP) module to extract key information, specifically applying entity recognition and relationship extraction algorithms to identify information like customer names, dates, action items, etc.

[0548] Output: Extracted key information (e.g. customer name, date, action items)

[0549] Step 6:

[0550] Input: Extracted important information

[0551] How it works: The server automatically inputs the extracted information through the customer management system's API, cross-checks it against existing data, and updates the database with any new or changed information.

[0552] Output: Updated customer relationship management system database

[0553] Step 7:

[0554] Input: Updated customer management system database

[0555] How it works: The server analyzes customer interaction data and generates a dashboard to visually display it. It visualizes data trends and statistics and provides them to the user as reports.

[0556] Output: Generated dashboards and reports

[0557] Step 8:

[0558] Input: Generated dashboards and reports

[0559] How it works: The server sends updated information to the user's device, allowing the user to check new information and analysis results through the notifications they receive.

[0560] Output: Notification sent to the user

[0561] The above are the specific processing steps of the system of the present invention, which automate customer service in the virtual store, significantly improving efficiency and accuracy.

[0562] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0563] Step 1:

[0564] User

[0565] The user initiates a conversation with the customer. This conversation can take place over the phone, an online conference, or in person. The user conducts the conversation using a specialized app.

[0566] Step 2:

[0567] Terminal

[0568] An application installed on the device captures the audio data of the conversation in real time and sends it to a server.

[0569] Step 3:

[0570] Terminal

[0571] At the same time, the emotion engine recognizes emotions from the user's voice in real time, generates emotion data, and sends it to the server.

[0572] Step 4:

[0573] server

[0574] The server passes the received voice data to a transcription engine, which converts the voice data into text data. For example, a conversation like "The client's name is Taro Yamada, and our next meeting is scheduled for next Monday" is generated as text data.

[0575] Step 5:

[0576] server

[0577] The server passes the generated text data to a natural language processing (NLP) module to generate a summary of the conversation, in this case "The next meeting with Taro Yamada is on Monday."

[0578] Step 6:

[0579] server

[0580] The server uses entity recognition and relation extraction algorithms to extract important information (such as customer name, date, and action item) from the summarized text. For example, the customer name is "Yamada Taro," the date is "next Monday," and the action item is "schedule a meeting."

[0581] Step 7:

[0582] server

[0583] The server receives the emotion data sent by the user and adds the emotion information to the extracted important information. For example, if the user indicates "joy," the emotion data is added as "joy."

[0584] Step 8:

[0585] server

[0586] The server automatically inputs this information via the sales management tool's API, matching it with existing customer data and updating the database as necessary.

[0587] Step 9:

[0588] server

[0589] The server analyzes sales activities based on all data and generates a dashboard containing sentiment data, which displays detailed information about sales activity performance and customer reactions.

[0590] Step 10:

[0591] server

[0592] The server sends the generated dashboard and report to the user's terminal, allowing the user to check the latest information.

[0593] Step 11:

[0594] Terminal

[0595] The user's device receives notifications from the server and displays dashboards and reports, allowing the user to plan their next actions.

[0596] Through these steps, the system of the present invention automates data recording and input work, significantly improving work efficiency while providing an environment in which salespeople can concentrate on conversation. Furthermore, by utilizing emotion data, it is expected to improve the quality of sales activities and increase customer satisfaction.

[0597] Example 2

[0598] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0599] In modern sales activities, it is important for sales representatives to record and manage detailed conversations with customers. However, this recording process is often done manually, prone to human error, and requires time and effort. It is also difficult to properly evaluate customer emotions and important information expressed during conversations and reflect them in sales management tools. This reduces the efficiency and quality of sales activities and affects customer satisfaction.

[0600] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for collecting conversation data, a means for transcribing, a means for summarizing, a means for extracting important information, a means for automatically inputting the data into a sales management tool, a means for recognizing emotions, and a means for evaluating the data. This enables accurate recording and management of conversation data, automates the evaluation of importance based on emotion data, and improves the efficiency and quality of sales activities.

[0601] "Conversation data" refers to the content of voice communication between a user and a customer.

[0602] "Transcription" is the process of analyzing audio data and converting its content into text data.

[0603] "Summarization" is the act of concisely summarizing transcribed text data and extracting the main points.

[0604] "Important information" refers to specific elements or facts extracted from conversation data, and is data that is necessary and useful in sales activities.

[0605] A "sales management tool" is software or an application for managing customer information, sales activity progress, results, etc.

[0606] "Auto-fill" is the process by which data is automatically entered by the system without manual input.

[0607] An "emotion engine" is a technology that recognizes a user's emotions from voice data or text data and extracts them as data.

[0608] "Emotion data" is data that represents the emotional state of the user as recognized by the emotion engine.

[0609] "Evaluation" is the act of assigning importance or priority to a conversation based on emotional data and other information.

[0610] "Matching" is the process of comparing new data with information in an existing database to identify matches.

[0611] "Update" is an operation that keeps existing data up to date with new information.

[0612] "Visualization" is the act of displaying analyzed data in a visual format such as a graph or chart to make it easier to understand.

[0613] This invention is a system that automatically records and manages conversations with customers during sales activities. This system has the functions of collecting conversation data, transcribing it, summarizing it, extracting important information, and automatically inputting it into a sales management tool. It also has the functions of analyzing and visualizing sales activity data and notifying users of updated information. Furthermore, it includes a function that combines an emotion engine that recognizes user emotions, uses emotion data to evaluate the importance of the conversation, and reflects this in the sales management tool.

[0614] System configuration

[0615] User terminal

[0616] The user terminal includes hardware for capturing voice data and dedicated application software for processing the data in real time. As the voice data is collected, an emotion engine recognizes the user's emotions and adds them to the data.

[0617] server

[0618] The server is equipped with a high-performance speech recognition engine, a natural language processing (NLP) module, a database management system, and an interface (API) with sales management tools.

[0619] 1. Capture and transmit audio data

[0620] When a user starts a conversation with a customer, a dedicated application installed on the user's terminal captures the voice data in real time and adds emotional data using an emotion engine.

[0621] This data is immediately sent to the server.

[0622] 2. Transcription and Summarization

[0623] The server passes the received audio data to a transcription engine, which converts the audio into text data. For example, a speech that says, "Hello, Yamada-san. I'd like to talk about our next meeting." is converted into text data.

[0624] The transcribed text data is sent to an NLP module, where the main points are summarized.

[0625] 3. Extracting important information

[0626] From the summarized text, the server uses entity recognition and relationship extraction algorithms to extract key information, such as customer names, dates, and action items.

[0627] 4. Evaluating Emotional Data

[0628] Emotion data sent from the user terminal is received and evaluated by the emotion engine, which determines the importance of the conversation data.

[0629] 5. Input and update to sales management tools

[0630] The server automatically inputs the extracted information and evaluation results through the sales management tool's API, comparing them with the existing database and updating any new information.

[0631] 6. Data Analysis and Visualization

[0632] The server analyzes sales activity data and generates and visualizes a dashboard that includes sentiment data, allowing users to get a detailed overview of sales activities.

[0633] 7. Notification

[0634] Finally, the generated dashboards and reports are sent to the user's device, allowing the user to check the latest information in real time.

[0635] Specific examples

[0636] As a specific example, consider a case where a user is having a telephone conversation with a customer.

[0637] User

[0638] When a user starts a call with a customer, the conversation is captured in real time by an application installed on the user's device. If the user shows strong emotions (e.g., joy, anger, etc.), the emotion engine recognizes the emotion and sends it to the server as emotion data.

[0639] Terminal

[0640] The captured voice data and emotional data are sent from the user's device to the server, allowing the content of the conversation and the user's emotional state to be instantly transmitted to the server.

[0641] server

[0642] The server immediately passes the received voice data to a transcription engine. For example, if a user says, "The next meeting is very important," the transcription engine converts this into text data. Next, a natural language processing module converts this text data into a summary such as "The next meeting is important" and extracts important information. The importance is evaluated taking into account emotional data, and the results are automatically entered into a sales management tool. The information is then compared with an existing customer database, and any new information is updated.

[0643] Prompt Sentence Examples

[0644] Below are some example prompts to input to a generative AI model:

[0645] "Please tell me the procedure for a system that automatically records conversations with customers and inputs emotional data into a sales management tool."

[0646] "Please explain each step of the process when a user starts a conversation with a customer."

[0647] "Can you tell me more about how the emotion engine works and how to feed data into a sales management tool?"

[0648] As a result, the system of the present invention provides an environment where salespeople can concentrate on the conversation, significantly improving work efficiency and quality. Furthermore, by utilizing emotion data, it is expected that customer satisfaction will be improved.

[0649] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0650] Step 1:

[0651] A user initiates a conversation with a customer. The conversation may take place over the phone, an online conference, or in person. A dedicated application installed on the user's device captures the conversation data in real time. The input is voice data. The output is voice data that is sent to the server in real time. Specifically, when the user says "hello," the voice is captured by the application.

[0652] Step 2:

[0653] The device sends the captured voice data along with the emotion data generated by the emotion engine to the server. The input is the captured voice data and emotion data, and the output is the data sent to the server. In concrete terms, if a user expresses high emotion by saying, "The next meeting is very important," that emotion is also sent as data to the server.

[0654] Step 3:

[0655] The server sends the received voice data to the transcription engine. The input is voice data and the output is text data. Specifically, the server converts the voice data "Hello, Yamada-san. I'd like to talk about our next meeting." into text data "Hello, Yamada-san. I'd like to talk about our next meeting."

[0656] Step 4:

[0657] The server sends the transcribed text data to a natural language processing (NLP) module for summarization. The input is the transcribed text data, and the output is the summarized text data. Specifically, the server converts the text data "I'd like to talk about our next meeting" into the summary "Meeting setup."

[0658] Step 5:

[0659] The server extracts important information from the summarized text data using entity recognition and relation extraction algorithms. The input is the summarized text data, and the output is important information (e.g., customer name, date, action item, etc.). Specifically, the server converts the summary "Meeting setup" into "Customer name: Yamada, action item: Meeting setup".

[0660] Step 6:

[0661] The server analyzes the emotion data sent from the user terminal and evaluates the importance of the conversation data. The input is emotion data, and the output is the evaluation result. Specifically, the server evaluates emotion data that is "important" as "high priority."

[0662] Step 7:

[0663] The server automatically inputs the extracted information and evaluation results through the API of the sales management tool. The input is the extracted information and evaluation results, and the output is the updated data in the sales management tool. Specifically, the server automatically reflects "Meeting arrangement with Yamada" in the sales management tool.

[0664] Step 8:

[0665] The server analyzes sales activity data and generates and visualizes a dashboard. The input is sales activity data, and the output is the visualized dashboard. Specifically, the server visualizes the data "Setting up a meeting with Yamada" in graphs and charts.

[0666] Step 9:

[0667] The server notifies the user device of the generated dashboard or report. The input is the visualized data, and the output is a notification to the user device. Specifically, the server notifies the user device of the "next meeting setting" information as an alert.

[0668] (Application example 2)

[0669] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0670] In conventional sales activities, accurately recording conversations with customers, converting them into data, and entering them into sales management tools is labor-intensive and prone to typographical errors and overlooking important information. Furthermore, because emotional data is processed without consideration, it is difficult to create sales strategies that reflect the customer's true feelings and emotions. As a result, there are limitations to improving the efficiency of sales activities and customer satisfaction. A system that solves these problems is needed.

[0671] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting conversation data, means for transcribing the collected conversation data, means for summarizing the transcribed text data, means for extracting important information from the summarized text data, means for automatically inputting the extracted information into a sales management tool, means for analyzing and visualizing sales activity data, means for notifying the user of update information, means for recognizing emotional data of the conversation, means for evaluating the importance of the conversation data based on the emotional data, and means for reflecting the evaluation results in the sales management tool. This makes it possible to accurately record the content of conversations with customers and their emotional data at the time, and automatically input them into the sales management tool. This makes it possible to improve the efficiency of sales activities and customer satisfaction.

[0672] "Conversation data" is information relating to the verbal exchanges that take place between a customer and a user.

[0673] "Means for collection" refers to devices or software for acquiring conversation data with customers and inputting it into the system.

[0674] "Transcribing means" means speech recognition technology or software for converting collected audio data into text data.

[0675] A "summarization tool" is a natural language processing technique or software that extracts key points from transcribed text data and summarizes them in a concise format.

[0676] A "means for extracting key information" is technology or software that identifies key elements from the summarized text data, such as customer names, dates, and action items to track.

[0677] "Means for automatically inputting information into a sales management tool" refers to a device or software that automatically reflects extracted important information in a sales management system (such as CRM).

[0678] "Means for analysis" refers to technology or software used to analyze collected sales activity data and evaluate its performance and trends.

[0679] "Visualization means" refers to technology or software for displaying analyzed data in a visually easy-to-understand format, such as a graph or chart.

[0680] "Means for notifying updated information" refers to technology or software that automatically notifies users of the latest sales activity data and analysis results.

[0681] "Means for recognizing emotional data" refers to technology or software for detecting emotional states from collected conversation data.

[0682] The "means for assessing importance" is a technology or software for assessing the content and importance of a conversation based on emotional data.

[0683] "Database matching means" means technology or software used to compare newly collected information with existing customer databases to identify matches.

[0684] "Means for updating information" refers to technology or software for updating the database in the sales management tool with the latest information based on the collated data.

[0685] This invention relates to a system that automatically records and processes conversations between salespeople and customers, primarily during sales activities in brick-and-mortar stores, and inputs the content into a sales management tool. The system of the present invention is equipped with a series of means for collecting conversation data, transcribing it, summarizing it, extracting important information, and automatically inputting it into the sales management tool. It can also recognize emotional data in the conversation, evaluate the importance of the conversation based on that information, and reflect the evaluation results in the sales management tool.

[0686] User

[0687] A salesperson (user) starts a conversation with a customer. This conversation is usually conducted face-to-face. A dedicated application is installed on the user's device, and this application captures conversation data in real time and sends it to a server.

[0688] Terminal

[0689] The user device transmits the captured voice data to the server, which instantly transmits the conversation data to the server. The device is equipped with an emotion engine that recognizes emotions from the voice data in real time.

[0690] server

[0691] The server sends the received audio data to a transcription engine using the Google Cloud Speech-to-Text API. The server converts the audio data into text and then summarizes it using a natural language processing (NLP) module. The transcription and summary data are temporarily stored. The server then uses entity recognition and relationship extraction algorithms to extract important information from the summary (e.g., customer name, date, action items, etc.).

[0692] Next, the server receives the emotion data sent from the user's device and evaluates the importance of the conversation data based on the emotions recognized by the emotion engine (e.g., Sentiment Analysis API). Based on this evaluation result, the extracted information is automatically entered into the sales management tool (e.g., Salesforce) via its API. The sales management tool compares the data with the existing database and updates it if any new information or changes are found.

[0693] The server also analyzes sales activity data and generates and visualizes dashboards. The generated dashboards also include sentiment data, enabling a more detailed understanding of the overall picture of sales activities. Finally, the server notifies the user of the generated dashboards and reports on their devices, allowing the user to access the latest information.

[0694] Specific examples

[0695] As a concrete example, consider a scenario in which a salesperson in a physical store talks with a customer about a new product. The salesperson asks, "What do you think about the features of this new product?" and the customer replies, "I think it's very attractive." This conversation is captured in real time using the smartphone's microphone. The captured voice data is immediately sent to a server, where a transcription engine converts it into text data such as, "What do you think about the features of this new product? I think it's very attractive."

[0696] The natural language processing module summarizes this as "the new product's features are attractive," and the entity recognition algorithm extracts the important information "new product" and "attractive." The emotion engine also recognizes that the customer is expressing "positive emotion" and reflects this in the evaluation results. All data is entered into the sales management tool by the server, where it is compared and updated with existing data. The analysis results of this conversation are visualized and displayed on a dashboard such as Salesforce.

[0697] Example prompts to input to the generative AI model

[0698] "Generate a program that transcribes customer voice data and analyzes sentiment data."

[0699] "Please provide sample code for a system that automatically records customer conversations and inputs key information into a sales management tool."

[0700] As described above, the system of the present invention allows salespeople to concentrate on talking to customers and greatly improves work efficiency by automating data recording and input. In addition, by using emotional data, it is possible to grasp customer reactions in real time, providing a valuable clue for appropriate responses.

[0701] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0702] Step 1:

[0703] The user starts a conversation with the customer. The conversation data is captured in real time by an application installed on the user's terminal. The input is the voice conversation between the customer and the user, and the output is the captured voice data.

[0704] Step 2:

[0705] The terminal transmits the captured voice data to the server. The input is the captured voice data, and the output is the transmission of the voice data to the server. This step ensures that the conversation data is immediately transmitted to the server.

[0706] Step 3:

[0707] The server sends the received audio data to a transcription engine, which converts the audio into text using the Google Cloud Speech-to-Text API. The input is audio data, and the output is transcribed text data.

[0708] Step 4:

[0709] The transcribed text data is summarized by the server's natural language processing (NLP) module. The summarization module extracts the main content and summarizes it in a concise format. The input is the transcribed text data, and the output is the summarized text data.

[0710] Step 5:

[0711] The server extracts key information from the summarized text data, using entity recognition and relationship extraction algorithms to identify key information such as customer names, dates, action items, etc. The input is the summarized text data, and the output is the extracted key information.

[0712] Step 6:

[0713] The server receives emotion data sent from the user device and evaluates the importance of the conversation data based on the emotions recognized by the emotion engine (e.g., Sentiment Analysis API). The input is emotion data and text data, and the output is the importance evaluation result.

[0714] Step 7:

[0715] Based on the evaluation results, the server automatically inputs the extracted information through the API of the sales management tool. The input is important information and the importance evaluation results, and the output is data input into the sales management tool. This is compared with the existing database, and any new information or changes are updated.

[0716] Step 8:

[0717] The server analyzes the sales activity data and generates a dashboard. The analysis results, including sentiment data, are displayed. The input is the sales activity data to be analyzed, and the output is the generated dashboard.

[0718] Step 9:

[0719] Finally, the server notifies the user of the generated dashboard or report. The user receives the notification and can access the latest information. The input is the generated dashboard or report, and the output is the notification sent to the user's device.

[0720] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0721] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0722] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0723] [Third embodiment]

[0724] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0725] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0726] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0727] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0728] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0729] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0730] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0731] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0732] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0733] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0734] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0735] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0736] Step 1:

[0737] User

[0738] Users initiate conversations with customers, whether by phone, online conference, or face-to-face.

[0739] Step 2:

[0740] Terminal

[0741] A dedicated application installed on the user's device captures the audio data of the conversation in real time and sends it to the server.

[0742] Step 3:

[0743] server

[0744] The server sends the received audio data to a transcription engine, which analyzes the audio data and generates corresponding text data.

[0745] Step 4:

[0746] server

[0747] The server passes the generated text data to a natural language processing (NLP) module to generate a summary of the conversation.

[0748] Step 5:

[0749] server

[0750] The server extracts important information (e.g., customer names, dates, action items) from the summarized text using entity recognition and relationship extraction algorithms.

[0751] Step 6:

[0752] server

[0753] The server automatically inputs the extracted information via the sales management tool's API, comparing it with existing customer data and updating any new or changed information.

[0754] Step 7:

[0755] server

[0756] The server analyzes all collected data and generates a dashboard to visualize sales activities.

[0757] Step 8:

[0758] server

[0759] The server prepares the generated dashboards and reports and prepares notifications for the users.

[0760] Step 9:

[0761] Terminal

[0762] The user terminal receives notifications from the server and displays automatically generated sales activity reports and important update information.

[0763] Step 10:

[0764] User

[0765] Users can view new information and reports through their devices and plan their next actions.

[0766] Example 1

[0767] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0768] There is a need to efficiently extract important information from voice data and automatically input it into data management tools such as sales management, thereby reducing the time and effort required for manual input and improving the efficiency and accuracy of sales activities. Furthermore, there is a need to enable sales representatives to make quick decisions through information visualization and real-time information notifications.

[0769] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0770] In this invention, the server includes means for collecting voice data, means for converting the voice data into text data, means for summarizing the voice data, means for extracting important information, means for automatically inputting the voice data into a data management tool, means for comparing the voice data with a database and updating the data with new information, means for analyzing and visualizing the data, and means for notifying the user. This makes it possible to quickly and accurately extract important information from the voice data and automatically input the information into the data management tool. Furthermore, data visualization and real-time notifications improve the efficiency of overall sales activities.

[0771] "Audio data" means digital or analog information relating to audio.

[0772] A "collection means" is a device or method for acquiring audio data and storing or transmitting it in a form that can be further processed.

[0773] "Means for converting into text data" refers to a device or software for converting voice data into text format.

[0774] A "summarizing means" is a device or method for extracting the main points or important information from long text data and creating a shortened version.

[0775] "Significant information" is a data element that is necessary or valuable in a particular context, such as a customer's name or a meeting date.

[0776] A "data management tool" is software or a system for storing, retrieving, organizing, and manipulating data. Examples include sales management tools.

[0777] An "automated input means" is a device or method for inputting extracted information into a data management tool without manual intervention.

[0778] "Means for checking against the database and updating with new information" means a device or method for comparing information in an existing database with newly acquired information and updating the database as necessary.

[0779] A "data analysis and visualization tool" is a device or method for analyzing and visually transforming data in order to display it in an understandable format (e.g., graphs or charts).

[0780] "Notification means" refers to a device or method for notifying a user of new or updated information.

[0781] MODE FOR CARRYING OUT THE INVENTION

[0782] This invention relates to a system that automatically records and processes conversations with customers, primarily during sales activities, and inputs them into a sales management tool. The system of the present invention is equipped with a series of means for collecting conversation data, transcribing it, summarizing it, extracting important information, and automatically inputting it into the sales management tool. It also includes a function for analyzing and visualizing sales activity data and notifying users of updated information.

[0783] System program processing

[0784] The following is a detailed description of how this system works.

[0785] User

[0786] A user initiates a conversation with a customer. The conversation can take place over the phone, via online conference, or in person. For example, if a user initiates a conversation using an online conference tool (e.g., a general video conference tool), the content of the conversation is captured in real time by a dedicated application installed on the user's device.

[0787] Terminal

[0788] The user device sends the captured audio data to the server. Specifically, a dedicated application compresses the audio data and sends it to the server using the HTTPS protocol. For example, the audio data is chunked every 10 seconds and sent continuously with minimal delay.

[0789] server

[0790] The server passes the received voice data to a transcription engine (e.g., a general cloud-based speech recognition API) and converts the voice data into text data. For example, the speech "Please schedule the next meeting for Monday" is converted into text "Please schedule the next meeting for Monday."

[0791] The server then passes the transcribed text data to a natural language processing (NLP) module (e.g., a popular NLP framework) for summarization. For example, a one-hour conversation might be summarized as "agreements on the next meeting date and time and customer needs."

[0792] The server then uses entity recognition and relationship extraction algorithms to extract important information from the summarized text (such as customer names, dates, and action items). For example, from the text "The next meeting with Taro Yamada is scheduled for Monday," the server extracts information such as "Taro Yamada," "next meeting," and "Monday."

[0793] The extracted information is automatically entered into the sales management tool (e.g., general sales management software) via its API. For example, information such as the customer name "Yamada Taro" and the next meeting date "Monday" is added to the database.

[0794] The server also analyzes the sales activity data and generates a dashboard (e.g., a common data visualization tool) to visualize it, for example, by displaying the number of customer meetings last week or a schedule for next week in graphs and tables.

[0795] Finally, the server notifies the user of the generated dashboard or report. For example, when the user opens a dedicated application, the user is notified to display the new information or report.

[0796] Specific examples

[0797] As a concrete example, consider the case where a user is holding an online conference with a client. When the user starts a conversation using a general video conferencing tool, a dedicated application captures the audio data in real time. For example, when the user says, "Please schedule the next meeting for Monday," the audio is captured.

[0798] The dedicated application sends the captured audio data to a server using the HTTPS protocol. The audio data is divided into chunks every 10 seconds and received by the server.

[0799] The server passes the audio data to a transcription engine, which converts it into text: "Please schedule our next meeting for Monday." The server then passes the text data to an NLP module, which generates a summary: "Next meeting: Monday." From that summary, it extracts key information, such as "meeting" and "Monday."

[0800] The server automatically inputs the extracted information using the sales management tool's API and updates the database. The server also generates a dashboard based on the sales activity data and displays a graph of the next week's meeting schedule.

[0801] Finally, the server notifies the user that a report of next week's meeting schedule is available, and the user opens the dedicated application to check the latest information.

[0802] Prompt Sentence Examples

[0803] Below is an example of a prompt sentence to input to the generative AI model.

[0804] After transcribing and summarizing your customer conversation, extract the following information:

[0805] 1. Customer name

[0806] 2. Meeting date and time

[0807] 3. Action Items

[0808] example:

[0809] The client's name is Taro Yamada, and our next meeting can be scheduled for next Monday."

[0810] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0811] Step 1: Capture conversation data

[0812] User

[0813] A user starts a conversation with a customer by telephone, online conference, or face-to-face. For example, when a user starts a conversation using a video conferencing tool, the conversation content is captured in real time by a dedicated application installed on the user's terminal. The input is voice data, and the output is the captured digitized voice data.

[0814] Step 2: Sending audio data

[0815] Terminal

[0816] The user device sends the captured audio data to the server. Specifically, a dedicated application compresses the audio data and sends it to the server using the HTTPS protocol. For example, the audio data is divided into chunks every 10 seconds and sent continuously with minimal delay. The input is the captured audio data, and the output is the audio data sent to the server.

[0817] Step 3: Transcribe the audio data

[0818] server

[0819] The server passes the received voice data to a transcription engine (for example, a general cloud-based voice recognition API) and converts the voice data into text data. For example, voice data such as "Please schedule the next meeting for Monday" is converted into text data such as "Please schedule the next meeting for Monday." The input is the voice data sent to the server, and the output is the transcribed text data.

[0820] Step 4: Summarizing the text data

[0821] server

[0822] The server passes the transcribed text data to a natural language processing (NLP) module (e.g., a general NLP framework) for summarization. For example, a one-hour conversation is summarized as "agreement on the date and time of the next meeting and the customer's requests." The input is the transcribed text data, and the output is the summarized text data.

[0823] Step 5: Extracting important information

[0824] server

[0825] The server uses entity recognition and relationship extraction algorithms to extract important information (such as customer names, dates, and action items) from the summarized text. For example, from the text "The next meeting with Yamada Taro is scheduled for Monday," the server extracts information such as "Yamada Taro," "next meeting," and "Monday." The input is the summarized text data, and the output is the extracted important information.

[0826] Step 6: Enter information into your sales management tool

[0827] server

[0828] The server automatically inputs the extracted important information through the API of a sales management tool (e.g., general sales management software). For example, information such as "Customer name: Yamada Taro, next meeting: Monday" is added to the database. The input is the extracted important information, and the output is the information entered into the sales management tool.

[0829] Step 7: Analyze and visualize the data

[0830] server

[0831] The server analyzes the sales activity data and generates a dashboard (e.g., a common data visualization tool) to visualize it. For example, it displays the number of customer meetings last week or the schedule for next week in graphs and tables. The input is the sales activity data, and the output is the generated dashboard.

[0832] Step 8: Report Notification

[0833] server

[0834] The server notifies the user terminal of the generated dashboard or report. For example, when the user opens a dedicated application, the server notifies the user to display new information or reports. The input is the generated dashboard or report, and the output is the notified information.

[0835] (Application example 1)

[0836] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0837] When interacting with customers in a virtual store, it is necessary to efficiently respond to customer questions and requests, and to automatically record, process, and manage the content of the conversation. Currently, these tasks are often done manually, which is time-consuming and labor-intensive, so there is an urgent need to create an efficient system.

[0838] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0839] In this invention, the server includes means for collecting conversation data, means for transcribing the collected conversation data, means for summarizing the transcribed text data, means for extracting important information from the summarized text data, means for automatically inputting the extracted information into a customer management system, means for analyzing and visualizing customer interaction data, and means for notifying users of updated information. This makes it possible to automate everything from recording to managing customer interactions, thereby improving the efficiency and accuracy of customer interactions.

[0840] "Conversation Data" means records of voice or text communications with customers.

[0841] "Means of collection" refers to the equipment or software that captures the voice and text data necessary to interact with customers.

[0842] "Transcription method" refers to the process or technique for converting audio data into text data.

[0843] "Summarization methods" refers to techniques and processes for extracting key information from transcribed text data and summarizing it concisely.

[0844] "Key information extraction methods" are techniques and methods for identifying and extracting specific entities or action items from summarized text data.

[0845] A "customer management system" refers to software or a platform for centrally managing customer information and transaction history.

[0846] "Means for automatic input" refers to methods or technologies for mechanically registering extracted information into a customer management system rather than manually.

[0847] "Analysis and visualization means" refers to techniques and methods for analyzing collected data and displaying it in a visual format such as graphs or charts.

[0848] "Means for notifying users of updated information" refers to techniques and methods for notifying users when data is updated or new information is obtained.

[0849] A "virtual store" is a virtual shopping environment or service provided over the Internet.

[0850] A "virtual assistant" is a character or bot equipped with artificial intelligence that interacts with customers and provides services within a virtual store.

[0851] This invention relates to a system for automating and efficiently managing customer interactions in a virtual store. The system of the present invention is equipped with a series of means for collecting and recording customer conversations, transcribing them, summarizing them, extracting important information, and automatically inputting them into a customer management system. It also includes a function for analyzing and visualizing the collected customer interaction data and notifying users of updated information.

[0852] System program processing

[0853] This system is realized using the following hardware and software.

[0854] Hardware:

[0855] User devices for customer service (e.g., smartphones and desktop PCs)

[0856] Microphone (for voice input)

[0857] Server (for data processing and management)

[0858] software:

[0859] speech_recognition library: Captures and transcribes speech data.

[0860] The transformers library: performs natural language processing (NLP).

[0861] Virtual CRM API module: Automatically registers extracted important information into a customer management system.

[0862] The server receives customer conversation data from the user's device and converts the speech to text in real time using a transcription engine. This text data is temporarily stored and then summarized using a natural language processing (NLP) module. From the summarized text, the server uses entity recognition and relationship extraction algorithms to extract important information (e.g., customer name, date, action items, etc.).

[0863] The extracted information is automatically entered into the customer management system via API. At this time, it is compared with the existing database, and if there is new information or changes, the database is updated. The server also analyzes the customer interaction data and visualizes it as a dashboard. This dashboard is provided to the user as a report, and notifications are sent whenever the information is updated.

[0864] Specific examples

[0865] As a concrete example, consider the case where a customer asks a question to a virtual assistant in a virtual store. The customer's question is captured using the microphone of the user terminal. For example, the question might be, "Please tell me the features of this product."

[0866] User

[0867] When a customer initiates a conversation with the virtual assistant, the conversation is captured in real time by the user terminal.

[0868] Terminal

[0869] The captured audio data is sent from the user device to the server, where it is converted into text using a transcription library.

[0870] server

[0871] The converted text data is summarized using a natural language processing module. For example, a summary such as "This product is lightweight and durable" is generated. The NLP module extracts important information (product features, durability, lightweight) and automatically inputs it into a customer management system.

[0872] In this way, users can improve the efficiency of customer service within the virtual store and automate a series of tasks from recording to management.

[0873] Example prompt sentence:

[0874] "Hello, what products are you looking for today?"

[0875] "What's your favorite thing about the items you purchased today?"

[0876] "Please tell me the features of this product."

[0877] This system makes it possible to improve the efficiency and accuracy of customer service within the virtual store.

[0878] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0879] Step 1:

[0880] Input: Customer's voice or text question

[0881] How it works: A user initiates a conversation with a customer through a virtual assistant in a virtual store. For example, the customer uses the microphone to ask, "What are the features of this product?"

[0882] Output: Raw captured audio or text data

[0883] Step 2:

[0884] Input: Raw audio data

[0885] How it works: The device uses a microphone to capture the customer's speech, and the captured voice data is sent directly to the device for further processing.

[0886] Output: Audio data stored on the device

[0887] Step 3:

[0888] Input: Audio data stored on the device

[0889] How it works: The device transcribes audio data using the speech_recognition library, specifically the Google Speech Recognition API, which converts audio data into text.

[0890] Output: Converted text data

[0891] Step 4:

[0892] Input: Converted text data

[0893] How it works: The server uses the transformers library to summarize text data. Specifically, it uses a generative AI model (e.g., the BART model) to convert input text into a summary sentence.

[0894] Output: Summarized text data

[0895] Step 5:

[0896] Input: Summarized text data

[0897] How it works: The server uses a natural language processing (NLP) module to extract key information, specifically applying entity recognition and relationship extraction algorithms to identify information like customer names, dates, action items, etc.

[0898] Output: Extracted key information (e.g. customer name, date, action items)

[0899] Step 6:

[0900] Input: Extracted important information

[0901] How it works: The server automatically inputs the extracted information through the customer management system's API, cross-checks it against existing data, and updates the database with any new or changed information.

[0902] Output: Updated customer relationship management system database

[0903] Step 7:

[0904] Input: Updated customer management system database

[0905] How it works: The server analyzes customer interaction data and generates a dashboard to visually display it. It visualizes data trends and statistics and provides them to the user as reports.

[0906] Output: Generated dashboards and reports

[0907] Step 8:

[0908] Input: Generated dashboards and reports

[0909] How it works: The server sends updated information to the user's device, allowing the user to check new information and analysis results through the notifications they receive.

[0910] Output: Notification sent to the user

[0911] The above are the specific processing steps of the system of the present invention, which automate customer service in the virtual store, significantly improving efficiency and accuracy.

[0912] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0913] Step 1:

[0914] User

[0915] The user initiates a conversation with the customer. This conversation can take place over the phone, an online conference, or in person. The user conducts the conversation using a specialized app.

[0916] Step 2:

[0917] Terminal

[0918] An application installed on the device captures the audio data of the conversation in real time and sends it to a server.

[0919] Step 3:

[0920] Terminal

[0921] At the same time, the emotion engine recognizes emotions from the user's voice in real time, generates emotion data, and sends it to the server.

[0922] Step 4:

[0923] server

[0924] The server passes the received voice data to a transcription engine, which converts the voice data into text data. For example, a conversation like "The client's name is Taro Yamada, and our next meeting is scheduled for next Monday" is generated as text data.

[0925] Step 5:

[0926] server

[0927] The server passes the generated text data to a natural language processing (NLP) module to generate a summary of the conversation, in this case "The next meeting with Taro Yamada is on Monday."

[0928] Step 6:

[0929] server

[0930] The server uses entity recognition and relation extraction algorithms to extract important information (such as customer name, date, and action item) from the summarized text. For example, the customer name is "Yamada Taro," the date is "next Monday," and the action item is "schedule a meeting."

[0931] Step 7:

[0932] server

[0933] The server receives the emotion data sent by the user and adds the emotion information to the extracted important information. For example, if the user indicates "joy," the emotion data is added as "joy."

[0934] Step 8:

[0935] server

[0936] The server automatically inputs this information via the sales management tool's API, matching it with existing customer data and updating the database as necessary.

[0937] Step 9:

[0938] server

[0939] The server analyzes sales activities based on all data and generates a dashboard containing sentiment data, which displays detailed information about sales activity performance and customer reactions.

[0940] Step 10:

[0941] server

[0942] The server sends the generated dashboard and report to the user's terminal, allowing the user to check the latest information.

[0943] Step 11:

[0944] Terminal

[0945] The user's device receives notifications from the server and displays dashboards and reports, allowing the user to plan their next actions.

[0946] Through these steps, the system of the present invention automates data recording and input work, significantly improving work efficiency while providing an environment in which salespeople can concentrate on conversation. Furthermore, by utilizing emotion data, it is expected to improve the quality of sales activities and increase customer satisfaction.

[0947] Example 2

[0948] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0949] In modern sales activities, it is important for sales representatives to record and manage detailed conversations with customers. However, this recording process is often done manually, prone to human error, and requires time and effort. It is also difficult to properly evaluate customer emotions and important information expressed during conversations and reflect them in sales management tools. This reduces the efficiency and quality of sales activities and affects customer satisfaction.

[0950] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for collecting conversation data, a means for transcribing, a means for summarizing, a means for extracting important information, a means for automatically inputting the data into a sales management tool, a means for recognizing emotions, and a means for evaluating the data. This enables accurate recording and management of conversation data, automates the evaluation of importance based on emotion data, and improves the efficiency and quality of sales activities.

[0951] "Conversation data" refers to the content of voice communication between a user and a customer.

[0952] "Transcription" is the process of analyzing audio data and converting its content into text data.

[0953] "Summarization" is the act of concisely summarizing transcribed text data and extracting the main points.

[0954] "Important information" refers to specific elements or facts extracted from conversation data, and is data that is necessary and useful in sales activities.

[0955] A "sales management tool" is software or an application for managing customer information, sales activity progress, results, etc.

[0956] "Auto-fill" is the process by which data is automatically entered by the system without manual input.

[0957] An "emotion engine" is a technology that recognizes a user's emotions from voice data or text data and extracts them as data.

[0958] "Emotion data" is data that represents the emotional state of the user as recognized by the emotion engine.

[0959] "Evaluation" is the act of assigning importance or priority to a conversation based on emotional data and other information.

[0960] "Matching" is the process of comparing new data with information in an existing database to identify matches.

[0961] "Update" is an operation that keeps existing data up to date with new information.

[0962] "Visualization" is the act of displaying analyzed data in a visual format such as a graph or chart to make it easier to understand.

[0963] This invention is a system that automatically records and manages conversations with customers during sales activities. This system has the functions of collecting conversation data, transcribing it, summarizing it, extracting important information, and automatically inputting it into a sales management tool. It also has the functions of analyzing and visualizing sales activity data and notifying users of updated information. Furthermore, it includes a function that combines an emotion engine that recognizes user emotions, uses emotion data to evaluate the importance of the conversation, and reflects this in the sales management tool.

[0964] System configuration

[0965] User terminal

[0966] The user terminal includes hardware for capturing voice data and dedicated application software for processing the data in real time. As the voice data is collected, an emotion engine recognizes the user's emotions and adds them to the data.

[0967] server

[0968] The server is equipped with a high-performance speech recognition engine, a natural language processing (NLP) module, a database management system, and an interface (API) with sales management tools.

[0969] 1. Capture and transmit audio data

[0970] When a user starts a conversation with a customer, a dedicated application installed on the user's terminal captures the voice data in real time and adds emotional data using an emotion engine.

[0971] This data is immediately sent to the server.

[0972] 2. Transcription and Summarization

[0973] The server passes the received audio data to a transcription engine, which converts the audio into text data. For example, a speech that says, "Hello, Yamada-san. I'd like to talk about our next meeting." is converted into text data.

[0974] The transcribed text data is sent to an NLP module, where the main points are summarized.

[0975] 3. Extracting important information

[0976] From the summarized text, the server uses entity recognition and relationship extraction algorithms to extract key information, such as customer names, dates, and action items.

[0977] 4. Evaluating Emotional Data

[0978] Emotion data sent from the user terminal is received and evaluated by the emotion engine, which determines the importance of the conversation data.

[0979] 5. Input and update to sales management tools

[0980] The server automatically inputs the extracted information and evaluation results through the sales management tool's API, comparing them with the existing database and updating any new information.

[0981] 6. Data Analysis and Visualization

[0982] The server analyzes sales activity data and generates and visualizes a dashboard that includes sentiment data, allowing users to get a detailed overview of sales activities.

[0983] 7. Notification

[0984] Finally, the generated dashboards and reports are sent to the user's device, allowing the user to check the latest information in real time.

[0985] Specific examples

[0986] As a specific example, consider a case where a user is having a telephone conversation with a customer.

[0987] User

[0988] When a user starts a call with a customer, the conversation is captured in real time by an application installed on the user's device. If the user shows strong emotions (e.g., joy, anger, etc.), the emotion engine recognizes the emotion and sends it to the server as emotion data.

[0989] Terminal

[0990] The captured voice data and emotional data are sent from the user's device to the server, allowing the content of the conversation and the user's emotional state to be instantly transmitted to the server.

[0991] server

[0992] The server immediately passes the received voice data to a transcription engine. For example, if a user says, "The next meeting is very important," the transcription engine converts this into text data. Next, a natural language processing module converts this text data into a summary such as "The next meeting is important" and extracts important information. The importance is evaluated taking into account emotional data, and the results are automatically entered into a sales management tool. The information is then compared with an existing customer database, and any new information is updated.

[0993] Prompt Sentence Examples

[0994] Below are some example prompts to input to a generative AI model:

[0995] "Please tell me the procedure for a system that automatically records conversations with customers and inputs emotional data into a sales management tool."

[0996] "Please explain each step of the process when a user starts a conversation with a customer."

[0997] "Can you tell me more about how the emotion engine works and how to feed data into a sales management tool?"

[0998] As a result, the system of the present invention provides an environment where salespeople can concentrate on the conversation, significantly improving work efficiency and quality. Furthermore, by utilizing emotion data, it is expected that customer satisfaction will be improved.

[0999] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1000] Step 1:

[1001] A user initiates a conversation with a customer. The conversation may take place over the phone, an online conference, or in person. A dedicated application installed on the user's device captures the conversation data in real time. The input is voice data. The output is voice data that is sent to the server in real time. Specifically, when the user says "hello," the voice is captured by the application.

[1002] Step 2:

[1003] The device sends the captured voice data along with the emotion data generated by the emotion engine to the server. The input is the captured voice data and emotion data, and the output is the data sent to the server. In concrete terms, if a user expresses high emotion by saying, "The next meeting is very important," that emotion is also sent as data to the server.

[1004] Step 3:

[1005] The server sends the received voice data to the transcription engine. The input is voice data and the output is text data. Specifically, the server converts the voice data "Hello, Yamada-san. I'd like to talk about our next meeting." into text data "Hello, Yamada-san. I'd like to talk about our next meeting."

[1006] Step 4:

[1007] The server sends the transcribed text data to a natural language processing (NLP) module for summarization. The input is the transcribed text data, and the output is the summarized text data. Specifically, the server converts the text data "I'd like to talk about our next meeting" into the summary "Meeting setup."

[1008] Step 5:

[1009] The server extracts important information from the summarized text data using entity recognition and relation extraction algorithms. The input is the summarized text data, and the output is important information (e.g., customer name, date, action item, etc.). Specifically, the server converts the summary "Meeting setup" into "Customer name: Yamada, action item: Meeting setup".

[1010] Step 6:

[1011] The server analyzes the emotion data sent from the user terminal and evaluates the importance of the conversation data. The input is emotion data, and the output is the evaluation result. Specifically, the server evaluates emotion data that is "important" as "high priority."

[1012] Step 7:

[1013] The server automatically inputs the extracted information and evaluation results through the API of the sales management tool. The input is the extracted information and evaluation results, and the output is the updated data in the sales management tool. Specifically, the server automatically reflects "Meeting arrangement with Yamada" in the sales management tool.

[1014] Step 8:

[1015] The server analyzes sales activity data and generates and visualizes a dashboard. The input is sales activity data, and the output is the visualized dashboard. Specifically, the server visualizes the data "Setting up a meeting with Yamada" in graphs and charts.

[1016] Step 9:

[1017] The server notifies the user device of the generated dashboard or report. The input is the visualized data, and the output is a notification to the user device. Specifically, the server notifies the user device of the "next meeting setting" information as an alert.

[1018] (Application example 2)

[1019] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1020] In conventional sales activities, accurately recording conversations with customers, converting them into data, and entering them into sales management tools is labor-intensive and prone to typographical errors and overlooking important information. Furthermore, because emotional data is processed without consideration, it is difficult to create sales strategies that reflect the customer's true feelings and emotions. As a result, there are limitations to improving the efficiency of sales activities and customer satisfaction. A system that solves these problems is needed.

[1021] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting conversation data, means for transcribing the collected conversation data, means for summarizing the transcribed text data, means for extracting important information from the summarized text data, means for automatically inputting the extracted information into a sales management tool, means for analyzing and visualizing sales activity data, means for notifying the user of update information, means for recognizing emotional data of the conversation, means for evaluating the importance of the conversation data based on the emotional data, and means for reflecting the evaluation results in the sales management tool. This makes it possible to accurately record the content of conversations with customers and their emotional data at the time, and automatically input them into the sales management tool. This makes it possible to improve the efficiency of sales activities and customer satisfaction.

[1022] "Conversation data" is information relating to the verbal exchanges that take place between a customer and a user.

[1023] "Means for collection" refers to devices or software for acquiring conversation data with customers and inputting it into the system.

[1024] "Transcribing means" means speech recognition technology or software for converting collected audio data into text data.

[1025] A "summarization tool" is a natural language processing technique or software that extracts key points from transcribed text data and summarizes them in a concise format.

[1026] A "means for extracting key information" is technology or software that identifies key elements from the summarized text data, such as customer names, dates, and action items to track.

[1027] "Means for automatically inputting information into a sales management tool" refers to a device or software that automatically reflects extracted important information in a sales management system (such as CRM).

[1028] "Means for analysis" refers to technology or software used to analyze collected sales activity data and evaluate its performance and trends.

[1029] "Visualization means" refers to technology or software for displaying analyzed data in a visually easy-to-understand format, such as a graph or chart.

[1030] "Means for notifying updated information" refers to technology or software that automatically notifies users of the latest sales activity data and analysis results.

[1031] "Means for recognizing emotional data" refers to technology or software for detecting emotional states from collected conversation data.

[1032] The "means for assessing importance" is a technology or software for assessing the content and importance of a conversation based on emotional data.

[1033] "Database matching means" means technology or software used to compare newly collected information with existing customer databases to identify matches.

[1034] "Means for updating information" refers to technology or software for updating the database in the sales management tool with the latest information based on the collated data.

[1035] This invention relates to a system that automatically records and processes conversations between salespeople and customers, primarily during sales activities in brick-and-mortar stores, and inputs the content into a sales management tool. The system of the present invention is equipped with a series of means for collecting conversation data, transcribing it, summarizing it, extracting important information, and automatically inputting it into the sales management tool. It can also recognize emotional data in the conversation, evaluate the importance of the conversation based on that information, and reflect the evaluation results in the sales management tool.

[1036] User

[1037] A salesperson (user) starts a conversation with a customer. This conversation is usually conducted face-to-face. A dedicated application is installed on the user's device, and this application captures conversation data in real time and sends it to a server.

[1038] Terminal

[1039] The user device transmits the captured voice data to the server, which instantly transmits the conversation data to the server. The device is equipped with an emotion engine that recognizes emotions from the voice data in real time.

[1040] server

[1041] The server sends the received audio data to a transcription engine using the Google Cloud Speech-to-Text API. The server converts the audio data into text and then summarizes it using a natural language processing (NLP) module. The transcription and summary data are temporarily stored. The server then uses entity recognition and relationship extraction algorithms to extract important information from the summary (e.g., customer name, date, action items, etc.).

[1042] Next, the server receives the emotion data sent from the user's device and evaluates the importance of the conversation data based on the emotions recognized by the emotion engine (e.g., Sentiment Analysis API). Based on this evaluation result, the extracted information is automatically entered into the sales management tool (e.g., Salesforce) via its API. The sales management tool compares the data with the existing database and updates it if any new information or changes are found.

[1043] The server also analyzes sales activity data and generates and visualizes dashboards. The generated dashboards also include sentiment data, enabling a more detailed understanding of the overall picture of sales activities. Finally, the server notifies the user of the generated dashboards and reports on their devices, allowing the user to access the latest information.

[1044] Specific examples

[1045] As a concrete example, consider a scenario in which a salesperson in a physical store talks with a customer about a new product. The salesperson asks, "What do you think about the features of this new product?" and the customer replies, "I think it's very attractive." This conversation is captured in real time using the smartphone's microphone. The captured voice data is immediately sent to a server, where a transcription engine converts it into text data such as, "What do you think about the features of this new product? I think it's very attractive."

[1046] The natural language processing module summarizes this as "the new product's features are attractive," and the entity recognition algorithm extracts the important information "new product" and "attractive." The emotion engine also recognizes that the customer is expressing "positive emotion" and reflects this in the evaluation results. All data is entered into the sales management tool by the server, where it is compared and updated with existing data. The analysis results of this conversation are visualized and displayed on a dashboard such as Salesforce.

[1047] Example prompts to input to the generative AI model

[1048] "Generate a program that transcribes customer voice data and analyzes sentiment data."

[1049] "Please provide sample code for a system that automatically records customer conversations and inputs key information into a sales management tool."

[1050] As described above, the system of the present invention allows salespeople to concentrate on talking to customers and greatly improves work efficiency by automating data recording and input. In addition, by using emotional data, it is possible to grasp customer reactions in real time, providing a valuable clue for appropriate responses.

[1051] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1052] Step 1:

[1053] The user starts a conversation with the customer. The conversation data is captured in real time by an application installed on the user's terminal. The input is the voice conversation between the customer and the user, and the output is the captured voice data.

[1054] Step 2:

[1055] The terminal transmits the captured voice data to the server. The input is the captured voice data, and the output is the transmission of the voice data to the server. This step ensures that the conversation data is immediately transmitted to the server.

[1056] Step 3:

[1057] The server sends the received audio data to a transcription engine, which converts the audio into text using the Google Cloud Speech-to-Text API. The input is audio data, and the output is transcribed text data.

[1058] Step 4:

[1059] The transcribed text data is summarized by the server's natural language processing (NLP) module. The summarization module extracts the main content and summarizes it in a concise format. The input is the transcribed text data, and the output is the summarized text data.

[1060] Step 5:

[1061] The server extracts key information from the summarized text data, using entity recognition and relationship extraction algorithms to identify key information such as customer names, dates, action items, etc. The input is the summarized text data, and the output is the extracted key information.

[1062] Step 6:

[1063] The server receives emotion data sent from the user device and evaluates the importance of the conversation data based on the emotions recognized by the emotion engine (e.g., Sentiment Analysis API). The input is emotion data and text data, and the output is the importance evaluation result.

[1064] Step 7:

[1065] Based on the evaluation results, the server automatically inputs the extracted information through the API of the sales management tool. The input is important information and the importance evaluation results, and the output is data input into the sales management tool. This is compared with the existing database, and any new information or changes are updated.

[1066] Step 8:

[1067] The server analyzes the sales activity data and generates a dashboard. The analysis results, including sentiment data, are displayed. The input is the sales activity data to be analyzed, and the output is the generated dashboard.

[1068] Step 9:

[1069] Finally, the server notifies the user of the generated dashboard or report. The user receives the notification and can access the latest information. The input is the generated dashboard or report, and the output is the notification sent to the user's device.

[1070] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1071] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1072] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1073] [Fourth embodiment]

[1074] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1075] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1076] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1077] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1078] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1079] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1080] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1081] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1082] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1083] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1084] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1085] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1086] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1087] Step 1:

[1088] User

[1089] Users initiate conversations with customers, whether by phone, online conference, or face-to-face.

[1090] Step 2:

[1091] Terminal

[1092] A dedicated application installed on the user's device captures the audio data of the conversation in real time and sends it to the server.

[1093] Step 3:

[1094] server

[1095] The server sends the received audio data to a transcription engine, which analyzes the audio data and generates corresponding text data.

[1096] Step 4:

[1097] server

[1098] The server passes the generated text data to a natural language processing (NLP) module to generate a summary of the conversation.

[1099] Step 5:

[1100] server

[1101] The server extracts important information (e.g., customer names, dates, action items) from the summarized text using entity recognition and relationship extraction algorithms.

[1102] Step 6:

[1103] server

[1104] The server automatically inputs the extracted information via the sales management tool's API, comparing it with existing customer data and updating any new or changed information.

[1105] Step 7:

[1106] server

[1107] The server analyzes all collected data and generates a dashboard to visualize sales activities.

[1108] Step 8:

[1109] server

[1110] The server prepares the generated dashboards and reports and prepares notifications for the users.

[1111] Step 9:

[1112] Terminal

[1113] The user terminal receives notifications from the server and displays automatically generated sales activity reports and important update information.

[1114] Step 10:

[1115] User

[1116] Users can view new information and reports through their devices and plan their next actions.

[1117] Example 1

[1118] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1119] There is a need to efficiently extract important information from voice data and automatically input it into data management tools such as sales management, thereby reducing the time and effort required for manual input and improving the efficiency and accuracy of sales activities. Furthermore, there is a need to enable sales representatives to make quick decisions through information visualization and real-time information notifications.

[1120] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1121] In this invention, the server includes means for collecting voice data, means for converting the voice data into text data, means for summarizing the voice data, means for extracting important information, means for automatically inputting the voice data into a data management tool, means for comparing the voice data with a database and updating the data with new information, means for analyzing and visualizing the data, and means for notifying the user. This makes it possible to quickly and accurately extract important information from the voice data and automatically input the information into the data management tool. Furthermore, data visualization and real-time notifications improve the efficiency of overall sales activities.

[1122] "Audio data" means digital or analog information relating to audio.

[1123] A "collection means" is a device or method for acquiring audio data and storing or transmitting it in a form that can be further processed.

[1124] "Means for converting into text data" refers to a device or software for converting voice data into text format.

[1125] A "summarizing means" is a device or method for extracting the main points or important information from long text data and creating a shortened version.

[1126] "Significant information" is a data element that is necessary or valuable in a particular context, such as a customer's name or a meeting date.

[1127] A "data management tool" is software or a system for storing, retrieving, organizing, and manipulating data. Examples include sales management tools.

[1128] An "automated input means" is a device or method for inputting extracted information into a data management tool without manual intervention.

[1129] "Means for checking against the database and updating with new information" means a device or method for comparing information in an existing database with newly acquired information and updating the database as necessary.

[1130] A "data analysis and visualization tool" is a device or method for analyzing and visually transforming data in order to display it in an understandable format (e.g., graphs or charts).

[1131] "Notification means" refers to a device or method for notifying a user of new or updated information.

[1132] MODE FOR CARRYING OUT THE INVENTION

[1133] This invention relates to a system that automatically records and processes conversations with customers, primarily during sales activities, and inputs them into a sales management tool. The system of the present invention is equipped with a series of means for collecting conversation data, transcribing it, summarizing it, extracting important information, and automatically inputting it into the sales management tool. It also includes a function for analyzing and visualizing sales activity data and notifying users of updated information.

[1134] System program processing

[1135] The following is a detailed description of how this system works.

[1136] User

[1137] A user initiates a conversation with a customer. The conversation can take place over the phone, via online conference, or in person. For example, if a user initiates a conversation using an online conference tool (e.g., a general video conference tool), the content of the conversation is captured in real time by a dedicated application installed on the user's device.

[1138] Terminal

[1139] The user device sends the captured audio data to the server. Specifically, a dedicated application compresses the audio data and sends it to the server using the HTTPS protocol. For example, the audio data is chunked every 10 seconds and sent continuously with minimal delay.

[1140] server

[1141] The server passes the received voice data to a transcription engine (e.g., a general cloud-based speech recognition API) and converts the voice data into text data. For example, the speech "Please schedule the next meeting for Monday" is converted into text "Please schedule the next meeting for Monday."

[1142] The server then passes the transcribed text data to a natural language processing (NLP) module (e.g., a popular NLP framework) for summarization. For example, a one-hour conversation might be summarized as "agreements on the next meeting date and time and customer needs."

[1143] The server then uses entity recognition and relationship extraction algorithms to extract important information from the summarized text (such as customer names, dates, and action items). For example, from the text "The next meeting with Taro Yamada is scheduled for Monday," the server extracts information such as "Taro Yamada," "next meeting," and "Monday."

[1144] The extracted information is automatically entered into the sales management tool (e.g., general sales management software) via its API. For example, information such as the customer name "Yamada Taro" and the next meeting date "Monday" is added to the database.

[1145] The server also analyzes the sales activity data and generates a dashboard (e.g., a common data visualization tool) to visualize it, for example, by displaying the number of customer meetings last week or a schedule for next week in graphs and tables.

[1146] Finally, the server notifies the user of the generated dashboard or report. For example, when the user opens a dedicated application, the user is notified to display the new information or report.

[1147] Specific examples

[1148] As a concrete example, consider the case where a user is holding an online conference with a client. When the user starts a conversation using a general video conferencing tool, a dedicated application captures the audio data in real time. For example, when the user says, "Please schedule the next meeting for Monday," the audio is captured.

[1149] The dedicated application sends the captured audio data to a server using the HTTPS protocol. The audio data is divided into chunks every 10 seconds and received by the server.

[1150] The server passes the audio data to a transcription engine, which converts it into text: "Please schedule our next meeting for Monday." The server then passes the text data to an NLP module, which generates a summary: "Next meeting: Monday." From that summary, it extracts key information, such as "meeting" and "Monday."

[1151] The server automatically inputs the extracted information using the sales management tool's API and updates the database. The server also generates a dashboard based on the sales activity data and displays a graph of the next week's meeting schedule.

[1152] Finally, the server notifies the user that a report of next week's meeting schedule is available, and the user opens the dedicated application to check the latest information.

[1153] Prompt Sentence Examples

[1154] Below is an example of a prompt sentence to input to the generative AI model.

[1155] After transcribing and summarizing your customer conversation, extract the following information:

[1156] 1. Customer name

[1157] 2. Meeting date and time

[1158] 3. Action Items

[1159] example:

[1160] The client's name is Taro Yamada, and our next meeting can be scheduled for next Monday."

[1161] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1162] Step 1: Capture conversation data

[1163] User

[1164] A user starts a conversation with a customer by telephone, online conference, or face-to-face. For example, when a user starts a conversation using a video conferencing tool, the conversation content is captured in real time by a dedicated application installed on the user's terminal. The input is voice data, and the output is the captured digitized voice data.

[1165] Step 2: Sending audio data

[1166] Terminal

[1167] The user device sends the captured audio data to the server. Specifically, a dedicated application compresses the audio data and sends it to the server using the HTTPS protocol. For example, the audio data is divided into chunks every 10 seconds and sent continuously with minimal delay. The input is the captured audio data, and the output is the audio data sent to the server.

[1168] Step 3: Transcribe the audio data

[1169] server

[1170] The server passes the received voice data to a transcription engine (for example, a general cloud-based voice recognition API) and converts the voice data into text data. For example, voice data such as "Please schedule the next meeting for Monday" is converted into text data such as "Please schedule the next meeting for Monday." The input is the voice data sent to the server, and the output is the transcribed text data.

[1171] Step 4: Summarizing the text data

[1172] server

[1173] The server passes the transcribed text data to a natural language processing (NLP) module (e.g., a general NLP framework) for summarization. For example, a one-hour conversation is summarized as "agreement on the date and time of the next meeting and the customer's requests." The input is the transcribed text data, and the output is the summarized text data.

[1174] Step 5: Extracting important information

[1175] server

[1176] The server uses entity recognition and relationship extraction algorithms to extract important information (such as customer names, dates, and action items) from the summarized text. For example, from the text "The next meeting with Yamada Taro is scheduled for Monday," the server extracts information such as "Yamada Taro," "next meeting," and "Monday." The input is the summarized text data, and the output is the extracted important information.

[1177] Step 6: Enter information into your sales management tool

[1178] server

[1179] The server automatically inputs the extracted important information through the API of a sales management tool (e.g., general sales management software). For example, information such as "Customer name: Yamada Taro, next meeting: Monday" is added to the database. The input is the extracted important information, and the output is the information entered into the sales management tool.

[1180] Step 7: Analyze and visualize the data

[1181] server

[1182] The server analyzes the sales activity data and generates a dashboard (e.g., a common data visualization tool) to visualize it. For example, it displays the number of customer meetings last week or the schedule for next week in graphs and tables. The input is the sales activity data, and the output is the generated dashboard.

[1183] Step 8: Report Notification

[1184] server

[1185] The server notifies the user terminal of the generated dashboard or report. For example, when the user opens a dedicated application, the server notifies the user to display new information or reports. The input is the generated dashboard or report, and the output is the notified information.

[1186] (Application example 1)

[1187] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1188] When interacting with customers in a virtual store, it is necessary to efficiently respond to customer questions and requests, and to automatically record, process, and manage the content of the conversation. Currently, these tasks are often done manually, which is time-consuming and labor-intensive, so there is an urgent need to create an efficient system.

[1189] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1190] In this invention, the server includes means for collecting conversation data, means for transcribing the collected conversation data, means for summarizing the transcribed text data, means for extracting important information from the summarized text data, means for automatically inputting the extracted information into a customer management system, means for analyzing and visualizing customer interaction data, and means for notifying users of updated information. This makes it possible to automate everything from recording to managing customer interactions, thereby improving the efficiency and accuracy of customer interactions.

[1191] "Conversation Data" means records of voice or text communications with customers.

[1192] "Means of collection" refers to the equipment or software that captures the voice and text data necessary to interact with customers.

[1193] "Transcription method" refers to the process or technique for converting audio data into text data.

[1194] "Summarization methods" refers to techniques and processes for extracting key information from transcribed text data and summarizing it concisely.

[1195] "Key information extraction methods" are techniques and methods for identifying and extracting specific entities or action items from summarized text data.

[1196] A "customer management system" refers to software or a platform for centrally managing customer information and transaction history.

[1197] "Means for automatic input" refers to methods or technologies for mechanically registering extracted information into a customer management system rather than manually.

[1198] "Analysis and visualization means" refers to techniques and methods for analyzing collected data and displaying it in a visual format such as graphs or charts.

[1199] "Means for notifying users of updated information" refers to techniques and methods for notifying users when data is updated or new information is obtained.

[1200] A "virtual store" is a virtual shopping environment or service provided over the Internet.

[1201] A "virtual assistant" is a character or bot equipped with artificial intelligence that interacts with customers and provides services within a virtual store.

[1202] This invention relates to a system for automating and efficiently managing customer interactions in a virtual store. The system of the present invention is equipped with a series of means for collecting and recording customer conversations, transcribing them, summarizing them, extracting important information, and automatically inputting them into a customer management system. It also includes a function for analyzing and visualizing the collected customer interaction data and notifying users of updated information.

[1203] System program processing

[1204] This system is realized using the following hardware and software.

[1205] Hardware:

[1206] User devices for customer service (e.g., smartphones and desktop PCs)

[1207] Microphone (for voice input)

[1208] Server (for data processing and management)

[1209] software:

[1210] speech_recognition library: Captures and transcribes speech data.

[1211] The transformers library: performs natural language processing (NLP).

[1212] Virtual CRM API module: Automatically registers extracted important information into a customer management system.

[1213] The server receives customer conversation data from the user's device and converts the speech to text in real time using a transcription engine. This text data is temporarily stored and then summarized using a natural language processing (NLP) module. From the summarized text, the server uses entity recognition and relationship extraction algorithms to extract important information (e.g., customer name, date, action items, etc.).

[1214] The extracted information is automatically entered into the customer management system via API. At this time, it is compared with the existing database, and if there is new information or changes, the database is updated. The server also analyzes the customer interaction data and visualizes it as a dashboard. This dashboard is provided to the user as a report, and notifications are sent whenever the information is updated.

[1215] Specific examples

[1216] As a concrete example, consider the case where a customer asks a question to a virtual assistant in a virtual store. The customer's question is captured using the microphone of the user terminal. For example, the question might be, "Please tell me the features of this product."

[1217] User

[1218] When a customer initiates a conversation with the virtual assistant, the conversation is captured in real time by the user terminal.

[1219] Terminal

[1220] The captured audio data is sent from the user device to the server, where it is converted into text using a transcription library.

[1221] server

[1222] The converted text data is summarized using a natural language processing module. For example, a summary such as "This product is lightweight and durable" is generated. The NLP module extracts important information (product features, durability, lightweight) and automatically inputs it into a customer management system.

[1223] In this way, users can improve the efficiency of customer service within the virtual store and automate a series of tasks from recording to management.

[1224] Example prompt sentence:

[1225] "Hello, what products are you looking for today?"

[1226] "What's your favorite thing about the items you purchased today?"

[1227] "Please tell me the features of this product."

[1228] This system makes it possible to improve the efficiency and accuracy of customer service within the virtual store.

[1229] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1230] Step 1:

[1231] Input: Customer's voice or text question

[1232] How it works: A user initiates a conversation with a customer through a virtual assistant in a virtual store. For example, the customer uses the microphone to ask, "What are the features of this product?"

[1233] Output: Raw captured audio or text data

[1234] Step 2:

[1235] Input: Raw audio data

[1236] How it works: The device uses a microphone to capture the customer's speech, and the captured voice data is sent directly to the device for further processing.

[1237] Output: Audio data stored on the device

[1238] Step 3:

[1239] Input: Audio data stored on the device

[1240] How it works: The device transcribes audio data using the speech_recognition library, specifically the Google Speech Recognition API, which converts audio data into text.

[1241] Output: Converted text data

[1242] Step 4:

[1243] Input: Converted text data

[1244] How it works: The server uses the transformers library to summarize text data. Specifically, it uses a generative AI model (e.g., the BART model) to convert input text into a summary sentence.

[1245] Output: Summarized text data

[1246] Step 5:

[1247] Input: Summarized text data

[1248] How it works: The server uses a natural language processing (NLP) module to extract key information, specifically applying entity recognition and relationship extraction algorithms to identify information like customer names, dates, action items, etc.

[1249] Output: Extracted key information (e.g. customer name, date, action items)

[1250] Step 6:

[1251] Input: Extracted important information

[1252] How it works: The server automatically inputs the extracted information through the customer management system's API, cross-checks it against existing data, and updates the database with any new or changed information.

[1253] Output: Updated customer relationship management system database

[1254] Step 7:

[1255] Input: Updated customer management system database

[1256] How it works: The server analyzes customer interaction data and generates a dashboard to visually display it. It visualizes data trends and statistics and provides them to the user as reports.

[1257] Output: Generated dashboards and reports

[1258] Step 8:

[1259] Input: Generated dashboards and reports

[1260] How it works: The server sends updated information to the user's device, allowing the user to check new information and analysis results through the notifications they receive.

[1261] Output: Notification sent to the user

[1262] The above are the specific processing steps of the system of the present invention, which automate customer service in the virtual store, significantly improving efficiency and accuracy.

[1263] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1264] Step 1:

[1265] User

[1266] The user initiates a conversation with the customer. This conversation can take place over the phone, an online conference, or in person. The user conducts the conversation using a specialized app.

[1267] Step 2:

[1268] Terminal

[1269] An application installed on the device captures the audio data of the conversation in real time and sends it to a server.

[1270] Step 3:

[1271] Terminal

[1272] At the same time, the emotion engine recognizes emotions from the user's voice in real time, generates emotion data, and sends it to the server.

[1273] Step 4:

[1274] server

[1275] The server passes the received voice data to a transcription engine, which converts the voice data into text data. For example, a conversation like "The client's name is Taro Yamada, and our next meeting is scheduled for next Monday" is generated as text data.

[1276] Step 5:

[1277] server

[1278] The server passes the generated text data to a natural language processing (NLP) module to generate a summary of the conversation, in this case "The next meeting with Taro Yamada is on Monday."

[1279] Step 6:

[1280] server

[1281] The server uses entity recognition and relation extraction algorithms to extract important information (such as customer name, date, and action item) from the summarized text. For example, the customer name is "Yamada Taro," the date is "next Monday," and the action item is "schedule a meeting."

[1282] Step 7:

[1283] server

[1284] The server receives the emotion data sent by the user and adds the emotion information to the extracted important information. For example, if the user indicates "joy," the emotion data is added as "joy."

[1285] Step 8:

[1286] server

[1287] The server automatically inputs this information via the sales management tool's API, matching it with existing customer data and updating the database as necessary.

[1288] Step 9:

[1289] server

[1290] The server analyzes sales activities based on all data and generates a dashboard containing sentiment data, which displays detailed information about sales activity performance and customer reactions.

[1291] Step 10:

[1292] server

[1293] The server sends the generated dashboard and report to the user's terminal, allowing the user to check the latest information.

[1294] Step 11:

[1295] Terminal

[1296] The user's device receives notifications from the server and displays dashboards and reports, allowing the user to plan their next actions.

[1297] Through these steps, the system of the present invention automates data recording and input work, significantly improving work efficiency while providing an environment in which salespeople can concentrate on conversation. Furthermore, by utilizing emotion data, it is expected to improve the quality of sales activities and increase customer satisfaction.

[1298] Example 2

[1299] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1300] In modern sales activities, it is important for sales representatives to record and manage detailed conversations with customers. However, this recording process is often done manually, prone to human error, and requires time and effort. It is also difficult to properly evaluate customer emotions and important information expressed during conversations and reflect them in sales management tools. This reduces the efficiency and quality of sales activities and affects customer satisfaction.

[1301] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for collecting conversation data, a means for transcribing, a means for summarizing, a means for extracting important information, a means for automatically inputting the data into a sales management tool, a means for recognizing emotions, and a means for evaluating the data. This enables accurate recording and management of conversation data, automates the evaluation of importance based on emotion data, and improves the efficiency and quality of sales activities.

[1302] "Conversation data" refers to the content of voice communication between a user and a customer.

[1303] "Transcription" is the process of analyzing audio data and converting its content into text data.

[1304] "Summarization" is the act of concisely summarizing transcribed text data and extracting the main points.

[1305] "Important information" refers to specific elements or facts extracted from conversation data, and is data that is necessary and useful in sales activities.

[1306] A "sales management tool" is software or an application for managing customer information, sales activity progress, results, etc.

[1307] "Auto-fill" is the process by which data is automatically entered by the system without manual input.

[1308] An "emotion engine" is a technology that recognizes a user's emotions from voice data or text data and extracts them as data.

[1309] "Emotion data" is data that represents the emotional state of the user as recognized by the emotion engine.

[1310] "Evaluation" is the act of assigning importance or priority to a conversation based on emotional data and other information.

[1311] "Matching" is the process of comparing new data with information in an existing database to identify matches.

[1312] "Update" is an operation that keeps existing data up to date with new information.

[1313] "Visualization" is the act of displaying analyzed data in a visual format such as a graph or chart to make it easier to understand.

[1314] This invention is a system that automatically records and manages conversations with customers during sales activities. This system has the functions of collecting conversation data, transcribing it, summarizing it, extracting important information, and automatically inputting it into a sales management tool. It also has the functions of analyzing and visualizing sales activity data and notifying users of updated information. Furthermore, it includes a function that combines an emotion engine that recognizes user emotions, uses emotion data to evaluate the importance of the conversation, and reflects this in the sales management tool.

[1315] System configuration

[1316] User terminal

[1317] The user terminal includes hardware for capturing voice data and dedicated application software for processing the data in real time. As the voice data is collected, an emotion engine recognizes the user's emotions and adds them to the data.

[1318] server

[1319] The server is equipped with a high-performance speech recognition engine, a natural language processing (NLP) module, a database management system, and an interface (API) with sales management tools.

[1320] 1. Capture and transmit audio data

[1321] When a user starts a conversation with a customer, a dedicated application installed on the user's terminal captures the voice data in real time and adds emotional data using an emotion engine.

[1322] This data is immediately sent to the server.

[1323] 2. Transcription and Summarization

[1324] The server passes the received audio data to a transcription engine, which converts the audio into text data. For example, a speech that says, "Hello, Yamada-san. I'd like to talk about our next meeting." is converted into text data.

[1325] The transcribed text data is sent to an NLP module, where the main points are summarized.

[1326] 3. Extracting important information

[1327] From the summarized text, the server uses entity recognition and relationship extraction algorithms to extract key information, such as customer names, dates, and action items.

[1328] 4. Evaluating Emotional Data

[1329] Emotion data sent from the user terminal is received and evaluated by the emotion engine, which determines the importance of the conversation data.

[1330] 5. Input and update to sales management tools

[1331] The server automatically inputs the extracted information and evaluation results through the sales management tool's API, comparing them with the existing database and updating any new information.

[1332] 6. Data Analysis and Visualization

[1333] The server analyzes sales activity data and generates and visualizes a dashboard that includes sentiment data, allowing users to get a detailed overview of sales activities.

[1334] 7. Notification

[1335] Finally, the generated dashboards and reports are sent to the user's device, allowing the user to check the latest information in real time.

[1336] Specific examples

[1337] As a specific example, consider a case where a user is having a telephone conversation with a customer.

[1338] User

[1339] When a user starts a call with a customer, the conversation is captured in real time by an application installed on the user's device. If the user shows strong emotions (e.g., joy, anger, etc.), the emotion engine recognizes the emotion and sends it to the server as emotion data.

[1340] Terminal

[1341] The captured voice data and emotional data are sent from the user's device to the server, allowing the content of the conversation and the user's emotional state to be instantly transmitted to the server.

[1342] server

[1343] The server immediately passes the received voice data to a transcription engine. For example, if a user says, "The next meeting is very important," the transcription engine converts this into text data. Next, a natural language processing module converts this text data into a summary such as "The next meeting is important" and extracts important information. The importance is evaluated taking into account emotional data, and the results are automatically entered into a sales management tool. The information is then compared with an existing customer database, and any new information is updated.

[1344] Prompt Sentence Examples

[1345] Below are some example prompts to input to a generative AI model:

[1346] "Please tell me the procedure for a system that automatically records conversations with customers and inputs emotional data into a sales management tool."

[1347] "Please explain each step of the process when a user starts a conversation with a customer."

[1348] "Can you tell me more about how the emotion engine works and how to feed data into a sales management tool?"

[1349] As a result, the system of the present invention provides an environment where salespeople can concentrate on the conversation, significantly improving work efficiency and quality. Furthermore, by utilizing emotion data, it is expected that customer satisfaction will be improved.

[1350] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1351] Step 1:

[1352] A user initiates a conversation with a customer. The conversation may take place over the phone, an online conference, or in person. A dedicated application installed on the user's device captures the conversation data in real time. The input is voice data. The output is voice data that is sent to the server in real time. Specifically, when the user says "hello," the voice is captured by the application.

[1353] Step 2:

[1354] The device sends the captured voice data along with the emotion data generated by the emotion engine to the server. The input is the captured voice data and emotion data, and the output is the data sent to the server. In concrete terms, if a user expresses high emotion by saying, "The next meeting is very important," that emotion is also sent as data to the server.

[1355] Step 3:

[1356] The server sends the received voice data to the transcription engine. The input is voice data and the output is text data. Specifically, the server converts the voice data "Hello, Yamada-san. I'd like to talk about our next meeting." into text data "Hello, Yamada-san. I'd like to talk about our next meeting."

[1357] Step 4:

[1358] The server sends the transcribed text data to a natural language processing (NLP) module for summarization. The input is the transcribed text data, and the output is the summarized text data. Specifically, the server converts the text data "I'd like to talk about our next meeting" into the summary "Meeting setup."

[1359] Step 5:

[1360] The server extracts important information from the summarized text data using entity recognition and relation extraction algorithms. The input is the summarized text data, and the output is important information (e.g., customer name, date, action item, etc.). Specifically, the server converts the summary "Meeting setup" into "Customer name: Yamada, action item: Meeting setup".

[1361] Step 6:

[1362] The server analyzes the emotion data sent from the user terminal and evaluates the importance of the conversation data. The input is emotion data, and the output is the evaluation result. Specifically, the server evaluates emotion data that is "important" as "high priority."

[1363] Step 7:

[1364] The server automatically inputs the extracted information and evaluation results through the API of the sales management tool. The input is the extracted information and evaluation results, and the output is the updated data in the sales management tool. Specifically, the server automatically reflects "Meeting arrangement with Yamada" in the sales management tool.

[1365] Step 8:

[1366] The server analyzes sales activity data and generates and visualizes a dashboard. The input is sales activity data, and the output is the visualized dashboard. Specifically, the server visualizes the data "Setting up a meeting with Yamada" in graphs and charts.

[1367] Step 9:

[1368] The server notifies the user device of the generated dashboard or report. The input is the visualized data, and the output is a notification to the user device. Specifically, the server notifies the user device of the "next meeting setting" information as an alert.

[1369] (Application example 2)

[1370] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1371] In conventional sales activities, accurately recording conversations with customers, converting them into data, and entering them into sales management tools is labor-intensive and prone to typographical errors and overlooking important information. Furthermore, because emotional data is processed without consideration, it is difficult to create sales strategies that reflect the customer's true feelings and emotions. As a result, there are limitations to improving the efficiency of sales activities and customer satisfaction. A system that solves these problems is needed.

[1372] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting conversation data, means for transcribing the collected conversation data, means for summarizing the transcribed text data, means for extracting important information from the summarized text data, means for automatically inputting the extracted information into a sales management tool, means for analyzing and visualizing sales activity data, means for notifying the user of update information, means for recognizing emotional data of the conversation, means for evaluating the importance of the conversation data based on the emotional data, and means for reflecting the evaluation results in the sales management tool. This makes it possible to accurately record the content of conversations with customers and their emotional data at the time, and automatically input them into the sales management tool. This makes it possible to improve the efficiency of sales activities and customer satisfaction.

[1373] "Conversation data" is information relating to the verbal exchanges that take place between a customer and a user.

[1374] "Means for collection" refers to devices or software for acquiring conversation data with customers and inputting it into the system.

[1375] "Transcribing means" means speech recognition technology or software for converting collected audio data into text data.

[1376] A "summarization tool" is a natural language processing technique or software that extracts key points from transcribed text data and summarizes them in a concise format.

[1377] A "means for extracting key information" is technology or software that identifies key elements from the summarized text data, such as customer names, dates, and action items to track.

[1378] "Means for automatically inputting information into a sales management tool" refers to a device or software that automatically reflects extracted important information in a sales management system (such as CRM).

[1379] "Means for analysis" refers to technology or software used to analyze collected sales activity data and evaluate its performance and trends.

[1380] "Visualization means" refers to technology or software for displaying analyzed data in a visually easy-to-understand format, such as a graph or chart.

[1381] "Means for notifying updated information" refers to technology or software that automatically notifies users of the latest sales activity data and analysis results.

[1382] "Means for recognizing emotional data" refers to technology or software for detecting emotional states from collected conversation data.

[1383] The "means for assessing importance" is a technology or software for assessing the content and importance of a conversation based on emotional data.

[1384] "Database matching means" means technology or software used to compare newly collected information with existing customer databases to identify matches.

[1385] "Means for updating information" refers to technology or software for updating the database in the sales management tool with the latest information based on the collated data.

[1386] This invention relates to a system that automatically records and processes conversations between salespeople and customers, primarily during sales activities in brick-and-mortar stores, and inputs the content into a sales management tool. The system of the present invention is equipped with a series of means for collecting conversation data, transcribing it, summarizing it, extracting important information, and automatically inputting it into the sales management tool. It can also recognize emotional data in the conversation, evaluate the importance of the conversation based on that information, and reflect the evaluation results in the sales management tool.

[1387] User

[1388] A salesperson (user) starts a conversation with a customer. This conversation is usually conducted face-to-face. A dedicated application is installed on the user's device, and this application captures conversation data in real time and sends it to a server.

[1389] Terminal

[1390] The user device transmits the captured voice data to the server, which instantly transmits the conversation data to the server. The device is equipped with an emotion engine that recognizes emotions from the voice data in real time.

[1391] server

[1392] The server sends the received audio data to a transcription engine using the Google Cloud Speech-to-Text API. The server converts the audio data into text and then summarizes it using a natural language processing (NLP) module. The transcription and summary data are temporarily stored. The server then uses entity recognition and relationship extraction algorithms to extract important information from the summary (e.g., customer name, date, action items, etc.).

[1393] Next, the server receives the emotion data sent from the user's device and evaluates the importance of the conversation data based on the emotions recognized by the emotion engine (e.g., Sentiment Analysis API). Based on this evaluation result, the extracted information is automatically entered into the sales management tool (e.g., Salesforce) via its API. The sales management tool compares the data with the existing database and updates it if any new information or changes are found.

[1394] The server also analyzes sales activity data and generates and visualizes dashboards. The generated dashboards also include sentiment data, enabling a more detailed understanding of the overall picture of sales activities. Finally, the server notifies the user of the generated dashboards and reports on their devices, allowing the user to access the latest information.

[1395] Specific examples

[1396] As a concrete example, consider a scenario in which a salesperson in a physical store talks with a customer about a new product. The salesperson asks, "What do you think about the features of this new product?" and the customer replies, "I think it's very attractive." This conversation is captured in real time using the smartphone's microphone. The captured voice data is immediately sent to a server, where a transcription engine converts it into text data such as, "What do you think about the features of this new product? I think it's very attractive."

[1397] The natural language processing module summarizes this as "the new product's features are attractive," and the entity recognition algorithm extracts the important information "new product" and "attractive." The emotion engine also recognizes that the customer is expressing "positive emotion" and reflects this in the evaluation results. All data is entered into the sales management tool by the server, where it is compared and updated with existing data. The analysis results of this conversation are visualized and displayed on a dashboard such as Salesforce.

[1398] Example prompts to input to the generative AI model

[1399] "Generate a program that transcribes customer voice data and analyzes sentiment data."

[1400] "Please provide sample code for a system that automatically records customer conversations and inputs key information into a sales management tool."

[1401] As described above, the system of the present invention allows salespeople to concentrate on talking to customers and greatly improves work efficiency by automating data recording and input. In addition, by using emotional data, it is possible to grasp customer reactions in real time, providing a valuable clue for appropriate responses.

[1402] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1403] Step 1:

[1404] The user starts a conversation with the customer. The conversation data is captured in real time by an application installed on the user's terminal. The input is the voice conversation between the customer and the user, and the output is the captured voice data.

[1405] Step 2:

[1406] The terminal transmits the captured voice data to the server. The input is the captured voice data, and the output is the transmission of the voice data to the server. This step ensures that the conversation data is immediately transmitted to the server.

[1407] Step 3:

[1408] The server sends the received audio data to a transcription engine, which converts the audio into text using the Google Cloud Speech-to-Text API. The input is audio data, and the output is transcribed text data.

[1409] Step 4:

[1410] The transcribed text data is summarized by the server's natural language processing (NLP) module. The summarization module extracts the main content and summarizes it in a concise format. The input is the transcribed text data, and the output is the summarized text data.

[1411] Step 5:

[1412] The server extracts key information from the summarized text data, using entity recognition and relationship extraction algorithms to identify key information such as customer names, dates, action items, etc. The input is the summarized text data, and the output is the extracted key information.

[1413] Step 6:

[1414] The server receives emotion data sent from the user device and evaluates the importance of the conversation data based on the emotions recognized by the emotion engine (e.g., Sentiment Analysis API). The input is emotion data and text data, and the output is the importance evaluation result.

[1415] Step 7:

[1416] Based on the evaluation results, the server automatically inputs the extracted information through the API of the sales management tool. The input is important information and the importance evaluation results, and the output is data input into the sales management tool. This is compared with the existing database, and any new information or changes are updated.

[1417] Step 8:

[1418] The server analyzes the sales activity data and generates a dashboard. The analysis results, including sentiment data, are displayed. The input is the sales activity data to be analyzed, and the output is the generated dashboard.

[1419] Step 9:

[1420] Finally, the server notifies the user of the generated dashboard or report. The user receives the notification and can access the latest information. The input is the generated dashboard or report, and the output is the notification sent to the user's device.

[1421] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1422] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1423] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1424] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1425] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1426] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1427] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1428] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1429] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1430] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1431] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1432] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1433] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1434] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1435] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1436] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1437] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1438] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1439] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1440] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1441] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1442] The following is further disclosed regarding the above embodiment.

[1443] (Claim 1)

[1444] a means for collecting conversation data;

[1445] A means of transcribing the collected conversation data;

[1446] a means for summarizing the transcribed text data;

[1447] means for extracting important information from the summarized text data;

[1448] A means for automatically inputting the extracted information into a sales management tool;

[1449] A means of analyzing and visualizing sales activity data,

[1450] a means for notifying users of updates;

[1451] A system including:

[1452] (Claim 2)

[1453] 10. The system of claim 1, wherein the conversation data is collected in real time.

[1454] (Claim 3)

[1455] The system of claim 1, which checks against the database of the sales management tool and updates new information.

[1456] "Example 1"

[1457] (Claim 1)

[1458] means for collecting audio data;

[1459] A means for converting the collected voice data into text data;

[1460] means for summarizing the converted character data;

[1461] means for extracting important information from the summarized text data;

[1462] a means for automatically inputting the extracted information into a data management tool;

[1463] A means of checking and updating the database in the data management tool with new information;

[1464] A means of analyzing and visualizing data;

[1465] means for notifying a user terminal;

[1466] A system including:

[1467] (Claim 2)

[1468] 10. The system of claim 1, wherein the audio data is collected in real time.

[1469] (Claim 3)

[1470] 10. The system of claim 1, further comprising: generating a summary of the data based on the extracted information.

[1471] "Application Example 1"

[1472] Claims based on new inventions

[1473] (Claim 1)

[1474] a means for collecting conversation data;

[1475] A means of transcribing the collected conversation data;

[1476] a means for summarizing the transcribed text data;

[1477] means for extracting important information from the summarized text data;

[1478] A means for automatically inputting the extracted information into a customer management system;

[1479] A means to analyze and visualize customer interaction data,

[1480] a means for notifying users of updates;

[1481] A system including:

[1482] (Claim 2)

[1483] 10. The system of claim 1, wherein conversation data is collected in real time using a virtual assistant that interacts with customers in a virtual store.

[1484] (Claim 3)

[1485] 2. The system of claim 1, wherein the system checks against the database of the customer management system and updates the new information.

[1486] "Example 2: Combining Emotion Engines"

[1487] (Claim 1)

[1488] a means for collecting conversation data;

[1489] A means of transcribing the collected conversation data;

[1490] a means for summarizing the transcribed text data;

[1491] means for extracting important information from the summarized text data;

[1492] A means for automatically inputting the extracted information into a sales management tool;

[1493] A means of analyzing and visualizing sales activity data,

[1494] a means for notifying users of updates;

[1495] means for recognizing a user's emotion from voice data;

[1496] a means for evaluating the recognized emotion data as the importance of the conversation data;

[1497] How to reflect this in sales management tools,

[1498] A system including:

[1499] (Claim 2)

[1500] 10. The system of claim 1, wherein the conversation data is collected in real time.

[1501] (Claim 3)

[1502] The system of claim 1, which checks against the database of the sales management tool and updates new information.

[1503] "Application example 2 when combining emotion engines"

[1504] (Claim 1)

[1505] a means for collecting conversation data;

[1506] A means of transcribing the collected conversation data;

[1507] a means for summarizing the transcribed text data;

[1508] means for extracting important information from the summarized text data;

[1509] A means for automatically inputting the extracted information into a sales management tool;

[1510] A means of analyzing and visualizing sales activity data,

[1511] a means for notifying users of updates;

[1512] a means for recognizing emotion data of a conversation;

[1513] a means for evaluating the importance of the conversation data based on the emotion data;

[1514] A means of reflecting the evaluation results in sales management tools,

[1515] A system including:

[1516] (Claim 2)

[1517] 10. The system of claim 1, wherein the conversation data is collected in real time.

[1518] (Claim 3)

[1519] The system of claim 1, which checks against the database of the sales management tool and updates new information. [Explanation of symbols]

[1520] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for collecting conversation data; A means of transcribing the collected conversation data; a means for summarizing the transcribed text data; means for extracting important information from the summarized text data; A means for automatically inputting the extracted information into a sales management tool; A means of analyzing and visualizing sales activity data, a means for notifying users of updates; A system including:

2. 10. The system of claim 1, wherein the conversation data is collected in real time.

3. The system according to claim 1, wherein the system checks against the database of the sales management tool and updates new information.

Images