System

Abstract

A system is provided.SOLUTION: The system includes a means for recording a business talk with a customer by voice, a means for converting the recorded voice into a high-quality text, a means for analyzing the text data and dividing / arranging the text data into minutes, a calendar event, and a thank-you mail draft, and a means for transmitting the divided / arranged information to a terminal of a person in charge of sales.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In conventional sales activities, after visiting a customer, sales representatives had to manually organize the details of the sales meeting, create minutes, manage the next appointment, write thank-you emails, and perform many other tasks. This required a great deal of effort and time for reporting after the meeting, placing an excessive burden on sales activities. Furthermore, because this was manual work, there was a high risk of errors, and accuracy and efficiency of information were required. The present invention aims to solve these problems. [Means for solving the problem]

[0005] The system of the present invention includes the following means.

[0006] 1. A way to record audio of business meetings with customers

[0007] 2. A means of converting recorded audio into high-quality text

[0008] 3. A method to analyze text data and divide and organize it into minutes, calendar events, and thank-you email drafts

[0009] 4. A means of sending the divided and organized information to the sales representative's terminal

[0010] Specifically, the system has a means of recording voice in real time during customer visits and uploading the recordings to a server. It also uses natural language processing technology as a high-quality text conversion method. This automatically converts the content of sales negotiations into text, and generative AI is used to automatically divide and organize it into minutes, calendar events, and thank-you email drafts. This information is fed back to the sales representative's device, eliminating the need for manual reporting and enabling more efficient and accurate sales activities.

[0011] A "customer" is a person or company with whom a sales representative conducts business.

[0012] A "business negotiation" is the content of a transaction or negotiation between a sales representative and a customer.

[0013] "Audio recording" means the digital recording of a conversation during a business meeting.

[0014] "High-quality text" is text data that has been converted from audio recordings into accurate and easy-to-read Japanese sentences.

[0015] "Text data" is written information created from audio recordings.

[0016] "Analysis" is the process of analyzing the converted text data and extracting specific information.

[0017] Minutes are documents that record the contents of business negotiations and summarize the main points and decisions made.

[0018] "Calendar event" is date and time information for managing the schedule of the next business meeting or related activities.

[0019] A "thank you email draft" is an initial draft of an email to be sent to a customer after a business meeting to express gratitude.

[0020] "Splitting and organizing" is the process of classifying the analyzed text data into different categories (minutes, calendar events, thank you email drafts) and arranging them systematically.

[0021] "Terminals" are electronic devices such as computers, tablets, and smartphones used by sales representatives.

[0022] "Send" means to transfer the organized information from the server to the sales representative's terminal.

[0023] "Server" means a central processing system for data processing and storage.

[0024] "Real-time" refers to the process of collecting and processing data instantly while a deal is taking place.

[0025] "Upload" means sending an audio file stored on a terminal to a server. [Brief explanation of the drawings]

[0026] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0027] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0028] First, the terms used in the following description will be explained.

[0029] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0030] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0031] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0032] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0033] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0034] [First embodiment]

[0035] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0036] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0037] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0038] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0039] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0040] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0041] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0042] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0043] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0044] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0045] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0046] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0047] overview

[0048] The system of this invention aims to enable sales representatives to efficiently organize sales negotiations with customers and largely automate reporting tasks. The system includes voice recording, text conversion, text analysis using generative AI, and feedback functions.

[0049] Audio recording

[0050] Users can record audio during business meetings with customers using their smartphones or dedicated recording devices, and the audio files are saved locally for later uploading to the system.

[0051] Uploading an audio file

[0052] After the negotiation is completed, the user uploads the recorded audio file to the server. By using the upload function on the user's device to send the audio file to the server, the system can proceed to the next processing step.

[0053] Speech to text

[0054] The server converts the received audio file into high-quality text using natural language processing technology, specifically converting the audio data into text via an API. This converted text data is stored in a database and goes on to the next analysis step.

[0055] Text analysis, segmentation, and organization

[0056] The server analyzes the converted text data using generative AI. The analysis results are divided and organized into three categories: minutes, calendar events, and thank-you email drafts. Text analysis is performed using generative AI, which automatically extracts important information from the text data.

[0057] minutes

[0058] The server extracts important points and next action items from the text and compiles them into minutes, which include the key points discussed and decisions made during the business meeting.

[0059] Calendar Events

[0060] The server extracts date and time information about upcoming sales meetings and specific events and generates calendar events, allowing salespeople to efficiently manage their upcoming appointments.

[0061] Thank you email draft

[0062] The server generates a draft thank you email, which includes a thank you note for the business meeting with the customer and a brief message about next steps.

[0063] Organized information feedback

[0064] The server sends each piece of information that has been divided and organized to the user's device. The sales representative can then view the minutes, calendar events, and thank-you email draft on their own device and make any necessary revisions.

[0065] Adding an event to Google Calendar

[0066] The device will add the sent calendar event information to Google Calendar, and salespeople can add these events to their own calendars via API, so they won't forget about upcoming meetings or sales calls.

[0067] Sending a thank you email

[0068] The terminal creates and sends an email to the customer based on the draft thank-you email. The user checks the content of the email, makes any necessary corrections, and then presses the send email button, causing the system to send the email to the customer.

[0069] Specific operation example

[0070] 1. The user uses a smartphone to record a conversation during a business meeting.

[0071] 2. After the business meeting is completed, the user uses the application to upload the recording file to the server.

[0072] 3. The server converts the audio file to text using the Hyper CLOVA API.

[0073] 4. The server uses generative AI to analyze the text and extract key information.

[0074] 5. The server divides and organizes the minutes, calendar events, and thank you email drafts.

[0075] 6. The organized information is fed back to the user's device.

[0076] 7. The device adds the fed back calendar event information to Google Calendar.

[0077] 8. The terminal creates an email based on the draft thank-you email and sends it to the customer after the user confirms it.

[0078] In this way, the system of the present invention improves the efficiency of reporting work for sales representatives and reduces their burden.

[0079] The processing flow will be explained below.

[0080] Step 1:

[0081] The user records audio during a business meeting with a customer using a smartphone or recording device, and the audio file is saved in the device's local storage.

[0082] Step 2:

[0083] After the negotiation is over, the user uploads the recorded audio file to the server using a dedicated application. The file is sent by pressing the "upload" button in the application.

[0084] Step 3:

[0085] The server receives the uploaded audio file and starts the process of converting it into high-quality text. Specifically, it sends the audio file to the Hyper CLOVA API, which converts the audio data into text.

[0086] Step 4:

[0087] The server receives the text data returned from the Hyper CLOVA API and stores it in the database. At this point, the voice data has been converted into text data.

[0088] Step 5:

[0089] The server inputs the saved text data into a generative AI for analysis, which then divides it into categories such as minutes, calendar events, and thank-you email drafts.

[0090] Step 6:

[0091] The server stores the analysis results in a database and prepares the divided and organized data for transmission to the sales representative's terminal.

[0092] Step 7:

[0093] The server then sends the divided and organized information back to the user's device, including a meeting minutes, a to-do list, event information for Google Calendar, and a draft thank-you email.

[0094] Step 8:

[0095] The device receives the fed-back Google Calendar event information and automatically calls the Google Calendar API to add it to the calendar, automatically updating the sales representative's schedule.

[0096] Step 9:

[0097] The terminal displays the draft of the thank-you email to the user, prompting the user to confirm the content. After the user makes any corrections, the terminal uses the email sending function to send the thank-you email to the customer.

[0098] Example 1

[0099] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0100] Traditionally, sales representatives' post-negotiation reporting work has been manual, requiring a lot of time and effort. Important information is often overlooked or inaccurately organized, leading to concerns that this can reduce the efficiency of sales activities and impair productivity.

[0101] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0102] In this invention, the server includes a means for recording conversations between salespeople and customers, a means for converting the recorded voice into text, and a means for analyzing the converted text data and dividing and organizing it into summaries, planned events, and thank-you email proposals, thereby reducing the workload and preventing important information from being overlooked.

[0103] A "sales representative" is a position that is responsible for introducing and selling the products and services of a company or organization to customers.

[0104] "Customer" means anyone who purchases or considers purchasing a product or service from a business or organization.

[0105] "Means for recording conversations" refers to devices or software for saving audio recordings of business conversations with customers.

[0106] "Recorded audio" refers to audio data stored digitally via a device during a business meeting.

[0107] "Means of converting to text" refers to the technology or tools used to process audio data and convert its content into written information.

[0108] "Text data" refers to character information converted from audio data.

[0109] "Analyzing" refers to the process of reviewing and understanding the converted text data and extracting important information.

[0110] A "summary" refers to a document that extracts the main points and important information from text data and summarizes them concisely.

[0111] "Planned events" refer to future schedules and action items determined based on the content of the business negotiation.

[0112] A "thank you email draft" refers to a draft email document to be sent to a customer after a business meeting to express gratitude.

[0113] "Means of dividing and organizing" refers to the techniques and tools used to classify and organize the analyzed text data into different categories (summary, upcoming events, thank you email ideas).

[0114] "Terminal" refers to a device (e.g., a smartphone or computer) used to receive, review, or modify information.

[0115] "Transmission means" refers to the technologies and tools used to transfer digital information from one device or system to another.

[0116] "Server" refers to a computer system for storing, processing, and managing data.

[0117] "Data Server" refers to the central computer system for storing and processing audio files and analysis results.

[0118] "Advanced language processing technology" refers to advanced algorithms and technologies used to understand and process natural language, such as natural language processing (NLP) technology.

[0119] overview

[0120] This invention is a system that enables salespeople to efficiently record details of business negotiations with customers and automate reporting. The system has functions for recording voice data, converting it to text, analyzing the text using generative AI, and providing feedback on the information.

[0121] Hardware and software used

[0122] The hardware used by users includes smartphones and dedicated recording devices. The server is a computer system with advanced processing capabilities, and is installed with APIs and generative AI software for processing voice data and analyzing text. Specifically, HyperCLOVA API is used for voice-to-text conversion, and generative AI models are used for text analysis.

[0123] Program processing

[0124] During business meetings, users record conversations using a smartphone or dedicated recording device. When recording is complete, the audio file is automatically saved to the device's local storage. After the meeting is over, the user launches an application and uploads the recorded audio file to the server. The user's device has an upload function for selecting and sending audio files.

[0125] The server analyzes the received audio file and converts it into high-quality text using the Hyper CLOVA API. The converted text data is stored in a database on the server and then analyzed using a generative AI model. As a result of the analysis, the text data is divided and organized into three categories: summary, upcoming events, and thank-you email suggestions.

[0126] The server sends the divided and organized information to the user's device. Specifically, a summary of the meeting minutes, the next scheduled event, and a draft of the thank-you email are fed back to the user's device. The user can then check this information on their own device and make any necessary corrections.

[0127] The device adds the sent calendar event information to Google Calendar. Users can add these events to their own calendars via the Google Calendar API, allowing them to efficiently manage upcoming meetings and business negotiations. The device also creates a thank-you email based on the original draft, and once the user confirms it and sends it, the system automatically sends the email to the customer.

[0128] Examples of concrete examples and prompts

[0129] A concrete example would be the following:

[0130] 1. A user uses a recording app on their smartphone to record a conversation during a business meeting.

[0131] 2. After the negotiation is completed, the user selects the recording file in the application and uploads it to the server.

[0132] 3. The server uses the Hyper CLOVA API to convert the audio file into text data and save it in the database.

[0133] 4. The server analyzes the text data using a generative AI model and segments key information into summaries, upcoming events, and suggested thank-you emails.

[0134] 5. The server sends the organized data to the user's terminal.

[0135] 6. The user's device adds the sent calendar event information to Google Calendar and creates and sends a thank-you email.

[0136] An example of a specific prompt to be input to the generative AI model is as follows:

[0137] "Please analyze the following audio data and divide it into three parts: a summary, upcoming events, and a suggested thank you email. The audio data contains the content of a business meeting."

[0138] In this way, the system of the present invention consistently automates everything from recording voice data to text analysis, organizing information, and providing feedback, significantly reducing the workload of sales representatives.

[0139] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0140] System program processing flow

[0141] Step 1:

[0142] During business meetings, users record conversations using a smartphone or dedicated recording device. The input is "audio data" and the output is "an audio file saved on the local device." Specifically, the user launches a recording app, presses the record button to record the conversation, and the file is saved when recording is complete.

[0143] Step 2:

[0144] After the business meeting is over, the user uploads the recorded audio file to the server. The input is the "audio file saved on the local device" and the output is the "audio file uploaded to the server." Specifically, the user launches the application, selects the audio file using the upload function, and presses the send button.

[0145] Step 3:

[0146] The server receives the uploaded audio file and converts it into high-quality text. The input is the "audio file uploaded to the server" and the output is "text data stored in the database." Specifically, the server uses the Hyper CLOVA API to convert the audio data into text data and stores the text data in the database.

[0147] Step 4:

[0148] The server analyzes the converted text data using generative AI. The input is "text data stored in the database" and the output is "analyzed information (summary, scheduled events, and thank-you email suggestions)." Specifically, the server uses a generative AI model to analyze the text data and divide and organize it into summaries, scheduled events, and thank-you email suggestions.

[0149] Step 5:

[0150] The server sends the organized information to the user's device. The input is the analyzed information (summary, scheduled events, and thank-you email suggestions), and the output is the information sent to the user's device. Specifically, the server generates the summary, scheduled events, and thank-you email suggestions and sends them to the user's device.

[0151] Step 6:

[0152] The device adds the sent calendar event information to Google Calendar. The input is "calendar event information sent to the user's device" and the output is "event added to Google Calendar." Specifically, the device uses the Google Calendar API to add the fed-back calendar event information to the user's Google Calendar.

[0153] Step 7:

[0154] The terminal creates an email to the customer based on the draft thank you email, and sends it after the user has confirmed it. The input is the "draft thank you email sent to the user's terminal," and the output is the "thank you email sent to the customer." Specifically, the terminal displays the draft thank you email, and the user confirms and edits the content, then presses the send button to send the email to the customer.

[0155] In this way, the system of the present invention processes and calculates data at each processing step, thereby improving the efficiency of salespeople's work.

[0156] (Application example 1)

[0157] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0158] To improve the quality of customer service in brick-and-mortar stores, there is a need for technology that can efficiently record, analyze, and organize the content of conversations between employees and customers. However, current methods require a large portion of the work to be done manually, which is inefficient and places a heavy burden on employees. Furthermore, the content of conversations with customers cannot be used appropriately, making it difficult to utilize in business negotiations or future interactions. To solve these issues, it is necessary to introduce a system that converts speech to text and automatically classifies and organizes it.

[0159] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0160] In this invention, the server includes a means for recording audio of business negotiations with customers, a means for converting the recorded audio into high-quality text, a means for analyzing the text data and dividing and organizing it into minutes, scheduled events, and thank-you email drafts, and a means for feeding back the analyzed data, thereby making it possible to significantly improve the efficiency and quality of face-to-face interactions by employees in physical stores.

[0161] "Customer" means a visitor or user with whom you engage in face-to-face interaction to purchase a product or service.

[0162] "Business negotiations" refer to conversations and negotiations regarding products and services with customers.

[0163] "Audio recording means" refers to a device or system for recording conversations during business negotiations.

[0164] "High-quality text" refers to text data that can accurately convert speech into text and properly reproduce the content of business negotiations.

[0165] "Means for analyzing text data" refers to a device or system that uses a generative AI model to analyze text data and divide and organize it into specific categories.

[0166] A "minutes" is a document that records important matters discussed and decisions made during a business meeting.

[0167] "Scheduled event" refers to the date and time information of the next meeting or specific activity in a business negotiation.

[0168] A "thank you email draft" is a draft of an email to be sent to a customer after a business meeting to express gratitude.

[0169] "Feedback means" refers to a device or system that provides analyzed data to a user.

[0170] "Server" refers to a computer system that processes voice data and divides and organizes the analysis results.

[0171] A "terminal" is an electronic device used by a sales representative to display analyzed information.

[0172] The system of the present invention is designed to improve the efficiency and quality of customer service in brick-and-mortar stores. The system includes voice recording, speech-to-text conversion, text analysis using generative AI models, and feedback functionality.

[0173] Audio recording

[0174] A user uses an electronic device to record conversations with customers. The hardware used for this is a smartphone or other audio recording device. The audio data captured by the audio recording device is temporarily stored on a local device.

[0175] Uploading an audio file

[0176] After the negotiation is over, the user uploads the recorded audio file to the server using the application. This upload function is installed on the user's device and has the function of sending the recorded file to the server.

[0177] Speech to text

[0178] The server converts the received audio file into high-quality text. Natural language processing technology is used in this process. Specifically, the audio data is converted into text data using an API implemented on the server. An example of this API is the Google Speech Recognition API. The converted text data is stored in a database and moves on to the next analysis step.

[0179] Text analysis, segmentation, and organization

[0180] The server analyzes the converted text data using a generative AI model (e.g., OpenAI's GPT-3). This analysis divides and organizes the data into three categories: minutes, scheduled events, and draft thank-you emails. This automatically extracts important information, next action items, and date and time information.

[0181] Organized information feedback

[0182] The server sends the divided and organized information to the user's device and provides feedback to the user. The user can then check the minutes, scheduled events, and thank-you email draft on their own device and make any necessary corrections.

[0183] Adding an event to Google Calendar

[0184] The device can add the sent scheduled event information to Google Calendar, and the event will be automatically reflected on the calendar using the Google Calendar API.

[0185] Sending a thank you email

[0186] The terminal creates an email to the customer based on the draft of the thank-you email and sends it. In this process, the user checks the content of the email and makes any necessary corrections before sending it.

[0187] Specific examples

[0188] For example, if a conversation is recorded during a sales conversation with a customer saying, "Hello, can you tell me more about this product?", the system would work as follows:

[0189] 1. Use your smartphone to record conversations during business negotiations.

[0190] 2. After the business meeting is completed, the user uses the application to upload the recording file to the server.

[0191] 3. The server converts the audio file into text using the Google Speech Recognition API.

[0192] 4. The server uses a generative AI model to analyze the text and extract key information.

[0193] 5. The server divides and organizes the minutes, scheduled events, and thank you email drafts.

[0194] 6. The organized information is fed back to the user's device.

[0195] Prompt Sentence Examples

[0196] For example, the prompt used during parsing is:

[0197] "Please parse the following text and split it into meeting minutes, planned events, and a draft thank you email.\n\nHello, can you tell me more about this product?"

[0198] In this way, the present invention improves the efficiency of customer service in brick-and-mortar stores, reduces the burden on employees, and improves customer satisfaction.

[0199] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0200] Step 1:

[0201] A user records a conversation during a business meeting.

[0202] Input: Customer conversation

[0203] Output: Recorded audio file

[0204] What it does: A user uses a smartphone or other audio recording device to record a conversation during a business meeting. The audio file is saved locally on the device.

[0205] Step 2:

[0206] The user uploads the recorded audio file to the server.

[0207] Input: Audio file stored on the local device

[0208] Output: Audio file uploaded to the server

[0209] Specific operation: After the business meeting is completed, the user uploads the recorded audio file to the server using a dedicated application. The file is sent to the server using the application's upload function.

[0210] Step 3:

[0211] The server converts the audio file into high-quality text.

[0212] Input: Uploaded audio file

[0213] Output: Text data

[0214] How it works: The server receives the audio file and converts it into text using natural language processing technology such as Google Speech Recognition API. This text data is then stored in a database.

[0215] Step 4:

[0216] The server analyzes the text data using a generative AI model.

[0217] Input: Text data converted from audio

[0218] Output: Meeting minutes, scheduled events, draft thank you email

[0219] Specific operation: The server inputs the text data into a generative AI model (e.g., OpenAI's GPT-3), and divides and organizes the analysis results into three categories: minutes, scheduled events, and a draft thank-you email. An example of a prompt used during analysis is "Analyze the following text and divide it into minutes, scheduled events, and a draft thank-you email.\n\n(text data)".

[0220] Step 5:

[0221] The server feeds back the divided and organized information to the user's device.

[0222] Input: Minutes, scheduled events, and thank you email draft data

[0223] Output: Information fed back to the user's device

[0224] Specific operation: The server sends the analyzed data to the user's device, where the user can review it. The user can then edit the meeting minutes, scheduled events, and thank-you email drafts on their device as appropriate.

[0225] Step 6:

[0226] The device adds the sent scheduled event information to Google Calendar.

[0227] Input: Scheduled event information

[0228] Output: Scheduled events reflected in Google Calendar

[0229] Specific operation: The device uses the Google Calendar API to add the scheduled event information sent from the server to the Google Calendar.

[0230] Step 7:

[0231] The terminal creates and sends an email to the customer based on the original thank-you email draft.

[0232] Input: Draft thank you email

[0233] Output: Thank you email sent to customer

[0234] Specific operation: The terminal creates an email based on the draft thank-you email, and after the user checks the content and makes any necessary corrections, the email is sent to the customer.

[0235] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0236] overview

[0237] The system of the present invention not only enables salespeople to efficiently organize sales negotiations with customers and largely automates reporting tasks, but also incorporates an emotion engine to recognize and reflect the user's emotional state. The system includes voice recording, text conversion, generative AI text analysis, emotion recognition, and feedback functions.

[0238] Audio recording

[0239] Users record their voices during sales conversations with customers using their smartphones or dedicated recording devices, and the audio files are saved in the device's local storage. An emotion engine runs in parallel to analyze the user's real-time reactions and tone of voice.

[0240] Uploading an audio file

[0241] After the negotiation is over, the user uploads the recorded audio file to the server using a dedicated application. During the upload process, the emotion data analyzed by the emotion engine is also sent to the server.

[0242] Speech to text

[0243] The server converts the received audio file into high-quality text. This process uses natural language processing technology, specifically converting the audio data into text via an API. Emotion data analyzed by the emotion engine is also stored in association with the text data.

[0244] Text analysis, segmentation, and organization

[0245] The server inputs the converted text data into a generative AI for analysis. The analysis results are divided and organized into three categories: minutes, calendar events, and thank-you email drafts. Emotion data recognized by the emotion engine is also used in the analysis process.

[0246] minutes

[0247] The server extracts important points and next action items from the text and compiles them into minutes, which include the key points discussed and decisions made during the business meeting.

[0248] Calendar Events

[0249] The server extracts date and time information for upcoming business meetings or specific events, generates calendar events, and customizes reminder notifications based on the emotional data analyzed by the emotion engine.

[0250] Thank you email draft

[0251] The server generates a draft thank-you email, which includes a brief message expressing gratitude for the business interaction and a suggestion for next steps. The appropriate tone and content are automatically adjusted based on data from the emotion engine.

[0252] Organized information feedback

[0253] The server sends each piece of information that has been divided and organized to the user's device. The sales representative can then view the minutes, calendar events, and thank-you email draft on their own device and make any necessary revisions.

[0254] Adding an event to Google Calendar

[0255] The device will add the sent calendar event information to Google Calendar, and salespeople can add these events to their own calendars via API, so they won't forget about upcoming meetings or sales calls.

[0256] Sending a thank you email

[0257] The terminal displays the draft of the thank-you email to the user, prompting the user to confirm the content. After the user makes any corrections, the terminal uses the email sending function to send the thank-you email to the customer.

[0258] Specific operation example

[0259] 1. The user uses their smartphone to record conversations during business negotiations, and the emotion engine simultaneously analyzes the emotional state in real time.

[0260] 2. After the business meeting is completed, the user uploads the recording file and emotion data to the server using the application.

[0261] 3. The server converts the audio file into text using the Hyper CLOVA API and stores the emotion data in the database in parallel.

[0262] 4. The server uses generative AI to analyze the text and extract key information, incorporating emotional data into the analysis process.

[0263] 5. The server divides and organizes the data into minutes, calendar events, and thank-you email drafts, and reflects the emotional data in each.

[0264] 6. The organized information is fed back to the user's device.

[0265] 7. When the device adds the fed-back calendar event information to Google Calendar, it customizes the reminder content based on the emotion data.

[0266] 8. The terminal displays a draft of the thank-you email to the user, and after having the user confirm that the content has been adjusted based on the emotional data, the email is sent to the customer.

[0267] In this way, the system of the present invention improves the efficiency of reporting work by sales representatives and also takes into account the emotional aspects of communication with customers, enabling higher quality follow-up.

[0268] The processing flow will be explained below.

[0269] Step 1:

[0270] During a business meeting with a customer, the user starts recording audio using a smartphone or recording device. The recording is made using a dedicated application, and the emotion engine simultaneously analyzes the user's emotional state in real time. The audio and emotion data are stored in the device's local storage.

[0271] Step 2:

[0272] After the negotiation is over, the user uploads the recorded voice file and emotional data to the server using a dedicated application. By pressing the "upload" button in the application, the emotional data is sent to the server along with the voice file.

[0273] Step 3:

[0274] The server receives the uploaded audio file and first converts the audio data into high-quality text via the Hyper CLOVA API. At this time, the audio data and emotion data are linked and stored synchronously.

[0275] Step 4:

[0276] The server stores the converted text data and the linked emotion data in a database, allowing the text data and emotion information to be stored in an integrated format.

[0277] Step 5:

[0278] The server inputs the saved text data into a generative AI system, which analyzes it and divides and organizes it into three categories: minutes, calendar events, and thank-you email drafts. Emotional data is also used in the analysis process.

[0279] Step 6:

[0280] The server generates information based on text analysis and sentiment data as follows:

[0281] Minutes: Extract important points and next action items from the text, and summarize them by taking into account their importance based on emotional data.

[0282] Calendar Events: Extract time and date information for upcoming sales meetings or specific events and customize reminders based on sentiment data.

[0283] Thank you email draft: Generates a draft of a thank you email to send to a customer, automatically adjusting the appropriate tone and content based on emotional data.

[0284] Step 7:

[0285] The server then feeds back the generated minutes, calendar events, and thank-you email drafts to the user's device, including adjustments that take into account emotional information.

[0286] Step 8:

[0287] The device adds the received calendar event information to Google Calendar. Specifically, it calls the Google Calendar API to add an event that reflects the reminder notification content based on the emotion data to the calendar.

[0288] Step 9:

[0289] The device displays a draft of the thank-you email to the user. The user checks the email content and makes any necessary corrections. The device then uses its email sending function to send the thank-you email to the customer, with the content adjusted based on the emotional data.

[0290] Through the above steps, the system of the present invention improves the efficiency of sales representatives' reporting work, and also takes into account the emotional aspects of communication with customers, enabling more personalized responses.

[0291] Example 2

[0292] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0293] While traditional sales negotiation systems can record voice and convert it into text, they do not include emotional analysis or the utilization of that data, which means the quality of communication with customers is not fully considered. This can lead to poor customer service efficiency and the risk of missing important market opportunities. Furthermore, organizing and reporting sales negotiation results takes a lot of time, preventing work efficiency.

[0294] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0295] In this invention, the server includes means for recording audio of business negotiations with customers, means for converting the recorded audio into high-quality text, means for analyzing the text data and dividing and organizing it into minutes, calendar events, and thank-you email drafts, means for analyzing emotional states and reflecting the results in data, means for sending the divided and organized information to the sales representative's terminal, means for performing real-time emotional analysis during customer visits and uploading recordings including the analyzed data to the server, means for adding the transmitted calendar event information to a schedule management service, and means for displaying the thank-you email draft on the sales representative's terminal and sending it to the customer after revision. This enables efficient organization of business negotiations with customers and high-quality customer service utilizing emotional data.

[0296] - "Customer" refers to a business partner or other person with whom we have a business relationship.

[0297] "Business negotiation" refers to a place for negotiation or consultation for business purposes.

[0298] "Audio recording" refers to the act of electronically recording sound.

[0299] "Text conversion" refers to the process of converting voice data into text format.

[0300] "Text data" refers to data in the form of text converted from speech.

[0301] "Minutes" refers to a document that records what was said and what decisions were made at a meeting or business negotiation.

[0302] "Calendar event" refers to an appointment or schedule that includes date and time information.

[0303] A "thank you email draft" refers to a draft of an email to be sent to a customer to express gratitude.

[0304] "Emotional state" refers to the speaker's feelings or state of mind.

[0305] "Sentiment analysis" refers to the process of analyzing emotions from speech or text.

[0306] A "database" refers to a system that efficiently stores and manages large amounts of data.

[0307] "Schedule management service" refers to an online service that electronically manages schedules and appointments.

[0308] "Upload" refers to the act of sending data from local storage to a server.

[0309] "Generative AI" refers to a system that uses artificial intelligence to generate or analyze data.

[0310] "API" stands for Application Program Interface and refers to a set of rules that enable communication between software programs.

[0311] System Configuration

[0312] The system of the present invention is designed to help salespeople efficiently organize sales negotiations with customers and largely automate reporting tasks. It also incorporates an emotion engine to recognize the user's emotional state and reflect it in the data. The system has the following main functions:

[0313] Audio recording

[0314] Users record audio during business meetings with customers. This recording is done using a smartphone or a dedicated recording device, and the audio file is saved in the device's local storage. Specific hardware used is a standard smartphone or portable recorder. During the recording, the emotion engine analyzes the user's tone of voice and reactions in real time to collect emotional data.

[0315] Uploading an audio file

[0316] After the negotiation is over, the user uploads the recorded voice file and emotional data to the server using a dedicated application. During the upload process, the emotional data analyzed by the emotion engine is also sent to the server.

[0317] Speech to text

[0318] The server uses natural language processing technology to convert the uploaded audio file into high-quality text. Specifically, it uses the Hyper CLOVA API to convert the audio data into text. The emotion data is associated with this text data and stored in a database.

[0319] Text analysis, segmentation, and organization

[0320] The server uses generative AI to analyze the text data. The analysis results are divided and organized into three categories: minutes, calendar events, and thank-you email drafts. Emotional data is also incorporated into this analysis process, generating content based on the user's emotional state.

[0321] Information Feedback

[0322] The server sends each piece of information that has been divided and organized to the user's device. The salesperson can then review the minutes, calendar events, and draft thank-you emails on their device and make any necessary revisions. Furthermore, the device can add the sent calendar event information to a schedule management service such as Google Calendar and set reminders based on the emotion data.

[0323] Sending a thank you email

[0324] The terminal displays the draft of the thank you email to the user for confirmation. After the user makes any necessary corrections, the terminal uses the email sending function to send the thank you email to the customer.

[0325] Specific examples

[0326] For example, the prompt when analyzing voice data from a business meeting might look like this:

[0327] Example prompt: "Transcribe the audio data below into text, summarize the key points, and draft a thank you email with an appropriate tone based on the emotional data."

[0328] Audio data: (Audio file link or file name)

[0329] As can be seen from this specific operation example, the system of the present invention highly automates a series of business processes, from voice recording to text analysis, incorporating emotional data, and generating thank-you emails. This significantly reduces the workload of sales representatives and improves the quality of communication with customers.

[0330] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0331] Step 1:

[0332] Users record their business meetings using a voice recording device (smartphone or dedicated recorder). While recording, the emotion engine works in parallel, analyzing the user's tone of voice and reactions in real time.

[0333] Input: Voice data (recorded business meetings), real-time user emotional responses

[0334] Output: Recorded audio files, real-time emotion data

[0335] Specific operation: Launch the recording app and start recording as soon as the sales meeting begins. The emotion engine analyzes the tone of the voice and collects emotional data.

[0336] Step 2:

[0337] After the business negotiation is completed, the user uploads the recorded voice file and emotion data to the server using a dedicated application.

[0338] Input: Recorded audio file, emotion data

[0339] Output: Audio files and emotion data uploaded to the server

[0340] Specific operation: Tap the "Upload" button in the dedicated application to send the file and data to the server.

[0341] Step 3:

[0342] The server converts the audio file into text using the Hyper CLOVA API, and associates the emotion data with the text data and stores it in a database.

[0343] Input: Audio file uploaded to the server, emotion data

[0344] Output: Text data, associated emotion data

[0345] Specific operation: The server sends the audio file to the Hyper CLOVA API, receives it as text data, and stores the emotion data together with the text data in the database.

[0346] Step 4:

[0347] The server uses generative AI to analyze the text data and divide and organize it into three categories: minutes, calendar events, and thank-you email drafts. Emotional data is also used in this analysis.

[0348] Input: Text data, emotion data

[0349] Output: Separated and organized meeting minutes, calendar events, and a draft thank you email

[0350] How it works: The server inputs text data into the generative AI, extracts important information, and classifies it into categories. Content that reflects emotional data is also generated.

[0351] Step 5:

[0352] The server sends each piece of information (minutes, calendar events, and thank-you email draft) to the user's device.

[0353] Input: Separated and organized meeting minutes, calendar events, draft thank you email

[0354] Output: Each piece of information sent to the user's device

[0355] Specific operation: The server sends information to the user's device through a dedicated application.

[0356] Step 6:

[0357] The device will then add the sent calendar event information to a schedule management service such as Google Calendar, and the reminder content will also be customized based on the emotion data.

[0358] Input: Calendar event information, emotion data

[0359] Output: Events added to Google Calendar, customized reminders

[0360] Specific operation: Adds event information to Google Calendar via the Calendar API and sets reminders.

[0361] Step 7:

[0362] The terminal displays the draft of the thank-you email to the user, prompting the user to confirm the content. After the user makes any corrections, the terminal uses the email sending function to send the thank-you email to the customer.

[0363] Input: Draft thank you email, user modifications

[0364] Output: Thank you email sent

[0365] Specific operation: The draft email is displayed on the device, the user makes corrections and taps the "Send" button, and the email is sent to the customer.

[0366] (Application example 2)

[0367] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0368] In conventional sales activities, the tasks of recording sales negotiations, organizing the details, and reporting them were time-consuming and labor-intensive. Furthermore, it was difficult to properly grasp the emotional state of the customer during the negotiation and reflect this in subsequent follow-up, which led to a decline in the quality of communication with the customer. This made it difficult for sales representatives to carry out their work efficiently and respond to customers effectively.

[0369] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0370] In this invention, the server includes a means for recording audio of business negotiations with customers, a means for converting the recorded audio into high-quality text, a means for analyzing the text data and dividing and organizing it into minutes, calendar events, and thank-you email drafts, and a means for inputting the analyzed emotional data into a generative AI model and reflecting it in the information organization process. This allows sales representatives to automate the detailed recording and organization of business negotiations and enable effective follow-up that takes into account the emotional state of the customer.

[0371] The "business negotiation voice recording means" is a hardware or software function for recording audio data during a business negotiation.

[0372] The "text conversion means" is a function that converts recorded voice data into high-quality text data using natural language processing technology.

[0373] "Text analysis means" is a function that uses generative AI to analyze text data and divide and organize it into minutes, calendar events, and thank-you email drafts.

[0374] "Information transmission means" is a communication function for transmitting divided and organized information to the sales representative's terminal.

[0375] The "emotion analysis means" is a function that analyzes the tone of voice and language used by customers and sales representatives during negotiations to grasp their emotional state in real time.

[0376] An "emotion engine" is a software module used as an emotion analysis tool to generate emotion data in real time.

[0377] A "generative AI model" is an artificial intelligence model that performs analysis and information generation based on input text data and emotional data.

[0378] A "prompt" is an instruction statement that allows the generative AI model to properly analyze the text data it inputs.

[0379] The system of the present invention aims to enable sales representatives to efficiently organize business negotiations with customers and to largely automate reporting tasks. This system can be implemented by the following procedure.

[0380] The system configuration requires a smartphone used by the user, an emotion analysis engine, a generative AI model, natural language processing technology, and a server that integrates these. Specifically, the smartphone is equipped with voice recording and data communication functions and connects to the server application.

[0381] The general flow of the system is as follows: First, the salesperson (user) uses their smartphone to record the voice of the sales negotiation. At the same time, the emotion analysis engine analyzes the tone of the user's and customer's voice and the words they use in real time to understand their emotional state.

[0382] Once the negotiation is over, the user uploads the recorded voice data and real-time analyzed emotion data to the server. The server receives the voice data and converts it into high-quality text using natural language processing technology (such as the Google Speech-to-Text API). The emotion data is also stored in a database.

[0383] The server then uses a generative AI model (such as OpenAI's ChatGPT) to analyze the converted text data. The analysis is performed using pre-defined prompts, which are then divided and organized into three main categories: meeting minutes, calendar events, and thank-you email drafts. Specific examples of prompts are as follows:

[0384] "Please parse the following text and classify it into the following categories: meeting minutes, next appointment, thank you email content. Emotion data: {emotionData}

[0385] text:{

[0386] ...negotiation text...

[0387] }"

[0388] The information for each category, divided and organized, along with sentiment analysis data, is fed back to the sales representative's smartphone, allowing them to check meeting minutes, calendar events, and thank-you email drafts on their own screens and make any necessary adjustments.

[0389] Calendar events are automatically added to Google Calendar via API. Reminders are customized based on the customer's emotional data. Thank-you emails are then sent to the customer after the sales representative has reviewed and revised the content.

[0390] The entire system processes and calculates data using the following technical elements:

[0391] Smartphone: Responsible for recording and data uploading.

[0392] Emotion analysis engine: Analyzes emotional state from voice in real time.

[0393] Natural language processing technology: Converts voice data into high-quality text data.

[0394] Generative AI model: Analyzes text data and organizes information into categories.

[0395] Google Calendar API: Used to automatically add calendar events and adjust reminders.

[0396] This allows sales representatives to efficiently record and organize sales negotiation details while also enabling them to conduct high-quality follow-up that takes into account the customer's emotional state.

[0397] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0398] Step 1:

[0399] A user uses a smartphone to record audio during a business meeting.

[0400] Input: Voice during negotiation

[0401] Output: Recorded audio file

[0402] How it works: The user launches the recording app on their smartphone and presses the record button when the sales meeting begins. As the recording progresses, the emotion analysis engine analyzes the tone and vocabulary of the user and customer in real time and records their emotional state.

[0403] Step 2:

[0404] After the business negotiation is completed, the user uploads the recorded voice file and real-time emotion data to the server.

[0405] Input: Recorded audio files, real-time emotion data

[0406] Output: Audio files and emotion data stored on the server

[0407] Specific operation: After the business negotiation is completed, the user presses the upload button in the app. The audio file and emotion data are sent to the server and stored in the database.

[0408] Step 3:

[0409] The server receives the audio file and converts it into high-quality text using natural language processing techniques.

[0410] Input: Recorded audio file

[0411] Output: High-quality text data

[0412] Specific operation: The server analyzes the received audio file and converts it into text data using natural language processing technology (e.g., Google Speech-to-Text API). At the same time, emotion data is also stored in a database.

[0413] Step 4:

[0414] The server uses a generative AI model to analyze the converted text data and divide and organize it into three main categories: meeting minutes, calendar events, and thank-you email drafts.

[0415] Input: High-quality text data, emotion data

[0416] Output: Minutes data, calendar event data, thank you email draft data

[0417] Specific operation: The server uses a generative AI model (e.g., OpenAI's ChatGPT) to analyze and classify data using the following prompt:

[0418] "Please parse the following text and classify it into the following categories: meeting minutes, next appointment, thank you email content. Emotion data: {emotionData}

[0419] text:{

[0420] ...negotiation text...

[0421] }"

[0422] The analysis results are divided into minutes, calendar events, and a draft thank you email.

[0423] Step 5:

[0424] The server then sends the information for each divided and organized category back to the sales representative's smartphone.

[0425] Input: Minutes data, calendar event data, thank you email draft data

[0426] Output: Data displayed on the user's smartphone

[0427] Specific operation: The organized data is sent from the server to the user's smartphone, and the sales representative checks and edits the contents of meeting minutes, calendar events, and thank-you emails.

[0428] Step 6:

[0429] Your device will automatically add calendar events to your Google Calendar and customize reminders based on your emotional data.

[0430] Input: Calendar event data

[0431] Output: Events added to Google Calendar with customized reminders

[0432] Specific operation: An event is added from the user's smartphone via the Google Calendar API, and the reminder content is adjusted based on emotional data.

[0433] Step 7:

[0434] The terminal displays a draft of the thank-you email to the sales representative, who then edits the content and sends it to the customer.

[0435] Input: Draft data for thank you email

[0436] Output: Thank you email sent

[0437] Specific actions: After the sales representative checks and edits the content of the thank-you email, they send it to the customer via the email app on their smartphone.

[0438] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0439] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0440] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0441] [Second embodiment]

[0442] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0443] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0444] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0445] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0446] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0447] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0448] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0449] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0450] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0451] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0452] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0453] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0454] overview

[0455] The system of this invention aims to enable sales representatives to efficiently organize sales negotiations with customers and largely automate reporting tasks. The system includes voice recording, text conversion, text analysis using generative AI, and feedback functions.

[0456] Audio recording

[0457] Users can record audio during business meetings with customers using their smartphones or dedicated recording devices, and the audio files are saved locally for later uploading to the system.

[0458] Uploading an audio file

[0459] After the negotiation is completed, the user uploads the recorded audio file to the server. By using the upload function on the user's device to send the audio file to the server, the system can proceed to the next processing step.

[0460] Speech to text

[0461] The server converts the received audio file into high-quality text using natural language processing technology, specifically converting the audio data into text via an API. This converted text data is stored in a database and goes on to the next analysis step.

[0462] Text analysis, segmentation, and organization

[0463] The server analyzes the converted text data using generative AI. The analysis results are divided and organized into three categories: minutes, calendar events, and thank-you email drafts. Text analysis is performed using generative AI, which automatically extracts important information from the text data.

[0464] minutes

[0465] The server extracts important points and next action items from the text and compiles them into minutes, which include the key points discussed and decisions made during the business meeting.

[0466] Calendar Events

[0467] The server extracts date and time information about upcoming sales meetings and specific events and generates calendar events, allowing salespeople to efficiently manage their upcoming appointments.

[0468] Thank you email draft

[0469] The server generates a draft thank you email, which includes a thank you note for the business meeting with the customer and a brief message about next steps.

[0470] Organized information feedback

[0471] The server sends each piece of information that has been divided and organized to the user's device. The sales representative can then view the minutes, calendar events, and thank-you email draft on their own device and make any necessary revisions.

[0472] Adding an event to Google Calendar

[0473] The device will add the sent calendar event information to Google Calendar, and salespeople can add these events to their own calendars via API, so they won't forget about upcoming meetings or sales calls.

[0474] Sending a thank you email

[0475] The terminal creates and sends an email to the customer based on the draft thank-you email. The user checks the content of the email, makes any necessary corrections, and then presses the send email button, causing the system to send the email to the customer.

[0476] Specific operation example

[0477] 1. The user uses a smartphone to record a conversation during a business meeting.

[0478] 2. After the business meeting is completed, the user uses the application to upload the recording file to the server.

[0479] 3. The server converts the audio file to text using the Hyper CLOVA API.

[0480] 4. The server uses generative AI to analyze the text and extract key information.

[0481] 5. The server divides and organizes the minutes, calendar events, and thank you email drafts.

[0482] 6. The organized information is fed back to the user's device.

[0483] 7. The device adds the fed back calendar event information to Google Calendar.

[0484] 8. The terminal creates an email based on the draft thank-you email and sends it to the customer after the user confirms it.

[0485] In this way, the system of the present invention improves the efficiency of reporting work for sales representatives and reduces their burden.

[0486] The processing flow will be explained below.

[0487] Step 1:

[0488] The user records audio during a business meeting with a customer using a smartphone or recording device, and the audio file is saved in the device's local storage.

[0489] Step 2:

[0490] After the negotiation is over, the user uploads the recorded audio file to the server using a dedicated application. The file is sent by pressing the "upload" button in the application.

[0491] Step 3:

[0492] The server receives the uploaded audio file and starts the process of converting it into high-quality text. Specifically, it sends the audio file to the Hyper CLOVA API, which converts the audio data into text.

[0493] Step 4:

[0494] The server receives the text data returned from the Hyper CLOVA API and stores it in the database. At this point, the voice data has been converted into text data.

[0495] Step 5:

[0496] The server inputs the saved text data into a generative AI for analysis, which then divides it into categories such as minutes, calendar events, and thank-you email drafts.

[0497] Step 6:

[0498] The server stores the analysis results in a database and prepares the divided and organized data for transmission to the sales representative's terminal.

[0499] Step 7:

[0500] The server then sends the divided and organized information back to the user's device, including a meeting minutes, a to-do list, event information for Google Calendar, and a draft thank-you email.

[0501] Step 8:

[0502] The device receives the fed-back Google Calendar event information and automatically calls the Google Calendar API to add it to the calendar, automatically updating the sales representative's schedule.

[0503] Step 9:

[0504] The terminal displays the draft of the thank-you email to the user, prompting the user to confirm the content. After the user makes any corrections, the terminal uses the email sending function to send the thank-you email to the customer.

[0505] Example 1

[0506] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0507] Traditionally, sales representatives' post-negotiation reporting work has been manual, requiring a lot of time and effort. Important information is often overlooked or inaccurately organized, leading to concerns that this can reduce the efficiency of sales activities and impair productivity.

[0508] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0509] In this invention, the server includes a means for recording conversations between salespeople and customers, a means for converting the recorded voice into text, and a means for analyzing the converted text data and dividing and organizing it into summaries, planned events, and thank-you email proposals, thereby reducing the workload and preventing important information from being overlooked.

[0510] A "sales representative" is a position that is responsible for introducing and selling the products and services of a company or organization to customers.

[0511] "Customer" means anyone who purchases or considers purchasing a product or service from a business or organization.

[0512] "Means for recording conversations" refers to devices or software for saving audio recordings of business conversations with customers.

[0513] "Recorded audio" refers to audio data stored digitally via a device during a business meeting.

[0514] "Means of converting to text" refers to the technology or tools used to process audio data and convert its content into written information.

[0515] "Text data" refers to character information converted from audio data.

[0516] "Analyzing" refers to the process of reviewing and understanding the converted text data and extracting important information.

[0517] A "summary" refers to a document that extracts the main points and important information from text data and summarizes them concisely.

[0518] "Planned events" refer to future schedules and action items determined based on the content of the business negotiation.

[0519] A "thank you email draft" refers to a draft email document to be sent to a customer after a business meeting to express gratitude.

[0520] "Means of dividing and organizing" refers to the techniques and tools used to classify and organize the analyzed text data into different categories (summary, upcoming events, thank you email ideas).

[0521] "Terminal" refers to a device (e.g., a smartphone or computer) used to receive, review, or modify information.

[0522] "Transmission means" refers to the technologies and tools used to transfer digital information from one device or system to another.

[0523] "Server" refers to a computer system for storing, processing, and managing data.

[0524] "Data Server" refers to the central computer system for storing and processing audio files and analysis results.

[0525] "Advanced language processing technology" refers to advanced algorithms and technologies used to understand and process natural language, such as natural language processing (NLP) technology.

[0526] overview

[0527] This invention is a system that enables salespeople to efficiently record details of business negotiations with customers and automate reporting. The system has functions for recording voice data, converting it to text, analyzing the text using generative AI, and providing feedback on the information.

[0528] Hardware and software used

[0529] The hardware used by users includes smartphones and dedicated recording devices. The server is a computer system with advanced processing capabilities, and is installed with APIs and generative AI software for processing voice data and analyzing text. Specifically, HyperCLOVA API is used for voice-to-text conversion, and generative AI models are used for text analysis.

[0530] Program processing

[0531] During business meetings, users record conversations using a smartphone or dedicated recording device. When recording is complete, the audio file is automatically saved to the device's local storage. After the meeting is over, the user launches an application and uploads the recorded audio file to the server. The user's device has an upload function for selecting and sending audio files.

[0532] The server analyzes the received audio file and converts it into high-quality text using the Hyper CLOVA API. The converted text data is stored in a database on the server and then analyzed using a generative AI model. As a result of the analysis, the text data is divided and organized into three categories: summary, upcoming events, and thank-you email suggestions.

[0533] The server sends the divided and organized information to the user's device. Specifically, a summary of the meeting minutes, the next scheduled event, and a draft of the thank-you email are fed back to the user's device. The user can then check this information on their own device and make any necessary corrections.

[0534] The device adds the sent calendar event information to Google Calendar. Users can add these events to their own calendars via the Google Calendar API, allowing them to efficiently manage upcoming meetings and business negotiations. The device also creates a thank-you email based on the original draft, and once the user confirms it and sends it, the system automatically sends the email to the customer.

[0535] Examples of concrete examples and prompts

[0536] A concrete example would be the following:

[0537] 1. A user uses a recording app on their smartphone to record a conversation during a business meeting.

[0538] 2. After the negotiation is completed, the user selects the recording file in the application and uploads it to the server.

[0539] 3. The server uses the Hyper CLOVA API to convert the audio file into text data and save it in the database.

[0540] 4. The server analyzes the text data using a generative AI model and segments key information into summaries, upcoming events, and suggested thank-you emails.

[0541] 5. The server sends the organized data to the user's terminal.

[0542] 6. The user's device adds the sent calendar event information to Google Calendar and creates and sends a thank-you email.

[0543] An example of a specific prompt to be input to the generative AI model is as follows:

[0544] "Please analyze the following audio data and divide it into three parts: a summary, upcoming events, and a suggested thank you email. The audio data contains the content of a business meeting."

[0545] In this way, the system of the present invention consistently automates everything from recording voice data to text analysis, organizing information, and providing feedback, significantly reducing the workload of sales representatives.

[0546] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0547] System program processing flow

[0548] Step 1:

[0549] During business meetings, users record conversations using a smartphone or dedicated recording device. The input is "audio data" and the output is "an audio file saved on the local device." Specifically, the user launches a recording app, presses the record button to record the conversation, and the file is saved when recording is complete.

[0550] Step 2:

[0551] After the business meeting is over, the user uploads the recorded audio file to the server. The input is the "audio file saved on the local device" and the output is the "audio file uploaded to the server." Specifically, the user launches the application, selects the audio file using the upload function, and presses the send button.

[0552] Step 3:

[0553] The server receives the uploaded audio file and converts it into high-quality text. The input is the "audio file uploaded to the server" and the output is "text data stored in the database." Specifically, the server uses the Hyper CLOVA API to convert the audio data into text data and stores the text data in the database.

[0554] Step 4:

[0555] The server analyzes the converted text data using generative AI. The input is "text data stored in the database" and the output is "analyzed information (summary, scheduled events, and thank-you email suggestions)." Specifically, the server uses a generative AI model to analyze the text data and divide and organize it into summaries, scheduled events, and thank-you email suggestions.

[0556] Step 5:

[0557] The server sends the organized information to the user's device. The input is the analyzed information (summary, scheduled events, and thank-you email suggestions), and the output is the information sent to the user's device. Specifically, the server generates the summary, scheduled events, and thank-you email suggestions and sends them to the user's device.

[0558] Step 6:

[0559] The device adds the sent calendar event information to Google Calendar. The input is "calendar event information sent to the user's device" and the output is "event added to Google Calendar." Specifically, the device uses the Google Calendar API to add the fed-back calendar event information to the user's Google Calendar.

[0560] Step 7:

[0561] The terminal creates an email to the customer based on the draft thank you email, and sends it after the user has confirmed it. The input is the "draft thank you email sent to the user's terminal," and the output is the "thank you email sent to the customer." Specifically, the terminal displays the draft thank you email, and the user confirms and edits the content, then presses the send button to send the email to the customer.

[0562] In this way, the system of the present invention processes and calculates data at each processing step, thereby improving the efficiency of salespeople's work.

[0563] (Application example 1)

[0564] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0565] To improve the quality of customer service in brick-and-mortar stores, there is a need for technology that can efficiently record, analyze, and organize the content of conversations between employees and customers. However, current methods require a large portion of the work to be done manually, which is inefficient and places a heavy burden on employees. Furthermore, the content of conversations with customers cannot be used appropriately, making it difficult to utilize in business negotiations or future interactions. To solve these issues, it is necessary to introduce a system that converts speech to text and automatically classifies and organizes it.

[0566] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0567] In this invention, the server includes a means for recording audio of business negotiations with customers, a means for converting the recorded audio into high-quality text, a means for analyzing the text data and dividing and organizing it into minutes, scheduled events, and thank-you email drafts, and a means for feeding back the analyzed data, thereby making it possible to significantly improve the efficiency and quality of face-to-face interactions by employees in physical stores.

[0568] "Customer" means a visitor or user with whom you engage in face-to-face interaction to purchase a product or service.

[0569] "Business negotiations" refer to conversations and negotiations regarding products and services with customers.

[0570] "Audio recording means" refers to a device or system for recording conversations during business negotiations.

[0571] "High-quality text" refers to text data that can accurately convert speech into text and properly reproduce the content of business negotiations.

[0572] "Means for analyzing text data" refers to a device or system that uses a generative AI model to analyze text data and divide and organize it into specific categories.

[0573] A "minutes" is a document that records important matters discussed and decisions made during a business meeting.

[0574] "Scheduled event" refers to the date and time information of the next meeting or specific activity in a business negotiation.

[0575] A "thank you email draft" is a draft of an email to be sent to a customer after a business meeting to express gratitude.

[0576] "Feedback means" refers to a device or system that provides analyzed data to a user.

[0577] "Server" refers to a computer system that processes voice data and divides and organizes the analysis results.

[0578] A "terminal" is an electronic device used by a sales representative to display analyzed information.

[0579] The system of the present invention is designed to improve the efficiency and quality of customer service in brick-and-mortar stores. The system includes voice recording, speech-to-text conversion, text analysis using generative AI models, and feedback functionality.

[0580] Audio recording

[0581] A user uses an electronic device to record conversations with customers. The hardware used for this is a smartphone or other audio recording device. The audio data captured by the audio recording device is temporarily stored on a local device.

[0582] Uploading an audio file

[0583] After the negotiation is over, the user uploads the recorded audio file to the server using the application. This upload function is installed on the user's device and has the function of sending the recorded file to the server.

[0584] Speech to text

[0585] The server converts the received audio file into high-quality text. Natural language processing technology is used in this process. Specifically, the audio data is converted into text data using an API implemented on the server. An example of this API is the Google Speech Recognition API. The converted text data is stored in a database and moves on to the next analysis step.

[0586] Text analysis, segmentation, and organization

[0587] The server analyzes the converted text data using a generative AI model (e.g., OpenAI's GPT-3). This analysis divides and organizes the data into three categories: minutes, scheduled events, and draft thank-you emails. This automatically extracts important information, next action items, and date and time information.

[0588] Organized information feedback

[0589] The server sends the divided and organized information to the user's device and provides feedback to the user. The user can then check the minutes, scheduled events, and thank-you email draft on their own device and make any necessary corrections.

[0590] Adding an event to Google Calendar

[0591] The device can add the sent scheduled event information to Google Calendar, and the event will be automatically reflected on the calendar using the Google Calendar API.

[0592] Sending a thank you email

[0593] The terminal creates an email to the customer based on the draft of the thank-you email and sends it. In this process, the user checks the content of the email and makes any necessary corrections before sending it.

[0594] Specific examples

[0595] For example, if a conversation is recorded during a sales conversation with a customer saying, "Hello, can you tell me more about this product?", the system would work as follows:

[0596] 1. Use your smartphone to record conversations during business negotiations.

[0597] 2. After the business meeting is completed, the user uses the application to upload the recording file to the server.

[0598] 3. The server converts the audio file into text using the Google Speech Recognition API.

[0599] 4. The server uses a generative AI model to analyze the text and extract key information.

[0600] 5. The server divides and organizes the minutes, scheduled events, and thank you email drafts.

[0601] 6. The organized information is fed back to the user's device.

[0602] Prompt Sentence Examples

[0603] For example, the prompt used during parsing is:

[0604] "Please parse the following text and split it into meeting minutes, planned events, and a draft thank you email.\n\nHello, can you tell me more about this product?"

[0605] In this way, the present invention improves the efficiency of customer service in brick-and-mortar stores, reduces the burden on employees, and improves customer satisfaction.

[0606] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0607] Step 1:

[0608] A user records a conversation during a business meeting.

[0609] Input: Customer conversation

[0610] Output: Recorded audio file

[0611] What it does: A user uses a smartphone or other audio recording device to record a conversation during a business meeting. The audio file is saved locally on the device.

[0612] Step 2:

[0613] The user uploads the recorded audio file to the server.

[0614] Input: Audio file stored on the local device

[0615] Output: Audio file uploaded to the server

[0616] Specific operation: After the business meeting is completed, the user uploads the recorded audio file to the server using a dedicated application. The file is sent to the server using the application's upload function.

[0617] Step 3:

[0618] The server converts the audio file into high-quality text.

[0619] Input: Uploaded audio file

[0620] Output: Text data

[0621] How it works: The server receives the audio file and converts it into text using natural language processing technology such as Google Speech Recognition API. This text data is then stored in a database.

[0622] Step 4:

[0623] The server analyzes the text data using a generative AI model.

[0624] Input: Text data converted from audio

[0625] Output: Meeting minutes, scheduled events, draft thank you email

[0626] Specific operation: The server inputs the text data into a generative AI model (e.g., OpenAI's GPT-3), and divides and organizes the analysis results into three categories: minutes, scheduled events, and a draft thank-you email. An example of a prompt used during analysis is "Analyze the following text and divide it into minutes, scheduled events, and a draft thank-you email.\n\n(text data)".

[0627] Step 5:

[0628] The server feeds back the divided and organized information to the user's device.

[0629] Input: Minutes, scheduled events, and thank you email draft data

[0630] Output: Information fed back to the user's device

[0631] Specific operation: The server sends the analyzed data to the user's device, where the user can review it. The user can then edit the meeting minutes, scheduled events, and thank-you email drafts on their device as appropriate.

[0632] Step 6:

[0633] The device adds the sent scheduled event information to Google Calendar.

[0634] Input: Scheduled event information

[0635] Output: Scheduled events reflected in Google Calendar

[0636] Specific operation: The device uses the Google Calendar API to add the scheduled event information sent from the server to the Google Calendar.

[0637] Step 7:

[0638] The terminal creates and sends an email to the customer based on the original thank-you email draft.

[0639] Input: Draft thank you email

[0640] Output: Thank you email sent to customer

[0641] Specific operation: The terminal creates an email based on the draft thank-you email, and after the user checks the content and makes any necessary corrections, the email is sent to the customer.

[0642] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0643] overview

[0644] The system of the present invention not only enables salespeople to efficiently organize sales negotiations with customers and largely automates reporting tasks, but also incorporates an emotion engine to recognize and reflect the user's emotional state. The system includes voice recording, text conversion, generative AI text analysis, emotion recognition, and feedback functions.

[0645] Audio recording

[0646] Users record their voices during sales conversations with customers using their smartphones or dedicated recording devices, and the audio files are saved in the device's local storage. An emotion engine runs in parallel to analyze the user's real-time reactions and tone of voice.

[0647] Uploading an audio file

[0648] After the negotiation is over, the user uploads the recorded audio file to the server using a dedicated application. During the upload process, the emotion data analyzed by the emotion engine is also sent to the server.

[0649] Speech to text

[0650] The server converts the received audio file into high-quality text. This process uses natural language processing technology, specifically converting the audio data into text via an API. Emotion data analyzed by the emotion engine is also stored in association with the text data.

[0651] Text analysis, segmentation, and organization

[0652] The server inputs the converted text data into a generative AI for analysis. The analysis results are divided and organized into three categories: minutes, calendar events, and thank-you email drafts. Emotion data recognized by the emotion engine is also used in the analysis process.

[0653] minutes

[0654] The server extracts important points and next action items from the text and compiles them into minutes, which include the key points discussed and decisions made during the business meeting.

[0655] Calendar Events

[0656] The server extracts date and time information for upcoming business meetings or specific events, generates calendar events, and customizes reminder notifications based on the emotional data analyzed by the emotion engine.

[0657] Thank you email draft

[0658] The server generates a draft thank-you email, which includes a brief message expressing gratitude for the business interaction and a suggestion for next steps. The appropriate tone and content are automatically adjusted based on data from the emotion engine.

[0659] Organized information feedback

[0660] The server sends each piece of information that has been divided and organized to the user's device. The sales representative can then view the minutes, calendar events, and thank-you email draft on their own device and make any necessary revisions.

[0661] Adding an event to Google Calendar

[0662] The device will add the sent calendar event information to Google Calendar, and salespeople can add these events to their own calendars via API, so they won't forget about upcoming meetings or sales calls.

[0663] Sending a thank you email

[0664] The terminal displays the draft of the thank-you email to the user, prompting the user to confirm the content. After the user makes any corrections, the terminal uses the email sending function to send the thank-you email to the customer.

[0665] Specific operation example

[0666] 1. The user uses their smartphone to record conversations during business negotiations, and the emotion engine simultaneously analyzes the emotional state in real time.

[0667] 2. After the business meeting is completed, the user uploads the recording file and emotion data to the server using the application.

[0668] 3. The server converts the audio file into text using the Hyper CLOVA API and stores the emotion data in the database in parallel.

[0669] 4. The server uses generative AI to analyze the text and extract key information, incorporating emotional data into the analysis process.

[0670] 5. The server divides and organizes the data into minutes, calendar events, and thank-you email drafts, and reflects the emotional data in each.

[0671] 6. The organized information is fed back to the user's device.

[0672] 7. When the device adds the fed-back calendar event information to Google Calendar, it customizes the reminder content based on the emotion data.

[0673] 8. The terminal displays a draft of the thank-you email to the user, and after having the user confirm that the content has been adjusted based on the emotional data, the email is sent to the customer.

[0674] In this way, the system of the present invention improves the efficiency of reporting work by sales representatives and also takes into account the emotional aspects of communication with customers, enabling higher quality follow-up.

[0675] The processing flow will be explained below.

[0676] Step 1:

[0677] During a business meeting with a customer, the user starts recording audio using a smartphone or recording device. The recording is made using a dedicated application, and the emotion engine simultaneously analyzes the user's emotional state in real time. The audio and emotion data are stored in the device's local storage.

[0678] Step 2:

[0679] After the negotiation is over, the user uploads the recorded voice file and emotional data to the server using a dedicated application. By pressing the "upload" button in the application, the emotional data is sent to the server along with the voice file.

[0680] Step 3:

[0681] The server receives the uploaded audio file and first converts the audio data into high-quality text via the Hyper CLOVA API. At this time, the audio data and emotion data are linked and stored synchronously.

[0682] Step 4:

[0683] The server stores the converted text data and the linked emotion data in a database, allowing the text data and emotion information to be stored in an integrated format.

[0684] Step 5:

[0685] The server inputs the saved text data into a generative AI system, which analyzes it and divides and organizes it into three categories: minutes, calendar events, and thank-you email drafts. Emotional data is also used in the analysis process.

[0686] Step 6:

[0687] The server generates information based on text analysis and sentiment data as follows:

[0688] Minutes: Extract important points and next action items from the text, and summarize them by taking into account their importance based on emotional data.

[0689] Calendar Events: Extract time and date information for upcoming sales meetings or specific events and customize reminders based on sentiment data.

[0690] Thank you email draft: Generates a draft of a thank you email to send to a customer, automatically adjusting the appropriate tone and content based on emotional data.

[0691] Step 7:

[0692] The server then feeds back the generated minutes, calendar events, and thank-you email drafts to the user's device, including adjustments that take into account emotional information.

[0693] Step 8:

[0694] The device adds the received calendar event information to Google Calendar. Specifically, it calls the Google Calendar API to add an event that reflects the reminder notification content based on the emotion data to the calendar.

[0695] Step 9:

[0696] The device displays a draft of the thank-you email to the user. The user checks the email content and makes any necessary corrections. The device then uses its email sending function to send the thank-you email to the customer, with the content adjusted based on the emotional data.

[0697] Through the above steps, the system of the present invention improves the efficiency of sales representatives' reporting work, and also takes into account the emotional aspects of communication with customers, enabling more personalized responses.

[0698] Example 2

[0699] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0700] While traditional sales negotiation systems can record voice and convert it into text, they do not include emotional analysis or the utilization of that data, which means the quality of communication with customers is not fully considered. This can lead to poor customer service efficiency and the risk of missing important market opportunities. Furthermore, organizing and reporting sales negotiation results takes a lot of time, preventing work efficiency.

[0701] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0702] In this invention, the server includes means for recording audio of business negotiations with customers, means for converting the recorded audio into high-quality text, means for analyzing the text data and dividing and organizing it into minutes, calendar events, and thank-you email drafts, means for analyzing emotional states and reflecting the results in data, means for sending the divided and organized information to the sales representative's terminal, means for performing real-time emotional analysis during customer visits and uploading recordings including the analyzed data to the server, means for adding the transmitted calendar event information to a schedule management service, and means for displaying the thank-you email draft on the sales representative's terminal and sending it to the customer after revision. This enables efficient organization of business negotiations with customers and high-quality customer service utilizing emotional data.

[0703] - "Customer" refers to a business partner or other person with whom we have a business relationship.

[0704] "Business negotiation" refers to a place for negotiation or consultation for business purposes.

[0705] "Audio recording" refers to the act of electronically recording sound.

[0706] "Text conversion" refers to the process of converting voice data into text format.

[0707] "Text data" refers to data in the form of text converted from speech.

[0708] "Minutes" refers to a document that records what was said and what decisions were made at a meeting or business negotiation.

[0709] "Calendar event" refers to an appointment or schedule that includes date and time information.

[0710] A "thank you email draft" refers to a draft of an email to be sent to a customer to express gratitude.

[0711] "Emotional state" refers to the speaker's feelings or state of mind.

[0712] "Sentiment analysis" refers to the process of analyzing emotions from speech or text.

[0713] A "database" refers to a system that efficiently stores and manages large amounts of data.

[0714] "Schedule management service" refers to an online service that electronically manages schedules and appointments.

[0715] "Upload" refers to the act of sending data from local storage to a server.

[0716] "Generative AI" refers to a system that uses artificial intelligence to generate or analyze data.

[0717] "API" stands for Application Program Interface and refers to a set of rules that enable communication between software programs.

[0718] System Configuration

[0719] The system of the present invention is designed to help salespeople efficiently organize sales negotiations with customers and largely automate reporting tasks. It also incorporates an emotion engine to recognize the user's emotional state and reflect it in the data. The system has the following main functions:

[0720] Audio recording

[0721] Users record audio during business meetings with customers. This recording is done using a smartphone or a dedicated recording device, and the audio file is saved in the device's local storage. Specific hardware used is a standard smartphone or portable recorder. During the recording, the emotion engine analyzes the user's tone of voice and reactions in real time to collect emotional data.

[0722] Uploading an audio file

[0723] After the negotiation is over, the user uploads the recorded voice file and emotional data to the server using a dedicated application. During the upload process, the emotional data analyzed by the emotion engine is also sent to the server.

[0724] Speech to text

[0725] The server uses natural language processing technology to convert the uploaded audio file into high-quality text. Specifically, it uses the Hyper CLOVA API to convert the audio data into text. The emotion data is associated with this text data and stored in a database.

[0726] Text analysis, segmentation, and organization

[0727] The server uses generative AI to analyze the text data. The analysis results are divided and organized into three categories: minutes, calendar events, and thank-you email drafts. Emotional data is also incorporated into this analysis process, generating content based on the user's emotional state.

[0728] Information Feedback

[0729] The server sends each piece of information that has been divided and organized to the user's device. The salesperson can then review the minutes, calendar events, and draft thank-you emails on their device and make any necessary revisions. Furthermore, the device can add the sent calendar event information to a schedule management service such as Google Calendar and set reminders based on the emotion data.

[0730] Sending a thank you email

[0731] The terminal displays the draft of the thank you email to the user for confirmation. After the user makes any necessary corrections, the terminal uses the email sending function to send the thank you email to the customer.

[0732] Specific examples

[0733] For example, the prompt when analyzing voice data from a business meeting might look like this:

[0734] Example prompt: "Transcribe the audio data below into text, summarize the key points, and draft a thank you email with an appropriate tone based on the emotional data."

[0735] Audio data: (Audio file link or file name)

[0736] As can be seen from this specific operation example, the system of the present invention highly automates a series of business processes, from voice recording to text analysis, incorporating emotional data, and generating thank-you emails. This significantly reduces the workload of sales representatives and improves the quality of communication with customers.

[0737] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0738] Step 1:

[0739] Users record their business meetings using a voice recording device (smartphone or dedicated recorder). While recording, the emotion engine works in parallel, analyzing the user's tone of voice and reactions in real time.

[0740] Input: Voice data (recorded business meetings), real-time user emotional responses

[0741] Output: Recorded audio files, real-time emotion data

[0742] Specific operation: Launch the recording app and start recording as soon as the sales meeting begins. The emotion engine analyzes the tone of the voice and collects emotional data.

[0743] Step 2:

[0744] After the business negotiation is completed, the user uploads the recorded voice file and emotion data to the server using a dedicated application.

[0745] Input: Recorded audio file, emotion data

[0746] Output: Audio files and emotion data uploaded to the server

[0747] Specific operation: Tap the "Upload" button in the dedicated application to send the file and data to the server.

[0748] Step 3:

[0749] The server converts the audio file into text using the Hyper CLOVA API, and associates the emotion data with the text data and stores it in a database.

[0750] Input: Audio file uploaded to the server, emotion data

[0751] Output: Text data, associated emotion data

[0752] Specific operation: The server sends the audio file to the Hyper CLOVA API, receives it as text data, and stores the emotion data together with the text data in the database.

[0753] Step 4:

[0754] The server uses generative AI to analyze the text data and divide and organize it into three categories: minutes, calendar events, and thank-you email drafts. Emotional data is also used in this analysis.

[0755] Input: Text data, emotion data

[0756] Output: Separated and organized meeting minutes, calendar events, and a draft thank you email

[0757] How it works: The server inputs text data into the generative AI, extracts important information, and classifies it into categories. Content that reflects emotional data is also generated.

[0758] Step 5:

[0759] The server sends each piece of information (minutes, calendar events, and thank-you email draft) to the user's device.

[0760] Input: Separated and organized meeting minutes, calendar events, draft thank you email

[0761] Output: Each piece of information sent to the user's device

[0762] Specific operation: The server sends information to the user's device through a dedicated application.

[0763] Step 6:

[0764] The device will then add the sent calendar event information to a schedule management service such as Google Calendar, and the reminder content will also be customized based on the emotion data.

[0765] Input: Calendar event information, emotion data

[0766] Output: Events added to Google Calendar, customized reminders

[0767] Specific operation: Adds event information to Google Calendar via the Calendar API and sets reminders.

[0768] Step 7:

[0769] The terminal displays the draft of the thank-you email to the user, prompting the user to confirm the content. After the user makes any corrections, the terminal uses the email sending function to send the thank-you email to the customer.

[0770] Input: Draft thank you email, user modifications

[0771] Output: Thank you email sent

[0772] Specific operation: The draft email is displayed on the device, the user makes corrections and taps the "Send" button, and the email is sent to the customer.

[0773] (Application example 2)

[0774] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0775] In conventional sales activities, the tasks of recording sales negotiations, organizing the details, and reporting them were time-consuming and labor-intensive. Furthermore, it was difficult to properly grasp the emotional state of the customer during the negotiation and reflect this in subsequent follow-up, which led to a decline in the quality of communication with the customer. This made it difficult for sales representatives to carry out their work efficiently and respond to customers effectively.

[0776] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0777] In this invention, the server includes a means for recording audio of business negotiations with customers, a means for converting the recorded audio into high-quality text, a means for analyzing the text data and dividing and organizing it into minutes, calendar events, and thank-you email drafts, and a means for inputting the analyzed emotional data into a generative AI model and reflecting it in the information organization process. This allows sales representatives to automate the detailed recording and organization of business negotiations and enable effective follow-up that takes into account the emotional state of the customer.

[0778] The "business negotiation voice recording means" is a hardware or software function for recording audio data during a business negotiation.

[0779] The "text conversion means" is a function that converts recorded voice data into high-quality text data using natural language processing technology.

[0780] "Text analysis means" is a function that uses generative AI to analyze text data and divide and organize it into minutes, calendar events, and thank-you email drafts.

[0781] "Information transmission means" is a communication function for transmitting divided and organized information to the sales representative's terminal.

[0782] The "emotion analysis means" is a function that analyzes the tone of voice and language used by customers and sales representatives during negotiations to grasp their emotional state in real time.

[0783] An "emotion engine" is a software module used as an emotion analysis tool to generate emotion data in real time.

[0784] A "generative AI model" is an artificial intelligence model that performs analysis and information generation based on input text data and emotional data.

[0785] A "prompt" is an instruction statement that allows the generative AI model to properly analyze the text data it inputs.

[0786] The system of the present invention aims to enable sales representatives to efficiently organize business negotiations with customers and to largely automate reporting tasks. This system can be implemented by the following procedure.

[0787] The system configuration requires a smartphone used by the user, an emotion analysis engine, a generative AI model, natural language processing technology, and a server that integrates these. Specifically, the smartphone is equipped with voice recording and data communication functions and connects to the server application.

[0788] The general flow of the system is as follows: First, the salesperson (user) uses their smartphone to record the voice of the sales negotiation. At the same time, the emotion analysis engine analyzes the tone of the user's and customer's voice and the words they use in real time to understand their emotional state.

[0789] Once the negotiation is over, the user uploads the recorded voice data and real-time analyzed emotion data to the server. The server receives the voice data and converts it into high-quality text using natural language processing technology (such as the Google Speech-to-Text API). The emotion data is also stored in a database.

[0790] The server then uses a generative AI model (such as OpenAI's ChatGPT) to analyze the converted text data. The analysis is performed using pre-defined prompts, which are then divided and organized into three main categories: meeting minutes, calendar events, and thank-you email drafts. Specific examples of prompts are as follows:

[0791] "Please parse the following text and classify it into the following categories: meeting minutes, next appointment, thank you email content. Emotion data: {emotionData}

[0792] text:{

[0793] ...negotiation text...

[0794] }"

[0795] The information for each category, divided and organized, along with sentiment analysis data, is fed back to the sales representative's smartphone, allowing them to check meeting minutes, calendar events, and thank-you email drafts on their own screens and make any necessary adjustments.

[0796] Calendar events are automatically added to Google Calendar via API. Reminders are customized based on the customer's emotional data. Thank-you emails are then sent to the customer after the sales representative has reviewed and revised the content.

[0797] The entire system processes and calculates data using the following technical elements:

[0798] Smartphone: Responsible for recording and data uploading.

[0799] Emotion analysis engine: Analyzes emotional state from voice in real time.

[0800] Natural language processing technology: Converts voice data into high-quality text data.

[0801] Generative AI model: Analyzes text data and organizes information into categories.

[0802] Google Calendar API: Used to automatically add calendar events and adjust reminders.

[0803] This allows sales representatives to efficiently record and organize sales negotiation details while also enabling them to conduct high-quality follow-up that takes into account the customer's emotional state.

[0804] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0805] Step 1:

[0806] A user uses a smartphone to record audio during a business meeting.

[0807] Input: Voice during negotiation

[0808] Output: Recorded audio file

[0809] How it works: The user launches the recording app on their smartphone and presses the record button when the sales meeting begins. As the recording progresses, the emotion analysis engine analyzes the tone and vocabulary of the user and customer in real time and records their emotional state.

[0810] Step 2:

[0811] After the business negotiation is completed, the user uploads the recorded voice file and real-time emotion data to the server.

[0812] Input: Recorded audio files, real-time emotion data

[0813] Output: Audio files and emotion data stored on the server

[0814] Specific operation: After the business negotiation is completed, the user presses the upload button in the app. The audio file and emotion data are sent to the server and stored in the database.

[0815] Step 3:

[0816] The server receives the audio file and converts it into high-quality text using natural language processing techniques.

[0817] Input: Recorded audio file

[0818] Output: High-quality text data

[0819] Specific operation: The server analyzes the received audio file and converts it into text data using natural language processing technology (e.g., Google Speech-to-Text API). At the same time, emotion data is also stored in a database.

[0820] Step 4:

[0821] The server uses a generative AI model to analyze the converted text data and divide and organize it into three main categories: meeting minutes, calendar events, and thank-you email drafts.

[0822] Input: High-quality text data, emotion data

[0823] Output: Minutes data, calendar event data, thank you email draft data

[0824] Specific operation: The server uses a generative AI model (e.g., OpenAI's ChatGPT) to analyze and classify data using the following prompt:

[0825] "Please parse the following text and classify it into the following categories: meeting minutes, next appointment, thank you email content. Emotion data: {emotionData}

[0826] text:{

[0827] ...negotiation text...

[0828] }"

[0829] The analysis results are divided into minutes, calendar events, and a draft thank you email.

[0830] Step 5:

[0831] The server then sends the information for each divided and organized category back to the sales representative's smartphone.

[0832] Input: Minutes data, calendar event data, thank you email draft data

[0833] Output: Data displayed on the user's smartphone

[0834] Specific operation: The organized data is sent from the server to the user's smartphone, and the sales representative checks and edits the contents of meeting minutes, calendar events, and thank-you emails.

[0835] Step 6:

[0836] Your device will automatically add calendar events to your Google Calendar and customize reminders based on your emotional data.

[0837] Input: Calendar event data

[0838] Output: Events added to Google Calendar with customized reminders

[0839] Specific operation: An event is added from the user's smartphone via the Google Calendar API, and the reminder content is adjusted based on emotional data.

[0840] Step 7:

[0841] The terminal displays a draft of the thank-you email to the sales representative, who then edits the content and sends it to the customer.

[0842] Input: Draft data for thank you email

[0843] Output: Thank you email sent

[0844] Specific actions: After the sales representative checks and edits the content of the thank-you email, they send it to the customer via the email app on their smartphone.

[0845] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0846] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0847] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0848] [Third embodiment]

[0849] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0850] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0851] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0852] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0853] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0854] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0855] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0856] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0857] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0858] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0859] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0860] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0861] overview

[0862] The system of this invention aims to enable sales representatives to efficiently organize sales negotiations with customers and largely automate reporting tasks. The system includes voice recording, text conversion, text analysis using generative AI, and feedback functions.

[0863] Audio recording

[0864] Users can record audio during business meetings with customers using their smartphones or dedicated recording devices, and the audio files are saved locally for later uploading to the system.

[0865] Uploading an audio file

[0866] After the negotiation is completed, the user uploads the recorded audio file to the server. By using the upload function on the user's device to send the audio file to the server, the system can proceed to the next processing step.

[0867] Speech to text

[0868] The server converts the received audio file into high-quality text using natural language processing technology, specifically converting the audio data into text via an API. This converted text data is stored in a database and goes on to the next analysis step.

[0869] Text analysis, segmentation, and organization

[0870] The server analyzes the converted text data using generative AI. The analysis results are divided and organized into three categories: minutes, calendar events, and thank-you email drafts. Text analysis is performed using generative AI, which automatically extracts important information from the text data.

[0871] minutes

[0872] The server extracts important points and next action items from the text and compiles them into minutes, which include the key points discussed and decisions made during the business meeting.

[0873] Calendar Events

[0874] The server extracts date and time information about upcoming sales meetings and specific events and generates calendar events, allowing salespeople to efficiently manage their upcoming appointments.

[0875] Thank you email draft

[0876] The server generates a draft thank you email, which includes a thank you note for the business meeting with the customer and a brief message about next steps.

[0877] Organized information feedback

[0878] The server sends each piece of information that has been divided and organized to the user's device. The sales representative can then view the minutes, calendar events, and thank-you email draft on their own device and make any necessary revisions.

[0879] Adding an event to Google Calendar

[0880] The device will add the sent calendar event information to Google Calendar, and salespeople can add these events to their own calendars via API, so they won't forget about upcoming meetings or sales calls.

[0881] Sending a thank you email

[0882] The terminal creates and sends an email to the customer based on the draft thank-you email. The user checks the content of the email, makes any necessary corrections, and then presses the send email button, causing the system to send the email to the customer.

[0883] Specific operation example

[0884] 1. The user uses a smartphone to record a conversation during a business meeting.

[0885] 2. After the business meeting is completed, the user uses the application to upload the recording file to the server.

[0886] 3. The server converts the audio file to text using the Hyper CLOVA API.

[0887] 4. The server uses generative AI to analyze the text and extract key information.

[0888] 5. The server divides and organizes the minutes, calendar events, and thank you email drafts.

[0889] 6. The organized information is fed back to the user's device.

[0890] 7. The device adds the fed back calendar event information to Google Calendar.

[0891] 8. The terminal creates an email based on the draft thank-you email and sends it to the customer after the user confirms it.

[0892] In this way, the system of the present invention improves the efficiency of reporting work for sales representatives and reduces their burden.

[0893] The processing flow will be explained below.

[0894] Step 1:

[0895] The user records audio during a business meeting with a customer using a smartphone or recording device, and the audio file is saved in the device's local storage.

[0896] Step 2:

[0897] After the negotiation is over, the user uploads the recorded audio file to the server using a dedicated application. The file is sent by pressing the "upload" button in the application.

[0898] Step 3:

[0899] The server receives the uploaded audio file and starts the process of converting it into high-quality text. Specifically, it sends the audio file to the Hyper CLOVA API, which converts the audio data into text.

[0900] Step 4:

[0901] The server receives the text data returned from the Hyper CLOVA API and stores it in the database. At this point, the voice data has been converted into text data.

[0902] Step 5:

[0903] The server inputs the saved text data into a generative AI for analysis, which then divides it into categories such as minutes, calendar events, and thank-you email drafts.

[0904] Step 6:

[0905] The server stores the analysis results in a database and prepares the divided and organized data for transmission to the sales representative's terminal.

[0906] Step 7:

[0907] The server then sends the divided and organized information back to the user's device, including a meeting minutes, a to-do list, event information for Google Calendar, and a draft thank-you email.

[0908] Step 8:

[0909] The device receives the fed-back Google Calendar event information and automatically calls the Google Calendar API to add it to the calendar, automatically updating the sales representative's schedule.

[0910] Step 9:

[0911] The terminal displays the draft of the thank-you email to the user, prompting the user to confirm the content. After the user makes any corrections, the terminal uses the email sending function to send the thank-you email to the customer.

[0912] Example 1

[0913] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0914] Traditionally, sales representatives' post-negotiation reporting work has been manual, requiring a lot of time and effort. Important information is often overlooked or inaccurately organized, leading to concerns that this can reduce the efficiency of sales activities and impair productivity.

[0915] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0916] In this invention, the server includes a means for recording conversations between salespeople and customers, a means for converting the recorded voice into text, and a means for analyzing the converted text data and dividing and organizing it into summaries, planned events, and thank-you email proposals, thereby reducing the workload and preventing important information from being overlooked.

[0917] A "sales representative" is a position that is responsible for introducing and selling the products and services of a company or organization to customers.

[0918] "Customer" means anyone who purchases or considers purchasing a product or service from a business or organization.

[0919] "Means for recording conversations" refers to devices or software for saving audio recordings of business conversations with customers.

[0920] "Recorded audio" refers to audio data stored digitally via a device during a business meeting.

[0921] "Means of converting to text" refers to the technology or tools used to process audio data and convert its content into written information.

[0922] "Text data" refers to character information converted from audio data.

[0923] "Analyzing" refers to the process of reviewing and understanding the converted text data and extracting important information.

[0924] A "summary" refers to a document that extracts the main points and important information from text data and summarizes them concisely.

[0925] "Planned events" refer to future schedules and action items determined based on the content of the business negotiation.

[0926] A "thank you email draft" refers to a draft email document to be sent to a customer after a business meeting to express gratitude.

[0927] "Means of dividing and organizing" refers to the techniques and tools used to classify and organize the analyzed text data into different categories (summary, upcoming events, thank you email ideas).

[0928] "Terminal" refers to a device (e.g., a smartphone or computer) used to receive, review, or modify information.

[0929] "Transmission means" refers to the technologies and tools used to transfer digital information from one device or system to another.

[0930] "Server" refers to a computer system for storing, processing, and managing data.

[0931] "Data Server" refers to the central computer system for storing and processing audio files and analysis results.

[0932] "Advanced language processing technology" refers to advanced algorithms and technologies used to understand and process natural language, such as natural language processing (NLP) technology.

[0933] overview

[0934] This invention is a system that enables salespeople to efficiently record details of business negotiations with customers and automate reporting. The system has functions for recording voice data, converting it to text, analyzing the text using generative AI, and providing feedback on the information.

[0935] Hardware and software used

[0936] The hardware used by users includes smartphones and dedicated recording devices. The server is a computer system with advanced processing capabilities, and is installed with APIs and generative AI software for processing voice data and analyzing text. Specifically, HyperCLOVA API is used for voice-to-text conversion, and generative AI models are used for text analysis.

[0937] Program processing

[0938] During business meetings, users record conversations using a smartphone or dedicated recording device. When recording is complete, the audio file is automatically saved to the device's local storage. After the meeting is over, the user launches an application and uploads the recorded audio file to the server. The user's device has an upload function for selecting and sending audio files.

[0939] The server analyzes the received audio file and converts it into high-quality text using the Hyper CLOVA API. The converted text data is stored in a database on the server and then analyzed using a generative AI model. As a result of the analysis, the text data is divided and organized into three categories: summary, upcoming events, and thank-you email suggestions.

[0940] The server sends the divided and organized information to the user's device. Specifically, a summary of the meeting minutes, the next scheduled event, and a draft of the thank-you email are fed back to the user's device. The user can then check this information on their own device and make any necessary corrections.

[0941] The device adds the sent calendar event information to Google Calendar. Users can add these events to their own calendars via the Google Calendar API, allowing them to efficiently manage upcoming meetings and business negotiations. The device also creates a thank-you email based on the original draft, and once the user confirms it and sends it, the system automatically sends the email to the customer.

[0942] Examples of concrete examples and prompts

[0943] A concrete example would be the following:

[0944] 1. A user uses a recording app on their smartphone to record a conversation during a business meeting.

[0945] 2. After the negotiation is completed, the user selects the recording file in the application and uploads it to the server.

[0946] 3. The server uses the Hyper CLOVA API to convert the audio file into text data and save it in the database.

[0947] 4. The server analyzes the text data using a generative AI model and segments key information into summaries, upcoming events, and suggested thank-you emails.

[0948] 5. The server sends the organized data to the user's terminal.

[0949] 6. The user's device adds the sent calendar event information to Google Calendar and creates and sends a thank-you email.

[0950] An example of a specific prompt to be input to the generative AI model is as follows:

[0951] "Please analyze the following audio data and divide it into three parts: a summary, upcoming events, and a suggested thank you email. The audio data contains the content of a business meeting."

[0952] In this way, the system of the present invention consistently automates everything from recording voice data to text analysis, organizing information, and providing feedback, significantly reducing the workload of sales representatives.

[0953] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0954] System program processing flow

[0955] Step 1:

[0956] During business meetings, users record conversations using a smartphone or dedicated recording device. The input is "audio data" and the output is "an audio file saved on the local device." Specifically, the user launches a recording app, presses the record button to record the conversation, and the file is saved when recording is complete.

[0957] Step 2:

[0958] After the business meeting is over, the user uploads the recorded audio file to the server. The input is the "audio file saved on the local device" and the output is the "audio file uploaded to the server." Specifically, the user launches the application, selects the audio file using the upload function, and presses the send button.

[0959] Step 3:

[0960] The server receives the uploaded audio file and converts it into high-quality text. The input is the "audio file uploaded to the server" and the output is "text data stored in the database." Specifically, the server uses the Hyper CLOVA API to convert the audio data into text data and stores the text data in the database.

[0961] Step 4:

[0962] The server analyzes the converted text data using generative AI. The input is "text data stored in the database" and the output is "analyzed information (summary, scheduled events, and thank-you email suggestions)." Specifically, the server uses a generative AI model to analyze the text data and divide and organize it into summaries, scheduled events, and thank-you email suggestions.

[0963] Step 5:

[0964] The server sends the organized information to the user's device. The input is the analyzed information (summary, scheduled events, and thank-you email suggestions), and the output is the information sent to the user's device. Specifically, the server generates the summary, scheduled events, and thank-you email suggestions and sends them to the user's device.

[0965] Step 6:

[0966] The device adds the sent calendar event information to Google Calendar. The input is "calendar event information sent to the user's device" and the output is "event added to Google Calendar." Specifically, the device uses the Google Calendar API to add the fed-back calendar event information to the user's Google Calendar.

[0967] Step 7:

[0968] The terminal creates an email to the customer based on the draft thank you email, and sends it after the user has confirmed it. The input is the "draft thank you email sent to the user's terminal," and the output is the "thank you email sent to the customer." Specifically, the terminal displays the draft thank you email, and the user confirms and edits the content, then presses the send button to send the email to the customer.

[0969] In this way, the system of the present invention processes and calculates data at each processing step, thereby improving the efficiency of salespeople's work.

[0970] (Application example 1)

[0971] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0972] To improve the quality of customer service in brick-and-mortar stores, there is a need for technology that can efficiently record, analyze, and organize the content of conversations between employees and customers. However, current methods require a large portion of the work to be done manually, which is inefficient and places a heavy burden on employees. Furthermore, the content of conversations with customers cannot be used appropriately, making it difficult to utilize in business negotiations or future interactions. To solve these issues, it is necessary to introduce a system that converts speech to text and automatically classifies and organizes it.

[0973] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0974] In this invention, the server includes a means for recording audio of business negotiations with customers, a means for converting the recorded audio into high-quality text, a means for analyzing the text data and dividing and organizing it into minutes, scheduled events, and thank-you email drafts, and a means for feeding back the analyzed data, thereby making it possible to significantly improve the efficiency and quality of face-to-face interactions by employees in physical stores.

[0975] "Customer" means a visitor or user with whom you engage in face-to-face interaction to purchase a product or service.

[0976] "Business negotiations" refer to conversations and negotiations regarding products and services with customers.

[0977] "Audio recording means" refers to a device or system for recording conversations during business negotiations.

[0978] "High-quality text" refers to text data that can accurately convert speech into text and properly reproduce the content of business negotiations.

[0979] "Means for analyzing text data" refers to a device or system that uses a generative AI model to analyze text data and divide and organize it into specific categories.

[0980] A "minutes" is a document that records important matters discussed and decisions made during a business meeting.

[0981] "Scheduled event" refers to the date and time information of the next meeting or specific activity in a business negotiation.

[0982] A "thank you email draft" is a draft of an email to be sent to a customer after a business meeting to express gratitude.

[0983] "Feedback means" refers to a device or system that provides analyzed data to a user.

[0984] "Server" refers to a computer system that processes voice data and divides and organizes the analysis results.

[0985] A "terminal" is an electronic device used by a sales representative to display analyzed information.

[0986] The system of the present invention is designed to improve the efficiency and quality of customer service in brick-and-mortar stores. The system includes voice recording, speech-to-text conversion, text analysis using generative AI models, and feedback functionality.

[0987] Audio recording

[0988] A user uses an electronic device to record conversations with customers. The hardware used for this is a smartphone or other audio recording device. The audio data captured by the audio recording device is temporarily stored on a local device.

[0989] Uploading an audio file

[0990] After the negotiation is over, the user uploads the recorded audio file to the server using the application. This upload function is installed on the user's device and has the function of sending the recorded file to the server.

[0991] Speech to text

[0992] The server converts the received audio file into high-quality text. Natural language processing technology is used in this process. Specifically, the audio data is converted into text data using an API implemented on the server. An example of this API is the Google Speech Recognition API. The converted text data is stored in a database and moves on to the next analysis step.

[0993] Text analysis, segmentation, and organization

[0994] The server analyzes the converted text data using a generative AI model (e.g., OpenAI's GPT-3). This analysis divides and organizes the data into three categories: minutes, scheduled events, and draft thank-you emails. This automatically extracts important information, next action items, and date and time information.

[0995] Organized information feedback

[0996] The server sends the divided and organized information to the user's device and provides feedback to the user. The user can then check the minutes, scheduled events, and thank-you email draft on their own device and make any necessary corrections.

[0997] Adding an event to Google Calendar

[0998] The device can add the sent scheduled event information to Google Calendar, and the event will be automatically reflected on the calendar using the Google Calendar API.

[0999] Sending a thank you email

[1000] The terminal creates an email to the customer based on the draft of the thank-you email and sends it. In this process, the user checks the content of the email and makes any necessary corrections before sending it.

[1001] Specific examples

[1002] For example, if a conversation is recorded during a sales conversation with a customer saying, "Hello, can you tell me more about this product?", the system would work as follows:

[1003] 1. Use your smartphone to record conversations during business negotiations.

[1004] 2. After the business meeting is completed, the user uses the application to upload the recording file to the server.

[1005] 3. The server converts the audio file into text using the Google Speech Recognition API.

[1006] 4. The server uses a generative AI model to analyze the text and extract key information.

[1007] 5. The server divides and organizes the minutes, scheduled events, and thank you email drafts.

[1008] 6. The organized information is fed back to the user's device.

[1009] Prompt Sentence Examples

[1010] For example, the prompt used during parsing is:

[1011] "Please parse the following text and split it into meeting minutes, planned events, and a draft thank you email.\n\nHello, can you tell me more about this product?"

[1012] In this way, the present invention improves the efficiency of customer service in brick-and-mortar stores, reduces the burden on employees, and improves customer satisfaction.

[1013] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1014] Step 1:

[1015] A user records a conversation during a business meeting.

[1016] Input: Customer conversation

[1017] Output: Recorded audio file

[1018] What it does: A user uses a smartphone or other audio recording device to record a conversation during a business meeting. The audio file is saved locally on the device.

[1019] Step 2:

[1020] The user uploads the recorded audio file to the server.

[1021] Input: Audio file stored on the local device

[1022] Output: Audio file uploaded to the server

[1023] Specific operation: After the business meeting is completed, the user uploads the recorded audio file to the server using a dedicated application. The file is sent to the server using the application's upload function.

[1024] Step 3:

[1025] The server converts the audio file into high-quality text.

[1026] Input: Uploaded audio file

[1027] Output: Text data

[1028] How it works: The server receives the audio file and converts it into text using natural language processing technology such as Google Speech Recognition API. This text data is then stored in a database.

[1029] Step 4:

[1030] The server analyzes the text data using a generative AI model.

[1031] Input: Text data converted from audio

[1032] Output: Meeting minutes, scheduled events, draft thank you email

[1033] Specific operation: The server inputs the text data into a generative AI model (e.g., OpenAI's GPT-3), and divides and organizes the analysis results into three categories: minutes, scheduled events, and a draft thank-you email. An example of a prompt used during analysis is "Analyze the following text and divide it into minutes, scheduled events, and a draft thank-you email.\n\n(text data)".

[1034] Step 5:

[1035] The server feeds back the divided and organized information to the user's device.

[1036] Input: Minutes, scheduled events, and thank you email draft data

[1037] Output: Information fed back to the user's device

[1038] Specific operation: The server sends the analyzed data to the user's device, where the user can review it. The user can then edit the meeting minutes, scheduled events, and thank-you email drafts on their device as appropriate.

[1039] Step 6:

[1040] The device adds the sent scheduled event information to Google Calendar.

[1041] Input: Scheduled event information

[1042] Output: Scheduled events reflected in Google Calendar

[1043] Specific operation: The device uses the Google Calendar API to add the scheduled event information sent from the server to the Google Calendar.

[1044] Step 7:

[1045] The terminal creates and sends an email to the customer based on the original thank-you email draft.

[1046] Input: Draft thank you email

[1047] Output: Thank you email sent to customer

[1048] Specific operation: The terminal creates an email based on the draft thank-you email, and after the user checks the content and makes any necessary corrections, the email is sent to the customer.

[1049] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1050] overview

[1051] The system of the present invention not only enables salespeople to efficiently organize sales negotiations with customers and largely automates reporting tasks, but also incorporates an emotion engine to recognize and reflect the user's emotional state. The system includes voice recording, text conversion, generative AI text analysis, emotion recognition, and feedback functions.

[1052] Audio recording

[1053] Users record their voices during sales conversations with customers using their smartphones or dedicated recording devices, and the audio files are saved in the device's local storage. An emotion engine runs in parallel to analyze the user's real-time reactions and tone of voice.

[1054] Uploading an audio file

[1055] After the negotiation is over, the user uploads the recorded audio file to the server using a dedicated application. During the upload process, the emotion data analyzed by the emotion engine is also sent to the server.

[1056] Speech to text

[1057] The server converts the received audio file into high-quality text. This process uses natural language processing technology, specifically converting the audio data into text via an API. Emotion data analyzed by the emotion engine is also stored in association with the text data.

[1058] Text analysis, segmentation, and organization

[1059] The server inputs the converted text data into a generative AI for analysis. The analysis results are divided and organized into three categories: minutes, calendar events, and thank-you email drafts. Emotion data recognized by the emotion engine is also used in the analysis process.

[1060] minutes

[1061] The server extracts important points and next action items from the text and compiles them into minutes, which include the key points discussed and decisions made during the business meeting.

[1062] Calendar Events

[1063] The server extracts date and time information for upcoming business meetings or specific events, generates calendar events, and customizes reminder notifications based on the emotional data analyzed by the emotion engine.

[1064] Thank you email draft

[1065] The server generates a draft thank-you email, which includes a brief message expressing gratitude for the business interaction and a suggestion for next steps. The appropriate tone and content are automatically adjusted based on data from the emotion engine.

[1066] Organized information feedback

[1067] The server sends each piece of information that has been divided and organized to the user's device. The sales representative can then view the minutes, calendar events, and thank-you email draft on their own device and make any necessary revisions.

[1068] Adding an event to Google Calendar

[1069] The device will add the sent calendar event information to Google Calendar, and salespeople can add these events to their own calendars via API, so they won't forget about upcoming meetings or sales calls.

[1070] Sending a thank you email

[1071] The terminal displays the draft of the thank-you email to the user, prompting the user to confirm the content. After the user makes any corrections, the terminal uses the email sending function to send the thank-you email to the customer.

[1072] Specific operation example

[1073] 1. The user uses their smartphone to record conversations during business negotiations, and the emotion engine simultaneously analyzes the emotional state in real time.

[1074] 2. After the business meeting is completed, the user uploads the recording file and emotion data to the server using the application.

[1075] 3. The server converts the audio file into text using the Hyper CLOVA API and stores the emotion data in the database in parallel.

[1076] 4. The server uses generative AI to analyze the text and extract key information, incorporating emotional data into the analysis process.

[1077] 5. The server divides and organizes the data into minutes, calendar events, and thank-you email drafts, and reflects the emotional data in each.

[1078] 6. The organized information is fed back to the user's device.

[1079] 7. When the device adds the fed-back calendar event information to Google Calendar, it customizes the reminder content based on the emotion data.

[1080] 8. The terminal displays a draft of the thank-you email to the user, and after having the user confirm that the content has been adjusted based on the emotional data, the email is sent to the customer.

[1081] In this way, the system of the present invention improves the efficiency of reporting work by sales representatives and also takes into account the emotional aspects of communication with customers, enabling higher quality follow-up.

[1082] The processing flow will be explained below.

[1083] Step 1:

[1084] During a business meeting with a customer, the user starts recording audio using a smartphone or recording device. The recording is made using a dedicated application, and the emotion engine simultaneously analyzes the user's emotional state in real time. The audio and emotion data are stored in the device's local storage.

[1085] Step 2:

[1086] After the negotiation is over, the user uploads the recorded voice file and emotional data to the server using a dedicated application. By pressing the "upload" button in the application, the emotional data is sent to the server along with the voice file.

[1087] Step 3:

[1088] The server receives the uploaded audio file and first converts the audio data into high-quality text via the Hyper CLOVA API. At this time, the audio data and emotion data are linked and stored synchronously.

[1089] Step 4:

[1090] The server stores the converted text data and the linked emotion data in a database, allowing the text data and emotion information to be stored in an integrated format.

[1091] Step 5:

[1092] The server inputs the saved text data into a generative AI system, which analyzes it and divides and organizes it into three categories: minutes, calendar events, and thank-you email drafts. Emotional data is also used in the analysis process.

[1093] Step 6:

[1094] The server generates information based on text analysis and sentiment data as follows:

[1095] Minutes: Extract important points and next action items from the text, and summarize them by taking into account their importance based on emotional data.

[1096] Calendar Events: Extract time and date information for upcoming sales meetings or specific events and customize reminders based on sentiment data.

[1097] Thank you email draft: Generates a draft of a thank you email to send to a customer, automatically adjusting the appropriate tone and content based on emotional data.

[1098] Step 7:

[1099] The server then feeds back the generated minutes, calendar events, and thank-you email drafts to the user's device, including adjustments that take into account emotional information.

[1100] Step 8:

[1101] The device adds the received calendar event information to Google Calendar. Specifically, it calls the Google Calendar API to add an event that reflects the reminder notification content based on the emotion data to the calendar.

[1102] Step 9:

[1103] The device displays a draft of the thank-you email to the user. The user checks the email content and makes any necessary corrections. The device then uses its email sending function to send the thank-you email to the customer, with the content adjusted based on the emotional data.

[1104] Through the above steps, the system of the present invention improves the efficiency of sales representatives' reporting work, and also takes into account the emotional aspects of communication with customers, enabling more personalized responses.

[1105] Example 2

[1106] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1107] While traditional sales negotiation systems can record voice and convert it into text, they do not include emotional analysis or the utilization of that data, which means the quality of communication with customers is not fully considered. This can lead to poor customer service efficiency and the risk of missing important market opportunities. Furthermore, organizing and reporting sales negotiation results takes a lot of time, preventing work efficiency.

[1108] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1109] In this invention, the server includes means for recording audio of business negotiations with customers, means for converting the recorded audio into high-quality text, means for analyzing the text data and dividing and organizing it into minutes, calendar events, and thank-you email drafts, means for analyzing emotional states and reflecting the results in data, means for sending the divided and organized information to the sales representative's terminal, means for performing real-time emotional analysis during customer visits and uploading recordings including the analyzed data to the server, means for adding the transmitted calendar event information to a schedule management service, and means for displaying the thank-you email draft on the sales representative's terminal and sending it to the customer after revision. This enables efficient organization of business negotiations with customers and high-quality customer service utilizing emotional data.

[1110] - "Customer" refers to a business partner or other person with whom we have a business relationship.

[1111] "Business negotiation" refers to a place for negotiation or consultation for business purposes.

[1112] "Audio recording" refers to the act of electronically recording sound.

[1113] "Text conversion" refers to the process of converting voice data into text format.

[1114] "Text data" refers to data in the form of text converted from speech.

[1115] "Minutes" refers to a document that records what was said and what decisions were made at a meeting or business negotiation.

[1116] "Calendar event" refers to an appointment or schedule that includes date and time information.

[1117] A "thank you email draft" refers to a draft of an email to be sent to a customer to express gratitude.

[1118] "Emotional state" refers to the speaker's feelings or state of mind.

[1119] "Sentiment analysis" refers to the process of analyzing emotions from speech or text.

[1120] A "database" refers to a system that efficiently stores and manages large amounts of data.

[1121] "Schedule management service" refers to an online service that electronically manages schedules and appointments.

[1122] "Upload" refers to the act of sending data from local storage to a server.

[1123] "Generative AI" refers to a system that uses artificial intelligence to generate or analyze data.

[1124] "API" stands for Application Program Interface and refers to a set of rules that enable communication between software programs.

[1125] System Configuration

[1126] The system of the present invention is designed to help salespeople efficiently organize sales negotiations with customers and largely automate reporting tasks. It also incorporates an emotion engine to recognize the user's emotional state and reflect it in the data. The system has the following main functions:

[1127] Audio recording

[1128] Users record audio during business meetings with customers. This recording is done using a smartphone or a dedicated recording device, and the audio file is saved in the device's local storage. Specific hardware used is a standard smartphone or portable recorder. During the recording, the emotion engine analyzes the user's tone of voice and reactions in real time to collect emotional data.

[1129] Uploading an audio file

[1130] After the negotiation is over, the user uploads the recorded voice file and emotional data to the server using a dedicated application. During the upload process, the emotional data analyzed by the emotion engine is also sent to the server.

[1131] Speech to text

[1132] The server uses natural language processing technology to convert the uploaded audio file into high-quality text. Specifically, it uses the Hyper CLOVA API to convert the audio data into text. The emotion data is associated with this text data and stored in a database.

[1133] Text analysis, segmentation, and organization

[1134] The server uses generative AI to analyze the text data. The analysis results are divided and organized into three categories: minutes, calendar events, and thank-you email drafts. Emotional data is also incorporated into this analysis process, generating content based on the user's emotional state.

[1135] Information Feedback

[1136] The server sends each piece of information that has been divided and organized to the user's device. The salesperson can then review the minutes, calendar events, and draft thank-you emails on their device and make any necessary revisions. Furthermore, the device can add the sent calendar event information to a schedule management service such as Google Calendar and set reminders based on the emotion data.

[1137] Sending a thank you email

[1138] The terminal displays the draft of the thank you email to the user for confirmation. After the user makes any necessary corrections, the terminal uses the email sending function to send the thank you email to the customer.

[1139] Specific examples

[1140] For example, the prompt when analyzing voice data from a business meeting might look like this:

[1141] Example prompt: "Transcribe the audio data below into text, summarize the key points, and draft a thank you email with an appropriate tone based on the emotional data."

[1142] Audio data: (Audio file link or file name)

[1143] As can be seen from this specific operation example, the system of the present invention highly automates a series of business processes, from voice recording to text analysis, incorporating emotional data, and generating thank-you emails. This significantly reduces the workload of sales representatives and improves the quality of communication with customers.

[1144] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1145] Step 1:

[1146] Users record their business meetings using a voice recording device (smartphone or dedicated recorder). While recording, the emotion engine works in parallel, analyzing the user's tone of voice and reactions in real time.

[1147] Input: Voice data (recorded business meetings), real-time user emotional responses

[1148] Output: Recorded audio files, real-time emotion data

[1149] Specific operation: Launch the recording app and start recording as soon as the sales meeting begins. The emotion engine analyzes the tone of the voice and collects emotional data.

[1150] Step 2:

[1151] After the business negotiation is completed, the user uploads the recorded voice file and emotion data to the server using a dedicated application.

[1152] Input: Recorded audio file, emotion data

[1153] Output: Audio files and emotion data uploaded to the server

[1154] Specific operation: Tap the "Upload" button in the dedicated application to send the file and data to the server.

[1155] Step 3:

[1156] The server converts the audio file into text using the Hyper CLOVA API, and associates the emotion data with the text data and stores it in a database.

[1157] Input: Audio file uploaded to the server, emotion data

[1158] Output: Text data, associated emotion data

[1159] Specific operation: The server sends the audio file to the Hyper CLOVA API, receives it as text data, and stores the emotion data together with the text data in the database.

[1160] Step 4:

[1161] The server uses generative AI to analyze the text data and divide and organize it into three categories: minutes, calendar events, and thank-you email drafts. Emotional data is also used in this analysis.

[1162] Input: Text data, emotion data

[1163] Output: Separated and organized meeting minutes, calendar events, and a draft thank you email

[1164] How it works: The server inputs text data into the generative AI, extracts important information, and classifies it into categories. Content that reflects emotional data is also generated.

[1165] Step 5:

[1166] The server sends each piece of information (minutes, calendar events, and thank-you email draft) to the user's device.

[1167] Input: Separated and organized meeting minutes, calendar events, draft thank you email

[1168] Output: Each piece of information sent to the user's device

[1169] Specific operation: The server sends information to the user's device through a dedicated application.

[1170] Step 6:

[1171] The device will then add the sent calendar event information to a schedule management service such as Google Calendar, and the reminder content will also be customized based on the emotion data.

[1172] Input: Calendar event information, emotion data

[1173] Output: Events added to Google Calendar, customized reminders

[1174] Specific operation: Adds event information to Google Calendar via the Calendar API and sets reminders.

[1175] Step 7:

[1176] The terminal displays the draft of the thank-you email to the user, prompting the user to confirm the content. After the user makes any corrections, the terminal uses the email sending function to send the thank-you email to the customer.

[1177] Input: Draft thank you email, user modifications

[1178] Output: Thank you email sent

[1179] Specific operation: The draft email is displayed on the device, the user makes corrections and taps the "Send" button, and the email is sent to the customer.

[1180] (Application example 2)

[1181] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1182] In conventional sales activities, the tasks of recording sales negotiations, organizing the details, and reporting them were time-consuming and labor-intensive. Furthermore, it was difficult to properly grasp the emotional state of the customer during the negotiation and reflect this in subsequent follow-up, which led to a decline in the quality of communication with the customer. This made it difficult for sales representatives to carry out their work efficiently and respond to customers effectively.

[1183] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1184] In this invention, the server includes a means for recording audio of business negotiations with customers, a means for converting the recorded audio into high-quality text, a means for analyzing the text data and dividing and organizing it into minutes, calendar events, and thank-you email drafts, and a means for inputting the analyzed emotional data into a generative AI model and reflecting it in the information organization process. This allows sales representatives to automate the detailed recording and organization of business negotiations and enable effective follow-up that takes into account the emotional state of the customer.

[1185] The "business negotiation voice recording means" is a hardware or software function for recording audio data during a business negotiation.

[1186] The "text conversion means" is a function that converts recorded voice data into high-quality text data using natural language processing technology.

[1187] "Text analysis means" is a function that uses generative AI to analyze text data and divide and organize it into minutes, calendar events, and thank-you email drafts.

[1188] "Information transmission means" is a communication function for transmitting divided and organized information to the sales representative's terminal.

[1189] The "emotion analysis means" is a function that analyzes the tone of voice and language used by customers and sales representatives during negotiations to grasp their emotional state in real time.

[1190] An "emotion engine" is a software module used as an emotion analysis tool to generate emotion data in real time.

[1191] A "generative AI model" is an artificial intelligence model that performs analysis and information generation based on input text data and emotional data.

[1192] A "prompt" is an instruction statement that allows the generative AI model to properly analyze the text data it inputs.

[1193] The system of the present invention aims to enable sales representatives to efficiently organize business negotiations with customers and to largely automate reporting tasks. This system can be implemented by the following procedure.

[1194] The system configuration requires a smartphone used by the user, an emotion analysis engine, a generative AI model, natural language processing technology, and a server that integrates these. Specifically, the smartphone is equipped with voice recording and data communication functions and connects to the server application.

[1195] The general flow of the system is as follows: First, the salesperson (user) uses their smartphone to record the voice of the sales negotiation. At the same time, the emotion analysis engine analyzes the tone of the user's and customer's voice and the words they use in real time to understand their emotional state.

[1196] Once the negotiation is over, the user uploads the recorded voice data and real-time analyzed emotion data to the server. The server receives the voice data and converts it into high-quality text using natural language processing technology (such as the Google Speech-to-Text API). The emotion data is also stored in a database.

[1197] The server then uses a generative AI model (such as OpenAI's ChatGPT) to analyze the converted text data. The analysis is performed using pre-defined prompts, which are then divided and organized into three main categories: meeting minutes, calendar events, and thank-you email drafts. Specific examples of prompts are as follows:

[1198] "Please parse the following text and classify it into the following categories: meeting minutes, next appointment, thank you email content. Emotion data: {emotionData}

[1199] text:{

[1200] ...negotiation text...

[1201] }"

[1202] The information for each category, divided and organized, along with sentiment analysis data, is fed back to the sales representative's smartphone, allowing them to check meeting minutes, calendar events, and thank-you email drafts on their own screens and make any necessary adjustments.

[1203] Calendar events are automatically added to Google Calendar via API. Reminders are customized based on the customer's emotional data. Thank-you emails are then sent to the customer after the sales representative has reviewed and revised the content.

[1204] The entire system processes and calculates data using the following technical elements:

[1205] Smartphone: Responsible for recording and data uploading.

[1206] Emotion analysis engine: Analyzes emotional state from voice in real time.

[1207] Natural language processing technology: Converts voice data into high-quality text data.

[1208] Generative AI model: Analyzes text data and organizes information into categories.

[1209] Google Calendar API: Used to automatically add calendar events and adjust reminders.

[1210] This allows sales representatives to efficiently record and organize sales negotiation details while also enabling them to conduct high-quality follow-up that takes into account the customer's emotional state.

[1211] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1212] Step 1:

[1213] A user uses a smartphone to record audio during a business meeting.

[1214] Input: Voice during negotiation

[1215] Output: Recorded audio file

[1216] How it works: The user launches the recording app on their smartphone and presses the record button when the sales meeting begins. As the recording progresses, the emotion analysis engine analyzes the tone and vocabulary of the user and customer in real time and records their emotional state.

[1217] Step 2:

[1218] After the business negotiation is completed, the user uploads the recorded voice file and real-time emotion data to the server.

[1219] Input: Recorded audio files, real-time emotion data

[1220] Output: Audio files and emotion data stored on the server

[1221] Specific operation: After the business negotiation is completed, the user presses the upload button in the app. The audio file and emotion data are sent to the server and stored in the database.

[1222] Step 3:

[1223] The server receives the audio file and converts it into high-quality text using natural language processing techniques.

[1224] Input: Recorded audio file

[1225] Output: High-quality text data

[1226] Specific operation: The server analyzes the received audio file and converts it into text data using natural language processing technology (e.g., Google Speech-to-Text API). At the same time, emotion data is also stored in a database.

[1227] Step 4:

[1228] The server uses a generative AI model to analyze the converted text data and divide and organize it into three main categories: meeting minutes, calendar events, and thank-you email drafts.

[1229] Input: High-quality text data, emotion data

[1230] Output: Minutes data, calendar event data, thank you email draft data

[1231] Specific operation: The server uses a generative AI model (e.g., OpenAI's ChatGPT) to analyze and classify data using the following prompt:

[1232] "Please parse the following text and classify it into the following categories: meeting minutes, next appointment, thank you email content. Emotion data: {emotionData}

[1233] text:{

[1234] ...negotiation text...

[1235] }"

[1236] The analysis results are divided into minutes, calendar events, and a draft thank you email.

[1237] Step 5:

[1238] The server then sends the information for each divided and organized category back to the sales representative's smartphone.

[1239] Input: Minutes data, calendar event data, thank you email draft data

[1240] Output: Data displayed on the user's smartphone

[1241] Specific operation: The organized data is sent from the server to the user's smartphone, and the sales representative checks and edits the contents of meeting minutes, calendar events, and thank-you emails.

[1242] Step 6:

[1243] Your device will automatically add calendar events to your Google Calendar and customize reminders based on your emotional data.

[1244] Input: Calendar event data

[1245] Output: Events added to Google Calendar with customized reminders

[1246] Specific operation: An event is added from the user's smartphone via the Google Calendar API, and the reminder content is adjusted based on emotional data.

[1247] Step 7:

[1248] The terminal displays a draft of the thank-you email to the sales representative, who then edits the content and sends it to the customer.

[1249] Input: Draft data for thank you email

[1250] Output: Thank you email sent

[1251] Specific actions: After the sales representative checks and edits the content of the thank-you email, they send it to the customer via the email app on their smartphone.

[1252] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1253] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1254] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1255] [Fourth embodiment]

[1256] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1257] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1258] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1259] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1260] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1261] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1262] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1263] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1264] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1265] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1266] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1267] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1268] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1269] overview

[1270] The system of this invention aims to enable sales representatives to efficiently organize sales negotiations with customers and largely automate reporting tasks. The system includes voice recording, text conversion, text analysis using generative AI, and feedback functions.

[1271] Audio recording

[1272] Users can record audio during business meetings with customers using their smartphones or dedicated recording devices, and the audio files are saved locally for later uploading to the system.

[1273] Uploading an audio file

[1274] After the negotiation is completed, the user uploads the recorded audio file to the server. By using the upload function on the user's device to send the audio file to the server, the system can proceed to the next processing step.

[1275] Speech to text

[1276] The server converts the received audio file into high-quality text using natural language processing technology, specifically converting the audio data into text via an API. This converted text data is stored in a database and goes on to the next analysis step.

[1277] Text analysis, segmentation, and organization

[1278] The server analyzes the converted text data using generative AI. The analysis results are divided and organized into three categories: minutes, calendar events, and thank-you email drafts. Text analysis is performed using generative AI, which automatically extracts important information from the text data.

[1279] minutes

[1280] The server extracts important points and next action items from the text and compiles them into minutes, which include the key points discussed and decisions made during the business meeting.

[1281] Calendar Events

[1282] The server extracts date and time information about upcoming sales meetings and specific events and generates calendar events, allowing salespeople to efficiently manage their upcoming appointments.

[1283] Thank you email draft

[1284] The server generates a draft thank you email, which includes a thank you note for the business meeting with the customer and a brief message about next steps.

[1285] Organized information feedback

[1286] The server sends each piece of information that has been divided and organized to the user's device. The sales representative can then view the minutes, calendar events, and thank-you email draft on their own device and make any necessary revisions.

[1287] Adding an event to Google Calendar

[1288] The device will add the sent calendar event information to Google Calendar, and salespeople can add these events to their own calendars via API, so they won't forget about upcoming meetings or sales calls.

[1289] Sending a thank you email

[1290] The terminal creates and sends an email to the customer based on the draft thank-you email. The user checks the content of the email, makes any necessary corrections, and then presses the send email button, causing the system to send the email to the customer.

[1291] Specific operation example

[1292] 1. The user uses a smartphone to record a conversation during a business meeting.

[1293] 2. After the business meeting is completed, the user uses the application to upload the recording file to the server.

[1294] 3. The server converts the audio file to text using the Hyper CLOVA API.

[1295] 4. The server uses generative AI to analyze the text and extract key information.

[1296] 5. The server divides and organizes the minutes, calendar events, and thank you email drafts.

[1297] 6. The organized information is fed back to the user's device.

[1298] 7. The device adds the fed back calendar event information to Google Calendar.

[1299] 8. The terminal creates an email based on the draft thank-you email and sends it to the customer after the user confirms it.

[1300] In this way, the system of the present invention improves the efficiency of reporting work for sales representatives and reduces their burden.

[1301] The processing flow will be explained below.

[1302] Step 1:

[1303] The user records audio during a business meeting with a customer using a smartphone or recording device, and the audio file is saved in the device's local storage.

[1304] Step 2:

[1305] After the negotiation is over, the user uploads the recorded audio file to the server using a dedicated application. The file is sent by pressing the "upload" button in the application.

[1306] Step 3:

[1307] The server receives the uploaded audio file and starts the process of converting it into high-quality text. Specifically, it sends the audio file to the Hyper CLOVA API, which converts the audio data into text.

[1308] Step 4:

[1309] The server receives the text data returned from the Hyper CLOVA API and stores it in the database. At this point, the voice data has been converted into text data.

[1310] Step 5:

[1311] The server inputs the saved text data into a generative AI for analysis, which then divides it into categories such as minutes, calendar events, and thank-you email drafts.

[1312] Step 6:

[1313] The server stores the analysis results in a database and prepares the divided and organized data for transmission to the sales representative's terminal.

[1314] Step 7:

[1315] The server then sends the divided and organized information back to the user's device, including a meeting minutes, a to-do list, event information for Google Calendar, and a draft thank-you email.

[1316] Step 8:

[1317] The device receives the fed-back Google Calendar event information and automatically calls the Google Calendar API to add it to the calendar, automatically updating the sales representative's schedule.

[1318] Step 9:

[1319] The terminal displays the draft of the thank-you email to the user, prompting the user to confirm the content. After the user makes any corrections, the terminal uses the email sending function to send the thank-you email to the customer.

[1320] Example 1

[1321] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1322] Traditionally, sales representatives' post-negotiation reporting work has been manual, requiring a lot of time and effort. Important information is often overlooked or inaccurately organized, leading to concerns that this can reduce the efficiency of sales activities and impair productivity.

[1323] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1324] In this invention, the server includes a means for recording conversations between salespeople and customers, a means for converting the recorded voice into text, and a means for analyzing the converted text data and dividing and organizing it into summaries, planned events, and thank-you email proposals, thereby reducing the workload and preventing important information from being overlooked.

[1325] A "sales representative" is a position that is responsible for introducing and selling the products and services of a company or organization to customers.

[1326] "Customer" means anyone who purchases or considers purchasing a product or service from a business or organization.

[1327] "Means for recording conversations" refers to devices or software for saving audio recordings of business conversations with customers.

[1328] "Recorded audio" refers to audio data stored digitally via a device during a business meeting.

[1329] "Means of converting to text" refers to the technology or tools used to process audio data and convert its content into written information.

[1330] "Text data" refers to character information converted from audio data.

[1331] "Analyzing" refers to the process of reviewing and understanding the converted text data and extracting important information.

[1332] A "summary" refers to a document that extracts the main points and important information from text data and summarizes them concisely.

[1333] "Planned events" refer to future schedules and action items determined based on the content of the business negotiation.

[1334] A "thank you email draft" refers to a draft email document to be sent to a customer after a business meeting to express gratitude.

[1335] "Means of dividing and organizing" refers to the techniques and tools used to classify and organize the analyzed text data into different categories (summary, upcoming events, thank you email ideas).

[1336] "Terminal" refers to a device (e.g., a smartphone or computer) used to receive, review, or modify information.

[1337] "Transmission means" refers to the technologies and tools used to transfer digital information from one device or system to another.

[1338] "Server" refers to a computer system for storing, processing, and managing data.

[1339] "Data Server" refers to the central computer system for storing and processing audio files and analysis results.

[1340] "Advanced language processing technology" refers to advanced algorithms and technologies used to understand and process natural language, such as natural language processing (NLP) technology.

[1341] overview

[1342] This invention is a system that enables salespeople to efficiently record details of business negotiations with customers and automate reporting. The system has functions for recording voice data, converting it to text, analyzing the text using generative AI, and providing feedback on the information.

[1343] Hardware and software used

[1344] The hardware used by users includes smartphones and dedicated recording devices. The server is a computer system with advanced processing capabilities, and is installed with APIs and generative AI software for processing voice data and analyzing text. Specifically, HyperCLOVA API is used for voice-to-text conversion, and generative AI models are used for text analysis.

[1345] Program processing

[1346] During business meetings, users record conversations using a smartphone or dedicated recording device. When recording is complete, the audio file is automatically saved to the device's local storage. After the meeting is over, the user launches an application and uploads the recorded audio file to the server. The user's device has an upload function for selecting and sending audio files.

[1347] The server analyzes the received audio file and converts it into high-quality text using the Hyper CLOVA API. The converted text data is stored in a database on the server and then analyzed using a generative AI model. As a result of the analysis, the text data is divided and organized into three categories: summary, upcoming events, and thank-you email suggestions.

[1348] The server sends the divided and organized information to the user's device. Specifically, a summary of the meeting minutes, the next scheduled event, and a draft of the thank-you email are fed back to the user's device. The user can then check this information on their own device and make any necessary corrections.

[1349] The device adds the sent calendar event information to Google Calendar. Users can add these events to their own calendars via the Google Calendar API, allowing them to efficiently manage upcoming meetings and business negotiations. The device also creates a thank-you email based on the original draft, and once the user confirms it and sends it, the system automatically sends the email to the customer.

[1350] Examples of concrete examples and prompts

[1351] A concrete example would be the following:

[1352] 1. A user uses a recording app on their smartphone to record a conversation during a business meeting.

[1353] 2. After the negotiation is completed, the user selects the recording file in the application and uploads it to the server.

[1354] 3. The server uses the Hyper CLOVA API to convert the audio file into text data and save it in the database.

[1355] 4. The server analyzes the text data using a generative AI model and segments key information into summaries, upcoming events, and suggested thank-you emails.

[1356] 5. The server sends the organized data to the user's terminal.

[1357] 6. The user's device adds the sent calendar event information to Google Calendar and creates and sends a thank-you email.

[1358] An example of a specific prompt to be input to the generative AI model is as follows:

[1359] "Please analyze the following audio data and divide it into three parts: a summary, upcoming events, and a suggested thank you email. The audio data contains the content of a business meeting."

[1360] In this way, the system of the present invention consistently automates everything from recording voice data to text analysis, organizing information, and providing feedback, significantly reducing the workload of sales representatives.

[1361] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1362] System program processing flow

[1363] Step 1:

[1364] During business meetings, users record conversations using a smartphone or dedicated recording device. The input is "audio data" and the output is "an audio file saved on the local device." Specifically, the user launches a recording app, presses the record button to record the conversation, and the file is saved when recording is complete.

[1365] Step 2:

[1366] After the business meeting is over, the user uploads the recorded audio file to the server. The input is the "audio file saved on the local device" and the output is the "audio file uploaded to the server." Specifically, the user launches the application, selects the audio file using the upload function, and presses the send button.

[1367] Step 3:

[1368] The server receives the uploaded audio file and converts it into high-quality text. The input is the "audio file uploaded to the server" and the output is "text data stored in the database." Specifically, the server uses the Hyper CLOVA API to convert the audio data into text data and stores the text data in the database.

[1369] Step 4:

[1370] The server analyzes the converted text data using generative AI. The input is "text data stored in the database" and the output is "analyzed information (summary, scheduled events, and thank-you email suggestions)." Specifically, the server uses a generative AI model to analyze the text data and divide and organize it into summaries, scheduled events, and thank-you email suggestions.

[1371] Step 5:

[1372] The server sends the organized information to the user's device. The input is the analyzed information (summary, scheduled events, and thank-you email suggestions), and the output is the information sent to the user's device. Specifically, the server generates the summary, scheduled events, and thank-you email suggestions and sends them to the user's device.

[1373] Step 6:

[1374] The device adds the sent calendar event information to Google Calendar. The input is "calendar event information sent to the user's device" and the output is "event added to Google Calendar." Specifically, the device uses the Google Calendar API to add the fed-back calendar event information to the user's Google Calendar.

[1375] Step 7:

[1376] The terminal creates an email to the customer based on the draft thank you email, and sends it after the user has confirmed it. The input is the "draft thank you email sent to the user's terminal," and the output is the "thank you email sent to the customer." Specifically, the terminal displays the draft thank you email, and the user confirms and edits the content, then presses the send button to send the email to the customer.

[1377] In this way, the system of the present invention processes and calculates data at each processing step, thereby improving the efficiency of salespeople's work.

[1378] (Application example 1)

[1379] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1380] To improve the quality of customer service in brick-and-mortar stores, there is a need for technology that can efficiently record, analyze, and organize the content of conversations between employees and customers. However, current methods require a large portion of the work to be done manually, which is inefficient and places a heavy burden on employees. Furthermore, the content of conversations with customers cannot be used appropriately, making it difficult to utilize in business negotiations or future interactions. To solve these issues, it is necessary to introduce a system that converts speech to text and automatically classifies and organizes it.

[1381] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1382] In this invention, the server includes a means for recording audio of business negotiations with customers, a means for converting the recorded audio into high-quality text, a means for analyzing the text data and dividing and organizing it into minutes, scheduled events, and thank-you email drafts, and a means for feeding back the analyzed data, thereby making it possible to significantly improve the efficiency and quality of face-to-face interactions by employees in physical stores.

[1383] "Customer" means a visitor or user with whom you engage in face-to-face interaction to purchase a product or service.

[1384] "Business negotiations" refer to conversations and negotiations regarding products and services with customers.

[1385] "Audio recording means" refers to a device or system for recording conversations during business negotiations.

[1386] "High-quality text" refers to text data that can accurately convert speech into text and properly reproduce the content of business negotiations.

[1387] "Means for analyzing text data" refers to a device or system that uses a generative AI model to analyze text data and divide and organize it into specific categories.

[1388] A "minutes" is a document that records important matters discussed and decisions made during a business meeting.

[1389] "Scheduled event" refers to the date and time information of the next meeting or specific activity in a business negotiation.

[1390] A "thank you email draft" is a draft of an email to be sent to a customer after a business meeting to express gratitude.

[1391] "Feedback means" refers to a device or system that provides analyzed data to a user.

[1392] "Server" refers to a computer system that processes voice data and divides and organizes the analysis results.

[1393] A "terminal" is an electronic device used by a sales representative to display analyzed information.

[1394] The system of the present invention is designed to improve the efficiency and quality of customer service in brick-and-mortar stores. The system includes voice recording, speech-to-text conversion, text analysis using generative AI models, and feedback functionality.

[1395] Audio recording

[1396] A user uses an electronic device to record conversations with customers. The hardware used for this is a smartphone or other audio recording device. The audio data captured by the audio recording device is temporarily stored on a local device.

[1397] Uploading an audio file

[1398] After the negotiation is over, the user uploads the recorded audio file to the server using the application. This upload function is installed on the user's device and has the function of sending the recorded file to the server.

[1399] Speech to text

[1400] The server converts the received audio file into high-quality text. Natural language processing technology is used in this process. Specifically, the audio data is converted into text data using an API implemented on the server. An example of this API is the Google Speech Recognition API. The converted text data is stored in a database and moves on to the next analysis step.

[1401] Text analysis, segmentation, and organization

[1402] The server analyzes the converted text data using a generative AI model (e.g., OpenAI's GPT-3). This analysis divides and organizes the data into three categories: minutes, scheduled events, and draft thank-you emails. This automatically extracts important information, next action items, and date and time information.

[1403] Organized information feedback

[1404] The server sends the divided and organized information to the user's device and provides feedback to the user. The user can then check the minutes, scheduled events, and thank-you email draft on their own device and make any necessary corrections.

[1405] Adding an event to Google Calendar

[1406] The device can add the sent scheduled event information to Google Calendar, and the event will be automatically reflected on the calendar using the Google Calendar API.

[1407] Sending a thank you email

[1408] The terminal creates an email to the customer based on the draft of the thank-you email and sends it. In this process, the user checks the content of the email and makes any necessary corrections before sending it.

[1409] Specific examples

[1410] For example, if a conversation is recorded during a sales conversation with a customer saying, "Hello, can you tell me more about this product?", the system would work as follows:

[1411] 1. Use your smartphone to record conversations during business negotiations.

[1412] 2. After the business meeting is completed, the user uses the application to upload the recording file to the server.

[1413] 3. The server converts the audio file into text using the Google Speech Recognition API.

[1414] 4. The server uses a generative AI model to analyze the text and extract key information.

[1415] 5. The server divides and organizes the minutes, scheduled events, and thank you email drafts.

[1416] 6. The organized information is fed back to the user's device.

[1417] Prompt Sentence Examples

[1418] For example, the prompt used during parsing is:

[1419] "Please parse the following text and split it into meeting minutes, planned events, and a draft thank you email.\n\nHello, can you tell me more about this product?"

[1420] In this way, the present invention improves the efficiency of customer service in brick-and-mortar stores, reduces the burden on employees, and improves customer satisfaction.

[1421] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1422] Step 1:

[1423] A user records a conversation during a business meeting.

[1424] Input: Customer conversation

[1425] Output: Recorded audio file

[1426] What it does: A user uses a smartphone or other audio recording device to record a conversation during a business meeting. The audio file is saved locally on the device.

[1427] Step 2:

[1428] The user uploads the recorded audio file to the server.

[1429] Input: Audio file stored on the local device

[1430] Output: Audio file uploaded to the server

[1431] Specific operation: After the business meeting is completed, the user uploads the recorded audio file to the server using a dedicated application. The file is sent to the server using the application's upload function.

[1432] Step 3:

[1433] The server converts the audio file into high-quality text.

[1434] Input: Uploaded audio file

[1435] Output: Text data

[1436] How it works: The server receives the audio file and converts it into text using natural language processing technology such as Google Speech Recognition API. This text data is then stored in a database.

[1437] Step 4:

[1438] The server analyzes the text data using a generative AI model.

[1439] Input: Text data converted from audio

[1440] Output: Meeting minutes, scheduled events, draft thank you email

[1441] Specific operation: The server inputs the text data into a generative AI model (e.g., OpenAI's GPT-3), and divides and organizes the analysis results into three categories: minutes, scheduled events, and a draft thank-you email. An example of a prompt used during analysis is "Analyze the following text and divide it into minutes, scheduled events, and a draft thank-you email.\n\n(text data)".

[1442] Step 5:

[1443] The server feeds back the divided and organized information to the user's device.

[1444] Input: Minutes, scheduled events, and thank you email draft data

[1445] Output: Information fed back to the user's device

[1446] Specific operation: The server sends the analyzed data to the user's device, where the user can review it. The user can then edit the meeting minutes, scheduled events, and thank-you email drafts on their device as appropriate.

[1447] Step 6:

[1448] The device adds the sent scheduled event information to Google Calendar.

[1449] Input: Scheduled event information

[1450] Output: Scheduled events reflected in Google Calendar

[1451] Specific operation: The device uses the Google Calendar API to add the scheduled event information sent from the server to the Google Calendar.

[1452] Step 7:

[1453] The terminal creates and sends an email to the customer based on the original thank-you email draft.

[1454] Input: Draft thank you email

[1455] Output: Thank you email sent to customer

[1456] Specific operation: The terminal creates an email based on the draft thank-you email, and after the user checks the content and makes any necessary corrections, the email is sent to the customer.

[1457] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1458] overview

[1459] The system of the present invention not only enables salespeople to efficiently organize sales negotiations with customers and largely automates reporting tasks, but also incorporates an emotion engine to recognize and reflect the user's emotional state. The system includes voice recording, text conversion, generative AI text analysis, emotion recognition, and feedback functions.

[1460] Audio recording

[1461] Users record their voices during sales conversations with customers using their smartphones or dedicated recording devices, and the audio files are saved in the device's local storage. An emotion engine runs in parallel to analyze the user's real-time reactions and tone of voice.

[1462] Uploading an audio file

[1463] After the negotiation is over, the user uploads the recorded audio file to the server using a dedicated application. During the upload process, the emotion data analyzed by the emotion engine is also sent to the server.

[1464] Speech to text

[1465] The server converts the received audio file into high-quality text. This process uses natural language processing technology, specifically converting the audio data into text via an API. Emotion data analyzed by the emotion engine is also stored in association with the text data.

[1466] Text analysis, segmentation, and organization

[1467] The server inputs the converted text data into a generative AI for analysis. The analysis results are divided and organized into three categories: minutes, calendar events, and thank-you email drafts. Emotion data recognized by the emotion engine is also used in the analysis process.

[1468] minutes

[1469] The server extracts important points and next action items from the text and compiles them into minutes, which include the key points discussed and decisions made during the business meeting.

[1470] Calendar Events

[1471] The server extracts date and time information for upcoming business meetings or specific events, generates calendar events, and customizes reminder notifications based on the emotional data analyzed by the emotion engine.

[1472] Thank you email draft

[1473] The server generates a draft thank-you email, which includes a brief message expressing gratitude for the business interaction and a suggestion for next steps. The appropriate tone and content are automatically adjusted based on data from the emotion engine.

[1474] Organized information feedback

[1475] The server sends each piece of information that has been divided and organized to the user's device. The sales representative can then view the minutes, calendar events, and thank-you email draft on their own device and make any necessary revisions.

[1476] Adding an event to Google Calendar

[1477] The device will add the sent calendar event information to Google Calendar, and salespeople can add these events to their own calendars via API, so they won't forget about upcoming meetings or sales calls.

[1478] Sending a thank you email

[1479] The terminal displays the draft of the thank-you email to the user, prompting the user to confirm the content. After the user makes any corrections, the terminal uses the email sending function to send the thank-you email to the customer.

[1480] Specific operation example

[1481] 1. The user uses their smartphone to record conversations during business negotiations, and the emotion engine simultaneously analyzes the emotional state in real time.

[1482] 2. After the business meeting is completed, the user uploads the recording file and emotion data to the server using the application.

[1483] 3. The server converts the audio file into text using the Hyper CLOVA API and stores the emotion data in the database in parallel.

[1484] 4. The server uses generative AI to analyze the text and extract key information, incorporating emotional data into the analysis process.

[1485] 5. The server divides and organizes the data into minutes, calendar events, and thank-you email drafts, and reflects the emotional data in each.

[1486] 6. The organized information is fed back to the user's device.

[1487] 7. When the device adds the fed-back calendar event information to Google Calendar, it customizes the reminder content based on the emotion data.

[1488] 8. The terminal displays a draft of the thank-you email to the user, and after having the user confirm that the content has been adjusted based on the emotional data, the email is sent to the customer.

[1489] In this way, the system of the present invention improves the efficiency of reporting work by sales representatives and also takes into account the emotional aspects of communication with customers, enabling higher quality follow-up.

[1490] The processing flow will be explained below.

[1491] Step 1:

[1492] During a business meeting with a customer, the user starts recording audio using a smartphone or recording device. The recording is made using a dedicated application, and the emotion engine simultaneously analyzes the user's emotional state in real time. The audio and emotion data are stored in the device's local storage.

[1493] Step 2:

[1494] After the negotiation is over, the user uploads the recorded voice file and emotional data to the server using a dedicated application. By pressing the "upload" button in the application, the emotional data is sent to the server along with the voice file.

[1495] Step 3:

[1496] The server receives the uploaded audio file and first converts the audio data into high-quality text via the Hyper CLOVA API. At this time, the audio data and emotion data are linked and stored synchronously.

[1497] Step 4:

[1498] The server stores the converted text data and the linked emotion data in a database, allowing the text data and emotion information to be stored in an integrated format.

[1499] Step 5:

[1500] The server inputs the saved text data into a generative AI system, which analyzes it and divides and organizes it into three categories: minutes, calendar events, and thank-you email drafts. Emotional data is also used in the analysis process.

[1501] Step 6:

[1502] The server generates information based on text analysis and sentiment data as follows:

[1503] Minutes: Extract important points and next action items from the text, and summarize them by taking into account their importance based on emotional data.

[1504] Calendar Events: Extract time and date information for upcoming sales meetings or specific events and customize reminders based on sentiment data.

[1505] Thank you email draft: Generates a draft of a thank you email to send to a customer, automatically adjusting the appropriate tone and content based on emotional data.

[1506] Step 7:

[1507] The server then feeds back the generated minutes, calendar events, and thank-you email drafts to the user's device, including adjustments that take into account emotional information.

[1508] Step 8:

[1509] The device adds the received calendar event information to Google Calendar. Specifically, it calls the Google Calendar API to add an event that reflects the reminder notification content based on the emotion data to the calendar.

[1510] Step 9:

[1511] The device displays a draft of the thank-you email to the user. The user checks the email content and makes any necessary corrections. The device then uses its email sending function to send the thank-you email to the customer, with the content adjusted based on the emotional data.

[1512] Through the above steps, the system of the present invention improves the efficiency of sales representatives' reporting work, and also takes into account the emotional aspects of communication with customers, enabling more personalized responses.

[1513] Example 2

[1514] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1515] While traditional sales negotiation systems can record voice and convert it into text, they do not include emotional analysis or the utilization of that data, which means the quality of communication with customers is not fully considered. This can lead to poor customer service efficiency and the risk of missing important market opportunities. Furthermore, organizing and reporting sales negotiation results takes a lot of time, preventing work efficiency.

[1516] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1517] In this invention, the server includes means for recording audio of business negotiations with customers, means for converting the recorded audio into high-quality text, means for analyzing the text data and dividing and organizing it into minutes, calendar events, and thank-you email drafts, means for analyzing emotional states and reflecting the results in data, means for sending the divided and organized information to the sales representative's terminal, means for performing real-time emotional analysis during customer visits and uploading recordings including the analyzed data to the server, means for adding the transmitted calendar event information to a schedule management service, and means for displaying the thank-you email draft on the sales representative's terminal and sending it to the customer after revision. This enables efficient organization of business negotiations with customers and high-quality customer service utilizing emotional data.

[1518] - "Customer" refers to a business partner or other person with whom we have a business relationship.

[1519] "Business negotiation" refers to a place for negotiation or consultation for business purposes.

[1520] "Audio recording" refers to the act of electronically recording sound.

[1521] "Text conversion" refers to the process of converting voice data into text format.

[1522] "Text data" refers to data in the form of text converted from speech.

[1523] "Minutes" refers to a document that records what was said and what decisions were made at a meeting or business negotiation.

[1524] "Calendar event" refers to an appointment or schedule that includes date and time information.

[1525] A "thank you email draft" refers to a draft of an email to be sent to a customer to express gratitude.

[1526] "Emotional state" refers to the speaker's feelings or state of mind.

[1527] "Sentiment analysis" refers to the process of analyzing emotions from speech or text.

[1528] A "database" refers to a system that efficiently stores and manages large amounts of data.

[1529] "Schedule management service" refers to an online service that electronically manages schedules and appointments.

[1530] "Upload" refers to the act of sending data from local storage to a server.

[1531] "Generative AI" refers to a system that uses artificial intelligence to generate or analyze data.

[1532] "API" stands for Application Program Interface and refers to a set of rules that enable communication between software programs.

[1533] System Configuration

[1534] The system of the present invention is designed to help salespeople efficiently organize sales negotiations with customers and largely automate reporting tasks. It also incorporates an emotion engine to recognize the user's emotional state and reflect it in the data. The system has the following main functions:

[1535] Audio recording

[1536] Users record audio during business meetings with customers. This recording is done using a smartphone or a dedicated recording device, and the audio file is saved in the device's local storage. Specific hardware used is a standard smartphone or portable recorder. During the recording, the emotion engine analyzes the user's tone of voice and reactions in real time to collect emotional data.

[1537] Uploading an audio file

[1538] After the negotiation is over, the user uploads the recorded voice file and emotional data to the server using a dedicated application. During the upload process, the emotional data analyzed by the emotion engine is also sent to the server.

[1539] Speech to text

[1540] The server uses natural language processing technology to convert the uploaded audio file into high-quality text. Specifically, it uses the Hyper CLOVA API to convert the audio data into text. The emotion data is associated with this text data and stored in a database.

[1541] Text analysis, segmentation, and organization

[1542] The server uses generative AI to analyze the text data. The analysis results are divided and organized into three categories: minutes, calendar events, and thank-you email drafts. Emotional data is also incorporated into this analysis process, generating content based on the user's emotional state.

[1543] Information Feedback

[1544] The server sends each piece of information that has been divided and organized to the user's device. The salesperson can then review the minutes, calendar events, and draft thank-you emails on their device and make any necessary revisions. Furthermore, the device can add the sent calendar event information to a schedule management service such as Google Calendar and set reminders based on the emotion data.

[1545] Sending a thank you email

[1546] The terminal displays the draft of the thank you email to the user for confirmation. After the user makes any necessary corrections, the terminal uses the email sending function to send the thank you email to the customer.

[1547] Specific examples

[1548] For example, the prompt when analyzing voice data from a business meeting might look like this:

[1549] Example prompt: "Transcribe the audio data below into text, summarize the key points, and draft a thank you email with an appropriate tone based on the emotional data."

[1550] Audio data: (Audio file link or file name)

[1551] As can be seen from this specific operation example, the system of the present invention highly automates a series of business processes, from voice recording to text analysis, incorporating emotional data, and generating thank-you emails. This significantly reduces the workload of sales representatives and improves the quality of communication with customers.

[1552] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1553] Step 1:

[1554] Users record their business meetings using a voice recording device (smartphone or dedicated recorder). While recording, the emotion engine works in parallel, analyzing the user's tone of voice and reactions in real time.

[1555] Input: Voice data (recorded business meetings), real-time user emotional responses

[1556] Output: Recorded audio files, real-time emotion data

[1557] Specific operation: Launch the recording app and start recording as soon as the sales meeting begins. The emotion engine analyzes the tone of the voice and collects emotional data.

[1558] Step 2:

[1559] After the business negotiation is completed, the user uploads the recorded voice file and emotion data to the server using a dedicated application.

[1560] Input: Recorded audio file, emotion data

[1561] Output: Audio files and emotion data uploaded to the server

[1562] Specific operation: Tap the "Upload" button in the dedicated application to send the file and data to the server.

[1563] Step 3:

[1564] The server converts the audio file into text using the Hyper CLOVA API, and associates the emotion data with the text data and stores it in a database.

[1565] Input: Audio file uploaded to the server, emotion data

[1566] Output: Text data, associated emotion data

[1567] Specific operation: The server sends the audio file to the Hyper CLOVA API, receives it as text data, and stores the emotion data together with the text data in the database.

[1568] Step 4:

[1569] The server uses generative AI to analyze the text data and divide and organize it into three categories: minutes, calendar events, and thank-you email drafts. Emotional data is also used in this analysis.

[1570] Input: Text data, emotion data

[1571] Output: Separated and organized meeting minutes, calendar events, and a draft thank you email

[1572] How it works: The server inputs text data into the generative AI, extracts important information, and classifies it into categories. Content that reflects emotional data is also generated.

[1573] Step 5:

[1574] The server sends each piece of information (minutes, calendar events, and thank-you email draft) to the user's device.

[1575] Input: Separated and organized meeting minutes, calendar events, draft thank you email

[1576] Output: Each piece of information sent to the user's device

[1577] Specific operation: The server sends information to the user's device through a dedicated application.

[1578] Step 6:

[1579] The device will then add the sent calendar event information to a schedule management service such as Google Calendar, and the reminder content will also be customized based on the emotion data.

[1580] Input: Calendar event information, emotion data

[1581] Output: Events added to Google Calendar, customized reminders

[1582] Specific operation: Adds event information to Google Calendar via the Calendar API and sets reminders.

[1583] Step 7:

[1584] The terminal displays the draft of the thank-you email to the user, prompting the user to confirm the content. After the user makes any corrections, the terminal uses the email sending function to send the thank-you email to the customer.

[1585] Input: Draft thank you email, user modifications

[1586] Output: Thank you email sent

[1587] Specific operation: The draft email is displayed on the device, the user makes corrections and taps the "Send" button, and the email is sent to the customer.

[1588] (Application example 2)

[1589] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1590] In conventional sales activities, the tasks of recording sales negotiations, organizing the details, and reporting them were time-consuming and labor-intensive. Furthermore, it was difficult to properly grasp the emotional state of the customer during the negotiation and reflect this in subsequent follow-up, which led to a decline in the quality of communication with the customer. This made it difficult for sales representatives to carry out their work efficiently and respond to customers effectively.

[1591] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1592] In this invention, the server includes a means for recording audio of business negotiations with customers, a means for converting the recorded audio into high-quality text, a means for analyzing the text data and dividing and organizing it into minutes, calendar events, and thank-you email drafts, and a means for inputting the analyzed emotional data into a generative AI model and reflecting it in the information organization process. This allows sales representatives to automate the detailed recording and organization of business negotiations and enable effective follow-up that takes into account the emotional state of the customer.

[1593] The "business negotiation voice recording means" is a hardware or software function for recording audio data during a business negotiation.

[1594] The "text conversion means" is a function that converts recorded voice data into high-quality text data using natural language processing technology.

[1595] "Text analysis means" is a function that uses generative AI to analyze text data and divide and organize it into minutes, calendar events, and thank-you email drafts.

[1596] "Information transmission means" is a communication function for transmitting divided and organized information to the sales representative's terminal.

[1597] The "emotion analysis means" is a function that analyzes the tone of voice and language used by customers and sales representatives during negotiations to grasp their emotional state in real time.

[1598] An "emotion engine" is a software module used as an emotion analysis tool to generate emotion data in real time.

[1599] A "generative AI model" is an artificial intelligence model that performs analysis and information generation based on input text data and emotional data.

[1600] A "prompt" is an instruction statement that allows the generative AI model to properly analyze the text data it inputs.

[1601] The system of the present invention aims to enable sales representatives to efficiently organize business negotiations with customers and to largely automate reporting tasks. This system can be implemented by the following procedure.

[1602] The system configuration requires a smartphone used by the user, an emotion analysis engine, a generative AI model, natural language processing technology, and a server that integrates these. Specifically, the smartphone is equipped with voice recording and data communication functions and connects to the server application.

[1603] The general flow of the system is as follows: First, the salesperson (user) uses their smartphone to record the voice of the sales negotiation. At the same time, the emotion analysis engine analyzes the tone of the user's and customer's voice and the words they use in real time to understand their emotional state.

[1604] Once the negotiation is over, the user uploads the recorded voice data and real-time analyzed emotion data to the server. The server receives the voice data and converts it into high-quality text using natural language processing technology (such as the Google Speech-to-Text API). The emotion data is also stored in a database.

[1605] The server then uses a generative AI model (such as OpenAI's ChatGPT) to analyze the converted text data. The analysis is performed using pre-defined prompts, which are then divided and organized into three main categories: meeting minutes, calendar events, and thank-you email drafts. Specific examples of prompts are as follows:

[1606] "Please parse the following text and classify it into the following categories: meeting minutes, next appointment, thank you email content. Emotion data: {emotionData}

[1607] text:{

[1608] ...negotiation text...

[1609] }"

[1610] The information for each category, divided and organized, along with sentiment analysis data, is fed back to the sales representative's smartphone, allowing them to check meeting minutes, calendar events, and thank-you email drafts on their own screens and make any necessary adjustments.

[1611] Calendar events are automatically added to Google Calendar via API. Reminders are customized based on the customer's emotional data. Thank-you emails are then sent to the customer after the sales representative has reviewed and revised the content.

[1612] The entire system processes and calculates data using the following technical elements:

[1613] Smartphone: Responsible for recording and data uploading.

[1614] Emotion analysis engine: Analyzes emotional state from voice in real time.

[1615] Natural language processing technology: Converts voice data into high-quality text data.

[1616] Generative AI model: Analyzes text data and organizes information into categories.

[1617] Google Calendar API: Used to automatically add calendar events and adjust reminders.

[1618] This allows sales representatives to efficiently record and organize sales negotiation details while also enabling them to conduct high-quality follow-up that takes into account the customer's emotional state.

[1619] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1620] Step 1:

[1621] A user uses a smartphone to record audio during a business meeting.

[1622] Input: Voice during negotiation

[1623] Output: Recorded audio file

[1624] How it works: The user launches the recording app on their smartphone and presses the record button when the sales meeting begins. As the recording progresses, the emotion analysis engine analyzes the tone and vocabulary of the user and customer in real time and records their emotional state.

[1625] Step 2:

[1626] After the business negotiation is completed, the user uploads the recorded voice file and real-time emotion data to the server.

[1627] Input: Recorded audio files, real-time emotion data

[1628] Output: Audio files and emotion data stored on the server

[1629] Specific operation: After the business negotiation is completed, the user presses the upload button in the app. The audio file and emotion data are sent to the server and stored in the database.

[1630] Step 3:

[1631] The server receives the audio file and converts it into high-quality text using natural language processing techniques.

[1632] Input: Recorded audio file

[1633] Output: High-quality text data

[1634] Specific operation: The server analyzes the received audio file and converts it into text data using natural language processing technology (e.g., Google Speech-to-Text API). At the same time, emotion data is also stored in a database.

[1635] Step 4:

[1636] The server uses a generative AI model to analyze the converted text data and divide and organize it into three main categories: meeting minutes, calendar events, and thank-you email drafts.

[1637] Input: High-quality text data, emotion data

[1638] Output: Minutes data, calendar event data, thank you email draft data

[1639] Specific operation: The server uses a generative AI model (e.g., OpenAI's ChatGPT) to analyze and classify data using the following prompt:

[1640] "Please parse the following text and classify it into the following categories: meeting minutes, next appointment, thank you email content. Emotion data: {emotionData}

[1641] text:{

[1642] ...negotiation text...

[1643] }"

[1644] The analysis results are divided into minutes, calendar events, and a draft thank you email.

[1645] Step 5:

[1646] The server then sends the information for each divided and organized category back to the sales representative's smartphone.

[1647] Input: Minutes data, calendar event data, thank you email draft data

[1648] Output: Data displayed on the user's smartphone

[1649] Specific operation: The organized data is sent from the server to the user's smartphone, and the sales representative checks and edits the contents of meeting minutes, calendar events, and thank-you emails.

[1650] Step 6:

[1651] Your device will automatically add calendar events to your Google Calendar and customize reminders based on your emotional data.

[1652] Input: Calendar event data

[1653] Output: Events added to Google Calendar with customized reminders

[1654] Specific operation: An event is added from the user's smartphone via the Google Calendar API, and the reminder content is adjusted based on emotional data.

[1655] Step 7:

[1656] The terminal displays a draft of the thank-you email to the sales representative, who then edits the content and sends it to the customer.

[1657] Input: Draft data for thank you email

[1658] Output: Thank you email sent

[1659] Specific actions: After the sales representative checks and edits the content of the thank-you email, they send it to the customer via the email app on their smartphone.

[1660] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1661] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1662] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1663] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1664] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1665] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1666] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1667] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1668] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1669] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1670] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1671] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1672] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1673] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1674] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1675] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1676] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1677] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1678] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1679] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1680] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1681] The following is further disclosed regarding the above embodiment.

[1682] (Claim 1)

[1683] A means of recording audio of business meetings with customers;

[1684] A means of converting recorded speech into high-quality text;

[1685] A method to analyze text data and divide and organize it into minutes, calendar events, and thank-you email drafts.

[1686] A means for transmitting the divided and organized information to the terminal of the sales representative;

[1687] A system including:

[1688] (Claim 2)

[1689] 2. The system according to claim 1, further comprising means for recording voice in real time during a customer visit and uploading the recording to a server.

[1690] (Claim 3)

[1691] 2. The system according to claim 1, wherein natural language processing technology is used as a high-quality text conversion means.

[1692] "Example 1"

[1693] (Claim 1)

[1694] A way for salespeople to record their conversations with customers;

[1695] a means for converting recorded speech into text;

[1696] A means to analyze the converted text data and divide and organize it into summaries, planned events, and thank you email proposals;

[1697] A means for transmitting the organized information to a terminal of a sales representative;

[1698] A system including:

[1699] (Claim 2)

[1700] 2. The system according to claim 1, further comprising means for recording voice in real time during a customer visit and uploading the recording to a data server.

[1701] (Claim 3)

[1702] 10. The system of claim 1, wherein the speech-to-text conversion means uses advanced language processing techniques.

[1703] "Application Example 1"

[1704] (Claim 1)

[1705] A means of recording audio of business meetings with customers;

[1706] A means of converting recorded speech into high-quality text;

[1707] A method to analyze text data and divide and organize it into minutes, scheduled events, and thank-you email drafts.

[1708] A means for transmitting the divided and organized information to the terminal of the sales representative;

[1709] A means for feeding back the analyzed data;

[1710] A system including:

[1711] (Claim 2)

[1712] 2. The system according to claim 1, further comprising means for recording voice in real time during a customer visit and uploading the recording to a server.

[1713] (Claim 3)

[1714] 2. The system according to claim 1, wherein natural language processing technology is used as a high-quality text conversion means.

[1715] "Example 2: Combining Emotion Engines"

[1716] (Claim 1)

[1717] A means of recording audio of business meetings with customers;

[1718] A means of converting recorded speech into high-quality text;

[1719] A method to analyze text data and divide and organize it into minutes, calendar events, and thank-you email drafts.

[1720] A means of analyzing emotional states and reflecting them in data,

[1721] A means for transmitting the divided and organized information to the terminal of the sales representative;

[1722] A means of conducting real-time sentiment analysis during customer visits and uploading recordings containing that data to a server;

[1723] means for adding the submitted calendar event information to a schedule management service;

[1724] A method to display the draft of the thank you email on the sales representative's device and send it to the customer after editing,

[1725] A system including:

[1726] (Claim 2)

[1727] 2. The system according to claim 1, further comprising means for performing real-time emotion analysis when recording voice of a business meeting with a customer.

[1728] (Claim 3)

[1729] The system of claim 1, characterized in that it uses natural language processing technology as a means for high-quality text conversion and also includes means for associating and storing sentiment analysis data.

[1730] "Application example 2 when combining emotion engines"

[1731] (Claim 1)

[1732] A means of recording audio of business meetings with customers;

[1733] A means of converting recorded speech into high-quality text;

[1734] A method to analyze text data and divide and organize it into minutes, calendar events, and thank-you email drafts.

[1735] A means for transmitting the divided and organized information to the terminal of the sales representative;

[1736] A means of analyzing emotional states in real time during business negotiations with customers,

[1737] A means to input the analyzed emotion data into a generative AI model and reflect it in the information organization process;

[1738] A system including:

[1739] (Claim 2)

[1740] 2. The system according to claim 1, further comprising means for recording voice in real time during a customer visit and uploading the recording to a server.

[1741] (Claim 3)

[1742] 2. The system according to claim 1, wherein natural language processing technology is used as a high-quality text conversion means.

[1743] (Claim 4)

[1744] 2. The system of claim 1, wherein the means for analyzing the emotional state is an emotion recognition engine.

[1745] (Claim 5)

[1746] 10. The system of claim 1, wherein when using a generative AI model, prompt sentences are used to analyze and organize text data. [Explanation of symbols]

[1747] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of recording audio of business meetings with customers; A means of converting recorded speech into high-quality text; A method to analyze text data and divide and organize it into minutes, calendar events, and thank-you email drafts. A means for transmitting the divided and organized information to the terminal of the sales representative; A system including:

2. 2. The system according to claim 1, further comprising means for recording voice in real time during a customer visit and uploading the recording to a server.

3. 2. The system according to claim 1, wherein a natural language processing technique is used as a high-quality text conversion means.

Images