System

Abstract

A system is provided.SOLUTION: The system includes means for receiving communication with the customer as voice data, means for converting the received voice data into text data using a voice recognition technology, means for analyzing the converted text data using a natural language processing technology and generating a summary of a matter, means for automatically counting the number of matters related to a specific keyword or phrase and generating aggregation data, and means for automatically creating a material for a report based on the generated aggregation data.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In call center operations, the task of accurately and quickly counting the number of logs and cases generated from customer interactions and reporting them to managers is extremely labor-intensive. Furthermore, conventional methods rely on manual log entry, which is prone to human error and inaccurate counting. This increases the burden on field staff and reduces work efficiency. It is also difficult to accurately grasp the number of cases related to specific topics or irregular cases. To solve these problems, the present invention provides a system that automatically converts customer interactions into text, generates case summaries, automatically counts the number of cases, and creates reporting materials. [Means for solving the problem]

[0005] The present invention solves the problems by the following means. Specifically, the system includes a means for receiving interactions with customers as voice data, a means for converting the received voice data into text data using voice recognition technology, a means for analyzing the converted text data using natural language processing technology and generating a summary of the cases, a means for automatically counting the number of cases related to specific keywords or phrases and generating aggregated data, and a means for automatically creating reporting materials based on the generated aggregated data. This system reduces the labor required for staff to manually enter logs and count cases, enabling accurate and prompt reports to managers. In particular, the means for generating aggregated data as graphs or tables using a data visualization tool and the means for creating reporting materials using an automatic report generation tool enable the creation of visually easy-to-understand materials in a short amount of time.

[0006] "Interaction with customers" refers to the communication process, such as phone calls and chats, between customers and operators at a call center.

[0007] "Audio data" refers to data files that record interactions with customers in audio format.

[0008] "Speech recognition technology" is a technology for analyzing voice data and converting it into text data.

[0009] "Text data" is character information converted using voice recognition technology.

[0010] "Natural language processing technology" is a technology for analyzing text data and understanding and generating human language.

[0011] A "case summary" is a concise summary of the main information and key points of a case, extracted using natural language processing technology.

[0012] "Specific keywords and phrases" are words or phrases related to a particular topic or issue that the administrator is interested in.

[0013] "Count" refers to counting the number of jobs related to a particular keyword or phrase.

[0014] "Aggregated data" refers to data that has been organized and statistically compiled based on the number of counted cases.

[0015] "Reporting materials" are materials for reports and presentations created based on the generated aggregated data.

[0016] The "System" refers to a collection of devices and programs that receive voice data from interactions with customers and automatically perform voice recognition, natural language processing, counting, aggregation, and preparation of reporting materials.

[0017] A "data visualization tool" is software or technology for visually representing aggregated data.

[0018] "Graphs and tables" are visual formats for visually representing aggregated data.

[0019] An "automated report generation tool" is software or technology for automatically creating report materials.

[0020] "Means for receiving" refers to a method or device for inputting voice data of interactions with customers into the system.

[0021] "Conversion means" refers to the method or technology used to convert audio data into text data.

[0022] "Means of analysis" refers to methods and techniques for analyzing text data and extracting important information.

[0023] "Generating means" refers to the methods and techniques used to create a case summary based on key information.

[0024] A "counting method" is a method or technique for counting the number of listings related to a particular keyword or phrase.

[0025] "Generation means" refers to the methods and techniques used to aggregate the counted data and create reporting materials. [Brief explanation of the drawings]

[0026] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0027] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0028] First, the terms used in the following description will be explained.

[0029] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0030] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0031] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0032] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0033] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0034] [First embodiment]

[0035] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0036] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0037] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0038] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0039] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0040] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0041] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0042] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0043] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0044] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0045] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0046] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0047] This invention relates to a system for call center operations that converts customer interactions into text, generates case summaries, automatically counts the number of cases, and creates report materials. This system is composed of a combination of speech recognition technology, natural language processing technology, data visualization technology, and automatic report generation technology.

[0048] Program processing natural language explanation

[0049] Step 1: Transcribe your interactions with customers

[0050] The server receives the call voice data from the call center. The call voice is received from the VoIP system and can be processed in real time or in batches.

[0051] The server converts the received voice data into text using speech recognition technology, with the speech recognition engine utilizing a widely available cloud-based API.

[0052] Step 2: Generate a case summary

[0053] The server analyzes the converted text data using a natural language processing toolkit, which performs preprocessing of the text (tokenization, part-of-speech tagging, stemming, and lemmatization).

[0054] The server applies NLP models to extract key phrases and information, thereby organizing the case's main topics and issues and generating a case summary in a standardized format.

[0055] Step 3: Counting and data collection of cases

[0056] The terminals provide an interface for inputting specific keywords and phrases, in response to instructions from the administrator, who is interested in various topics that arise in the call center.

[0057] The server uses the entered keywords to search the text database, identify and count relevant cases. The use of a full-text search engine enables high-speed search processing.

[0058] The server aggregates the search results and compiles statistics on the number of cases, generating daily, weekly, and monthly aggregate data.

[0059] Step 4: Prepare your report

[0060] The server uses a data visualization tool to generate graphs and tables using the aggregated data, allowing the data to be presented in a visually understandable format.

[0061] The server uses an automated report generation tool to create reporting materials, including generated graphs and tables, which can be output as presentations, PDFs, or other formats.

[0062] Step 5: Final review and delivery

[0063] The user (person in charge) checks the created report materials on the terminal. The interface is designed to make it easy to check the contents of the materials.

[0064] The device sends the confirmed report materials to the administrator via the internal email system or saved on a shared drive.

[0065] Specific examples

[0066] Example 1: Counting returns

[0067] 1. The server receives the call audio data related to "returns" and converts it into text.

[0068] 2. The server analyzes the converted text data and generates a summary of the case, including important information such as the reason for return, the date of the request, and the status of the response.

[0069] 3. The administrator enters keywords related to "returns" on the terminal and issues an instruction to check the number of cases.

[0070] 4. The server searches for cases related to "returns" and counts the number of cases.

[0071] 5. The server aggregates the data based on the counting results and creates visual graphs and tables.

[0072] 6. The user checks the generated report materials on the terminal and sends them to the administrator.

[0073] Example 2: Counting cases related to delivery delays

[0074] 1. The server receives the call audio data related to "Delivery Delay" and converts it into text.

[0075] 2. The server analyzes the text data and generates a project summary.

[0076] 3. The administrator enters keywords related to "delivery delay" on the terminal and issues an instruction to check the number of cases.

[0077] 4. The server searches for the number of cases related to "Delivery Delay" and aggregates the results.

[0078] 5. The server creates graphs and tables based on the aggregated data and generates reporting materials.

[0079] 6. The user checks the materials and sends them to the administrator.

[0080] The introduction of this system will significantly improve the efficiency of call center operations, reduce the burden on staff, and enable the provision of accurate data.

[0081] The processing flow will be explained below.

[0082] Step 1:

[0083] The server acquires the call center's voice data, which is received in batches in real time from the VoIP system.

[0084] Step 2:

[0085] The server converts the acquired voice data into text data using speech recognition technology. The speech recognition engine uses a cloud-based API, and the converted text data is temporarily stored.

[0086] Step 3:

[0087] The server analyzes the stored text data using a natural language processing toolkit (e.g., spaCy or NLTK), which includes tokenization, part-of-speech tagging, stemming, and lemmatization.

[0088] Step 4:

[0089] The server applies an NLP model (e.g., BERT or GPT-3®) to extract key phrases and information from the text data. This generates a summary of the case, including information such as the problem, request details, and response status.

[0090] Step 5:

[0091] The terminal receives instructions from an administrator and provides an interface for inputting specific keywords or phrases, such as "returned goods" or "delayed delivery."

[0092] Step 6:

[0093] The server searches a text database using keywords entered from the terminal to identify relevant cases. High-speed search processing is performed using a full-text search engine (e.g., ElasticSearch (registered trademark)).

[0094] Step 7:

[0095] The server aggregates the search results and counts the number of relevant cases. The counted data is compiled and organized as daily, weekly, and monthly statistical information.

[0096] Step 8:

[0097] The server uses a data visualization tool (e.g., Matplotlib or Tableau) to generate graphs and tables that are easy to understand visually based on the aggregated data.

[0098] Step 9:

[0099] The server uses automated report generation tools (e.g., LaTeX or Pandas DataFrames) to create reports, including generated graphs and tables.

[0100] Step 10:

[0101] The user (person in charge) checks the report materials on the terminal, checking the content and format of the materials through the interface for any errors.

[0102] Step 11:

[0103] The device sends the verified report materials to the administrator, who then saves them in the internal email system or on a shared drive and notifies the appropriate administrator.

[0104] Example 1

[0105] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0106] Call center operations require efficient management of a large volume of customer interactions, as well as the ability to quickly and accurately summarize cases and generate statistical data. However, conventional systems did not integrate voice recognition, natural language processing, data visualization, or automatic report generation, which placed a heavy burden on staff and posed challenges to the accuracy and efficiency of processing.

[0107] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0108] In this invention, the server includes means for receiving interactions with customers as voice data, means for converting the received voice data into text data using voice recognition technology, means for analyzing the converted text data using natural language processing technology and generating case summaries, means for automatically counting the number of cases related to specific keywords or phrases and generating aggregated data, means for automatically creating report materials based on the generated aggregated data, means for statistically compiling the aggregated data in daily, weekly, and monthly formats, means for supporting real-time and batch processing of voice data, and means for performing tokenization, part-of-speech tagging, stemming, and lemmatization during the analysis process. This makes it possible to improve the efficiency of call center operations, reduce the burden on staff, and provide accurate data.

[0109] "Voice Data" means digital audio files or live audio streams recorded during customer interactions in a call center.

[0110] "Speech recognition technology" is a technology for analyzing voice data and converting it into text data.

[0111] "Text data" refers to textual information converted using voice recognition technology.

[0112] "Natural language processing technology" is a technology for analyzing text data, understanding its content, and extracting specific information.

[0113] The "Case Summary" is the result of organizing key information and important phrases extracted using natural language processing technology.

[0114] "Keywords" are key words or phrases used to identify a particular subject.

[0115] "Aggregated data" refers to statistically compiled data on the number of jobs related to a particular keyword.

[0116] A "data visualization tool" is software that displays aggregated data in a visually easy-to-understand format, such as a graph or table.

[0117] "Reporting materials" are documents that are created based on aggregated data and contain project summaries and statistical data.

[0118] "Automated report generation tool" means software or a tool for automatically generating reports based on data.

[0119] The "daily, weekly, and monthly format" is a format in which data is classified and summarized by day, week, or month.

[0120] "Real-time processing" refers to the process of processing audio data and producing results in real time.

[0121] "Batch processing" is a process of processing a large amount of audio data at once at regular time intervals.

[0122] "Tokenization" is the process of dividing text data into words and phrases.

[0123] "Part-of-speech tagging" is the process of assigning part-of-speech labels to segmented words and phrases.

[0124] "Stemming" is the process of extracting the roots of words.

[0125] "Lemmatization" is the process of extracting the basic forms of words.

[0126] This invention is a system for call center operations that receives customer interactions as voice data, converts it into text, generates case summaries and aggregated data, and automatically creates report materials. This system is realized by combining the following hardware and software.

[0127] Hardware and software used

[0128] 1. The server is a device that receives call voice data through the call center's VoIP system. The server is equipped with a high-speed speech recognition engine (e.g., Google® Cloud Speech-to-Text API or Amazon Transcribe).

[0129] 2. The server further has the function of converting the received voice data into text data in real time or batch processing using voice recognition technology.

[0130] 3. The server uses natural language processing techniques (e.g., spaCy, NLTK, BERT model) to extract key phrases and information from the converted text data, thereby organizing the main topics and issues of the case and generating a summary of the case in a standardized format.

[0131] 4. A terminal is a device that provides an interface for administrators and users to search for specific keywords or phrases. This interface may be in the form of a web form, where administrators input instructions about topics of interest.

[0132] 5. The server uses a full-text search engine such as Elasticsearch to search the text database based on the entered keywords, identifies relevant cases, and counts the number of cases. The search results are then statistically compiled by day, week, and month.

[0133] 6. The server uses a data visualization tool (e.g., Tableau, D3.js) to generate graphs and tables from the aggregated data, allowing the data to be presented in a visually understandable format.

[0134] 7. The server uses automated report generation tools (e.g., LaTeX, Google Docs API) to create a report containing the generated graphs and tables, which can be output as a presentation, PDF, or other format.

[0135] 8. The user checks the created report materials, confirms that the contents are accurate, and then sends them to the administrator. The interface on the terminal is designed to make it easy to check the contents of the materials.

[0136] Specific examples

[0137] Counting returns cases

[0138] 1. The server receives the call audio data related to "returns" from the VoIP system and converts it to text using the Google Cloud Speech-to-Text API. When converting the audio data to text, background noise is removed and the speaker is identified.

[0139] 2. The server analyzes the converted text data using spaCy, extracts important information such as the reason for return, request date, and response status, and generates a case summary.

[0140] 3. The administrator enters keywords related to "returns" on the terminal and issues an instruction to check the number of cases.

[0141] 4. The server uses Elasticsearch to search for cases related to "returns," lists the results, and performs further aggregation. The calculated data is saved in a database.

[0142] 5. The server uses Tableau to visualize the data and generate visual graphs and tables designed to show at a glance the increase or decrease in the number of returns.

[0143] 6. The user checks the generated report materials on the terminal and verifies that there are no problems with the contents. After checking, the materials are sent to the administrator.

[0144] Counting cases related to delivery delays

[0145] 1. The server receives the call audio data related to the "Delivery Delay" from the VoIP system and converts it to text using Amazon Transcribe. During the conversion process, the audio file format and sampling rate are adjusted.

[0146] 2. The server uses the BERT model to analyze the text data and extract important information such as the cause of the delay, the date and time of occurrence, and the response status.

[0147] 3. The administrator enters keywords related to "delivery delay" on the terminal and issues an instruction to check the number of cases.

[0148] 4. The server uses Elasticsearch to search for cases related to "Delivery Delays" and aggregates the results. The search results are then stored in an SQL database.

[0149] 5. The server uses D3.js to create graphs and tables based on the aggregated results and generate report materials. These materials are often output in PDF format and shared with all project stakeholders.

[0150] 6. The user checks the document and sends it to the administrator. The check is done on the screen, and the interface makes it easy to review and correct the document.

[0151] Examples of prompt statements

[0152] "Please transcribe the conversations about returns over the past week, tally the number of cases, and create a graph to generate a report."

[0153] "Count the number of inquiries about delivery delays and create a weekly report with monthly tallies and summaries."

[0154] The introduction of this system will significantly improve the efficiency of call center operations, ensuring the accuracy of information and reducing the burden on staff.

[0155] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0156] Step 1:

[0157] The server receives the call voice data through the call center's VoIP system. At this time, the server acquires the voice data in a format suitable for receiving the voice data in real time or batch processing. The input is the voice data from the VoIP system, and the output is the raw voice data stored in the server.

[0158] Step 2:

[0159] The server converts the received voice data into text data using the Google Cloud Speech-to-Text API. This process involves cleaning the voice data (noise removal and speaker identification). The input is voice data, and the output is text data.

[0160] Step 3:

[0161] The server analyzes the converted text data using a natural language processing toolkit (e.g., spaCy, NLTK). This analysis process involves tokenization, part-of-speech tagging, stemming, and lemmatization. The input is text data, and the output is preprocessed text data.

[0162] Step 4:

[0163] The server extracts important phrases and information from the preprocessed text data using the BERT model or GPT-3, and generates a summary of the case. The input is the preprocessed text data, and the output is the summary data of the case.

[0164] Step 5:

[0165] The terminal provides an interface for the administrator to input specific keywords or phrases. The administrator inputs topics of interest, such as "returns" or "delayed delivery," which in turn sends a search command to the server. The input is the administrator's keywords, and the output is a search request to the server.

[0166] Step 6:

[0167] The server uses Elasticsearch to search the text database based on keywords entered by the administrator. The search process quickly matches the text data with the target keywords to identify and count related jobs. The input is the keywords and the text database, and the output is a list of related jobs and their counts.

[0168] Step 7:

[0169] The server statistically compiles the list of cases and the counts it receives by day, week, and month. The results are saved in a database. The input is the list of cases and the counts, and the output is the statistically compiled aggregated data.

[0170] Step 8:

[0171] The server uses a data visualization tool (e.g., Tableau, D3.js) to generate graphs and tables based on the aggregated data. This allows the data to be displayed in a visually understandable format. The input is statistically compiled aggregated data, and the output is visual data such as graphs and tables.

[0172] Step 9:

[0173] The server uses automated report generation tools (e.g., LaTeX, Google Docs API) to create a report containing the generated visual data. The report is output in presentation or PDF format. The input is visual data such as graphs and tables, and the output is the completed report.

[0174] Step 10:

[0175] The user checks the report materials created on the terminal. The interface is designed to make it easy to check the contents of the materials. Once the materials have been checked, they are sent to the administrator. The input is the completed report materials, and the output is the checked materials and the results of their transmission.

[0176] (Application example 1)

[0177] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0178] In conventional call center operations, recording call content, summarizing cases, counting the number of cases, and creating reports are often done manually, resulting in inefficiency and the consumption of large amounts of human resources. Furthermore, manual data processing has a high error rate, making it difficult to quickly obtain accurate statistical data and reports. This makes it difficult for security services to respond quickly, leading to a decline in customer satisfaction. The present invention aims to solve these problems and improve the efficiency of call center operations and the accuracy of data.

[0179] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0180] In this invention, the server includes: means for receiving interactions with customers as voice data; means for converting the received voice data into text data using voice recognition technology; means for analyzing the converted text data using natural language processing technology and generating a summary of the cases; means for automatically counting the number of cases related to specific keywords or phrases and generating aggregated data; means for automatically creating report materials based on the generated aggregated data and distributing the generated report; means including technology for converting voice data into text in real time; and means for counting the number of summarized cases and visualizing the data based on that. This makes it possible to streamline call center operations, reduce the burden on staff, provide accurate data quickly, and improve the response speed and quality of security services.

[0181] "Customer" refers to general consumers, corporations, etc. who use the service.

[0182] "Interaction" refers to calls, conversations, and inquiries between customers and call center representatives.

[0183] "Voice data" refers to audio files and digital data that record the contents of conversations with customers.

[0184] "Means for receiving" refers to a device or a method for implementing a device for loading call center calls into a server.

[0185] "Speech recognition technology" refers to technology for converting voice data into text data.

[0186] "Text data" refers to data in the form of a string of characters converted using voice recognition technology.

[0187] "Natural language processing technology" refers to technology for analyzing text data and understanding its meaning and context.

[0188] "Means for generating a case summary" refers to a device or method that uses natural language processing technology to extract important information from text data and summarize it concisely.

[0189] "Specific keywords and phrases" refer to important words and expressions related to the subject or topic of your inquiry.

[0190] "Means for automatically counting the number of cases" refers to a method for automatically calculating within the system the number of cases that match a specific keyword or phrase.

[0191] "Aggregated data" refers to data that systematically compiles the number of counted cases and related information.

[0192] "Means for automatically creating report materials" refers to a device or method for automatically generating reports or presentation materials within a system based on aggregated data.

[0193] "Means for distributing reports" refers to devices and methods for transmitting generated report materials to interested parties.

[0194] "Technology for converting voice data into text in real time" refers to technology for converting voice data into text data in real time during a call.

[0195] "Means for visualizing data" refers to devices or methods that display aggregated data in a visually easy-to-understand format, such as graphs or tables.

[0196] The system required to realize this invention includes the following hardware and software: The main elements are speech recognition, natural language processing, data visualization, and automatic report generation technologies.

[0197] The server receives customer interactions as voice data. In this invention, calls at the call center are realized using a VoIP system. The voice data is sent to the server in real time and converted into text data using the Google Cloud Speech-to-Text API. The converted text data is analyzed on the server using the Python nltk (Natural Language Toolkit) library.

[0198] This is where analysis is performed using natural language processing techniques. First, the text data undergoes preprocessing, including tokenization, part-of-speech tagging, stemming, and lemmatization. Next, NLP models are applied to extract key phrases and information and generate a case summary, resulting in a summary that summarizes the key points of the call.

[0199] The server uses Elasticsearch (a full-text search engine) to count the number of jobs related to specific keywords or phrases. Administrators specify relevant keywords or phrases on the terminal and perform searches based on them. The search results are aggregated and compiled as statistical data. This statistical data is visualized as aggregate data by day, week, or month.

[0200] To visualize the aggregated data, the server generates graphs and tables using the Tableau (data visualization tool) API, which presents the data in a format that is easy to understand visually. Report materials containing the generated graphs and tables are then created using an automated report generation tool. The generated reports are then sent to the administrator via the internal email system API.

[0201] Examples of concrete examples and prompts

[0202] Example 1: Generating a report on phishing attacks

[0203] 1. Speech Recognition:

[0204] The server uses the Google Cloud Speech-to-Text API to convert the call audio into text data.

[0205] Example prompt: "I want to know about phishing attacks."

[0206] 2. Case summary generation:

[0207] The server analyzes the converted text data and generates a summary of the case using natural language processing technology.

[0208] Example prompt: "Summarize the details of the phishing attack contained in this text."

[0209] 3. Counting the number of cases:

[0210] The administrator enters keywords related to "phishing attacks" into the terminal, and the server uses Elasticsearch to count the number of cases.

[0211] Example prompt: "Count the number of incidents related to phishing attacks."

[0212] 4. Report material generation:

[0213] The server uses the Tableau API to visualize the data and create reporting materials.

[0214] Example prompt: "Graph the number of phishing attacks by month."

[0215] 5. Submit report:

[0216] The report is sent to the administrator via the internal email system.

[0217] Example prompt: "Please email this report to your administrator."

[0218] In this way, the present invention makes it possible to improve the efficiency of call center operations, reduce the burden on staff, and improve the response speed and quality of security services.

[0219] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0220] Step 1:

[0221] The server receives the call voice data from the call center. The input is real-time voice data from the VoIP system. The server sends this data to the Google Cloud Speech-to-Text API, which converts the voice data into text data. This converted text data is the output.

[0222] Step 2:

[0223] The server receives the converted text data and analyzes it using natural language processing techniques. The input is the text data obtained in step 1. Specifically, it uses the Python nltk library to perform preprocessing such as tokenization, part-of-speech tagging, stemming, and lemmatization. It then applies an NLP model to extract key phrases and information and generate a summary of the case. This summarized text is the output.

[0224] Step 3:

[0225] The administrator enters specific keywords or phrases into the terminal. The input is the keyword or phrase specified by the administrator and is used as a search condition. The server uses Elasticsearch to search the text database based on this keyword or phrase. The number of cases that match the entered conditions is counted and this becomes the output.

[0226] Step 4:

[0227] The server aggregates the search results and compiles them as statistical data. The input is the number of cases counted in step 3. The statistical data is compiled in an aggregated format by day, week, or month. This aggregated data is the output.

[0228] Step 5:

[0229] The server performs data visualization based on the aggregated data. The input is the aggregated data generated in step 4. The server uses the Tableau API to visualize the aggregated data as graphs and tables. This visualized data is the output.

[0230] Step 6:

[0231] The server automatically creates a report containing the visualized data. The input is the visualization data generated in step 5. An automatic report generation tool is used to generate a report. This report is the output.

[0232] Step 7:

[0233] The user (person in charge) checks the report generated on the terminal. The input is the report generated by the server. The user checks the contents and makes corrections if necessary. The checked and corrected report is the output.

[0234] Step 8:

[0235] The terminal sends a report to the administrator after confirmation is complete. The input is the report confirmed in step 7. The report is sent to the administrator via the internal mail system API or shared drive. This sent report becomes the final output.

[0236] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0237] This invention relates to a system for call center operations that converts customer interactions into text, generates case summaries, automatically counts the number of cases, and creates report materials while taking into account customer emotional information using an emotion engine. This system is composed of a combination of speech recognition technology, natural language processing technology, emotion recognition technology, data visualization technology, and automatic report generation technology.

[0238] Program processing natural language explanation

[0239] Step 1: Transcribe your interactions with customers

[0240] The server acquires the call voice data of the call center. The call voice is received in batches in real time from the VoIP system.

[0241] The server uses voice recognition technology to convert the received voice data into text data. The voice recognition engine utilizes a cloud-based API.

[0242] Step 2: Generate a case summary

[0243] The server analyzes the converted text data using a natural language processing toolkit, which includes tokenization, part-of-speech tagging, stemming, and lemmatization.

[0244] The server applies NLP models to extract key phrases and information from the text data, generating a summary of the case, including information such as the issue, request details, and response status.

[0245] Step 3: Recognizing and integrating emotional information

[0246] The server uses an emotion engine to recognize the customer's emotions during the call in real time, for example by analyzing the intonation, speed, and tone of the voice to identify the emotion.

[0247] The server adds the recognized emotional information to the case summary, which makes the importance and urgency of the case more clear.

[0248] Step 4: Counting and data collection of cases

[0249] The terminal receives instructions from an administrator and provides an interface for inputting specific keywords or phrases, such as "returned goods" or "delayed delivery."

[0250] The server uses the entered keywords to search a text database and identify relevant cases. High-speed search processing is performed using a full-text search engine.

[0251] The server aggregates the search results and counts the number of relevant cases. The counted data is compiled and organized as daily, weekly, and monthly statistical information.

[0252] Step 5: Integrate and analyze sentiment data

[0253] The server integrates the recognized emotion data into aggregate data and analyzes emotion trends, for example, analyzing the frequency of negative emotions when handling complaints.

[0254] Step 6: Prepare your report

[0255] The server uses a data visualization tool to generate graphs and tables that are easy to understand visually based on the aggregated data.

[0256] The server uses an automated report generation tool to create a report containing generated graphs and tables, including the emotion data.

[0257] Step 7: Final review and delivery

[0258] The user (person in charge) checks the report materials on the terminal, checking the content and format of the materials through the interface for any errors.

[0259] The device sends the verified report materials to the administrator, who then saves them in the internal email system or on a shared drive and notifies the appropriate administrator.

[0260] Specific examples

[0261] Example 1: Counting returns and recognizing emotions

[0262] 1. The server receives the call audio data related to "returns" and converts it into text.

[0263] 2. The server analyzes the converted text data and generates a summary of the case, including important information such as the reason for return, the date of the request, and the status of the response.

[0264] 3. The server uses an emotion engine to recognize the customer's emotions during the call and adds that information to the summary.

[0265] 4. The administrator enters keywords related to "returns" on the terminal and issues instructions to check the number of cases.

[0266] 5. The server searches for cases related to "returns" and counts the number of cases.

[0267] 6. The server aggregates the data based on the counting results, creates visual graphs and tables, and integrates the emotion data.

[0268] 7. The user checks the generated report materials on the terminal and sends them to the administrator.

[0269] Example 2: Counting delivery delay cases and recognizing emotions

[0270] 1. The server receives the call audio data related to "Delivery Delay" and converts it into text.

[0271] 2. The server analyzes the text data and generates a project summary.

[0272] 3. The server uses an emotion engine to recognize the customer's emotions during the call and adds that information to the summary.

[0273] 4. The administrator enters keywords related to "delivery delay" on the terminal and issues an instruction to check the number of cases.

[0274] 5. The server searches for the number of cases related to "Delivery Delay" and aggregates the results.

[0275] 6. The server creates graphs and tables based on the aggregated data and generates report materials including emotion data.

[0276] 7. The user checks the materials and sends them to the administrator.

[0277] The introduction of this system will significantly improve the efficiency of call center operations, reduce the burden on staff, and provide more accurate and useful data that takes into account customer emotional information.

[0278] The processing flow will be explained below.

[0279] Step 1:

[0280] The server acquires the call center's voice data, which is received in batches in real time from the VoIP system.

[0281] Step 2:

[0282] The server converts the acquired voice data into text data using speech recognition technology. The speech recognition engine uses a cloud-based API, and the converted text data is temporarily stored.

[0283] Step 3:

[0284] The server analyzes the stored text data using a natural language processing toolkit (e.g., spaCy or NLTK), which includes text preprocessing (tokenization, part-of-speech tagging, stemming, and lemmatization).

[0285] Step 4:

[0286] The server applies NLP models (e.g., BERT or GPT-3) to extract key phrases and information from the text data, summarizing information such as the problem, request details, and response status, and generating a case summary in a standardized format.

[0287] Step 5:

[0288] The server uses an emotion engine to recognize the customer's emotions in real time during a call. It analyzes the intonation, speed, and tone of the voice to identify emotions. For example, it determines whether the customer is angry or satisfied.

[0289] Step 6:

[0290] The server adds the recognized emotion information to the case summary, which more clearly indicates the importance and urgency of the case. For example, an emotion like "anger" will draw special attention.

[0291] Step 7:

[0292] The terminal provides an interface for administrators to input specific keywords or phrases of interest, such as "returned goods" or "delayed delivery."

[0293] Step 8:

[0294] The server searches a text database using keywords entered from the terminal to identify relevant cases. Using a full-text search engine (e.g., Elasticsearch) enables high-speed search processing.

[0295] Step 9:

[0296] The server aggregates the search results and counts the number of matching cases. The counted data is compiled and organized as daily, weekly, and monthly statistical information.

[0297] Step 10:

[0298] The server then aggregates the recognized emotion data and analyzes the trend of emotions, for example, analyzing the frequency and tendency of negative emotions in complaint handling.

[0299] Step 11:

[0300] The server uses a data visualization tool (e.g., Matplotlib or Tableau) to generate visually easy-to-understand graphs and tables based on the aggregated data.

[0301] Step 12:

[0302] The server uses automated report generation tools (e.g., LaTeX or Pandas DataFrame) to create a report containing generated graphs and tables, including the sentiment data.

[0303] Step 13:

[0304] The user (person in charge) checks the report materials on the terminal, checking the content and format of the materials through the interface for any errors.

[0305] Step 14:

[0306] The device sends the verified report materials to the administrator, who then saves them in the internal email system or on a shared drive and notifies the appropriate administrator.

[0307] Example 2

[0308] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0309] In call center operations, traditional methods require manual recording and analysis of call content, which is labor-intensive and time-consuming, making it difficult to provide efficient customer service. Additionally, the lack of data that reflects customer sentiment makes it difficult to develop specific measures to improve customer satisfaction and the quality of customer complaint handling.

[0310] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0311] In this invention, the server includes means for receiving interactions with customers as voice data, means for converting the received voice data into text data using voice recognition technology, means for analyzing the converted text data using natural language processing technology and generating a case summary, means for recognizing customer emotion information from the analyzed text data and integrating it into the case summary, means for automatically counting the number of cases related to specific keywords or phrases and generating aggregated data, and means for automatically creating reporting materials based on the generated aggregated data. This enables efficient recording and analysis of call content and enables data collection and reporting that takes customer emotion into consideration.

[0312] "Audio data" refers to data that has been recorded and stored in digital format.

[0313] "Speech recognition technology" is a technology in which a computer analyzes voice data and converts the content into text data.

[0314] "Text data" refers to text information converted using voice recognition technology.

[0315] "Natural language processing technology" refers to the technology that allows computers to understand and analyze human language.

[0316] A "case summary" is a summary of important information extracted from text data analyzed using natural language processing technology.

[0317] "Emotional information" refers to information that indicates the emotional state of the customer during a call.

[0318] "Specific keywords or phrases" are words or expressions that an administrator specifies they want to check.

[0319] "Job counting" refers to the process of automatically counting the number of jobs related to a particular keyword or phrase.

[0320] "Aggregated data" refers to data that compiles information such as the number of cases and emotional information.

[0321] "Reporting materials" refers to report-style materials that include graphs and tables visualized based on aggregated data.

[0322] MODE FOR CARRYING OUT THE INVENTION

[0323] This invention relates to a system that converts customer interactions into text in call center operations, organizes case summaries, and automatically creates reports that take emotional information into account. This system combines speech recognition technology, natural language processing technology, emotion recognition technology, and data visualization technology.

[0324] This system uses the following main hardware and software:

[0325] Server: Receives customer voice data from the VoIP system in real time and converts it into text using a speech recognition engine. For speech recognition, it utilizes a general cloud service API.

[0326] Natural language processing tools: Analyze text data, extract key information, and generate case summaries using techniques such as tokenization, part-of-speech tagging, stemming, and lemmatization, specifically using NLTK and BERT-based language models.

[0327] Sentiment engine: Recognizes customer sentiment from voice and text data and integrates it into case summaries. This includes sentiment analysis tools such as IBM Watson®.

[0328] Database system: Stores and manages text data, case summaries, emotional information, etc.

[0329] Data visualization tools: Use tools such as Tableau to visualize the collected data and create reporting materials.

[0330] Automatic report generation tool: Uses Apache (registered trademark) FOP to generate reports in PDF format based on various data.

[0331] For example, when a customer calls a call center regarding a return, the following process is performed:

[0332] 1. The server retrieves the call audio data related to the return from the VoIP system.

[0333] 2. The server calls a common cloud service API to convert the voice data into text and store it in a database.

[0334] 3. The server performs preprocessing using NLTK, and then uses a BERT-based model to extract important information such as the reason for return, request date, and response status, and generates a summary of the case.

[0335] 4. The server uses the IBM Watson emotion engine to recognize the customer's emotion and add it to the case summary.

[0336] 5. The terminal provides an interface for the administrator to enter keywords such as "return." The server uses Elasticsearch to search for matching cases and count the number of cases.

[0337] 6. The server uses Tableau to visually transform the aggregated data into graphs and tables, and uses Apache FOP to generate reporting materials.

[0338] 7. The user (person in charge) checks the report materials on the terminal and sends them to the administrator via the internal email system or shared drive.

[0339] Examples of prompts include:

[0340] "Analyze call data related to returns, generate case summaries, and add sentiment information. Include the specific reason for the return and the date of the request."

[0341] "Analyze trends in customer sentiment by compiling performance data on delivery delays. Create visual graphs and tables and generate reports."

[0342] As described above, the introduction of this system will improve the efficiency of call center operations and data accuracy.

[0343] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0344] Step 1:

[0345] The server receives real-time call audio data from the call center's VoIP system. The audio data is input, and an audio file is generated as output. Specifically, it establishes a connection with the VoIP system and digitally records the call audio.

[0346] Step 2:

[0347] The server sends the acquired voice data to the Google Cloud Speech-to-Text API and converts the voice data into text data. The input is voice data, and the output is the converted text data. Specifically, the server sends voice data to the API endpoint via an HTTP request and receives the converted text data as an HTTP response.

[0348] Step 3:

[0349] The server preprocesses the converted text data using NLTK. Preprocessing includes tokenization, part-of-speech tagging, stemming, and lemmatization. The input is text data, and the output is preprocessed text data. Specifically, the server breaks the text data down into words, tags each word with a part of speech, and converts it to its root form.

[0350] Step 4:

[0351] The server inputs the preprocessed text data into a BERT-based NLP model to extract important information. The input is the preprocessed text data, and the output is the extracted important information (such as "problem," "request content," and "response status"). Specifically, the server converts the text data into vectors, inputs them into the BERT model, and extracts important phrases that the model predicts.

[0352] Step 5:

[0353] The server generates a summary of the case based on the extracted important information and saves it in the database. The input is the important information and the output is the summary of the case. Specifically, the server applies the important information to a template, generates a summary, and saves it in the database.

[0354] Step 6:

[0355] The server sends the customer's emotional information from the voice and text data to the IBM Watson Tone Analyzer to recognize emotions. The input is voice and text data, and the output is emotional information. Specifically, it extracts features from the voice data and sends them along with the text data to the emotion analysis API to obtain emotional information.

[0356] Step 7:

[0357] The server integrates the acquired emotion information into the case summary and saves the updated summary in the database. The input is the emotion information and the existing case summary, and the output is the updated case summary. Specifically, the emotion information is added to the existing summary and re-saved as a new summary in the database.

[0358] Step 8:

[0359] The terminal provides an interface for the administrator to input specific keywords or phrases (e.g., "returned goods" or "delayed delivery"). The input is the administrator's keywords, and the output is the search criteria. Specifically, it provides a web form and retrieves the input keywords.

[0360] Step 9:

[0361] The server uses Elasticsearch to search a text database for the entered keywords and identify matching cases. The input is the search criteria, and the output is a list of the searched cases. Specifically, it generates and executes an Elasticsearch query to retrieve the search results.

[0362] Step 10:

[0363] The server aggregates the search results and counts the number of cases by day, week, month, etc. The input is the search results and the output is the aggregated data. Specifically, the server aggregates the search results, counts the cases for each specified period, and performs the aggregation.

[0364] Step 11:

[0365] The server then uses Tableau to visually transform the aggregated data into graphs and tables. The input is the aggregated data, and the output is the visualized graph or table. Specifically, the server loads the data into Tableau, selects the appropriate visualization, and generates it.

[0366] Step 12:

[0367] The server uses Apache FOP to automatically generate visualized data as report materials. The input is visualized graphs and tables, and the output is report materials (PDF format). Specifically, the data generated in XML format is input into Apache FOP, and a PDF file is generated.

[0368] Step 13:

[0369] The user (person in charge) checks the report materials on the terminal. The input is the report materials, and the output is the confirmation results. Specifically, the user opens the materials using a PDF viewer and checks for any errors in the content or format.

[0370] Step 14:

[0371] The terminal saves the confirmed report materials in the internal email system or shared drive and notifies the appropriate administrator. The input is the confirmed report materials, and the output is a notification to the administrator. Specifically, the terminal saves the materials in a specified folder and contacts the administrator via email or the notification system.

[0372] (Application example 2)

[0373] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0374] In customer service operations in brick-and-mortar stores, it is important to efficiently collect and analyze customer requests and feedback, and then respond appropriately based on the results. However, conventional methods have made it difficult to accurately record customer comments and analyze them in a timely manner, and there are also limited means of recognizing and reflecting customer emotions in real time. This has made it difficult to improve customer satisfaction and has increased the burden on staff.

[0375] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving interactions with customers as voice data, means for converting the received voice data into text data using voice recognition technology, means for analyzing the converted text data using natural language processing technology and generating a summary of the cases, means for automatically counting the number of cases related to specific keywords or phrases and generating aggregated data, means for adding recognized emotion data to the summary, and means for automatically creating reporting materials based on the generated aggregated data. This makes it possible to efficiently collect and analyze customer requests and feedback during customer service operations in physical stores and reflect the emotional information in real time.

[0376] "Interaction with customers" refers to communication such as dialogue and information exchange between service providers and customers.

[0377] "Audio data" means data for recording, storing, and processing audio information in digital format.

[0378] "Means of receiving" refers to the technical means or devices for acquiring data or information from the outside and incorporating it into the internal environment.

[0379] "Speech recognition technology" is a technology for analyzing voice data and converting it into text data.

[0380] "Text data" is data that represents character information in digital form.

[0381] "Natural language processing technology" is an artificial intelligence technology for analyzing, understanding, and generating human language.

[0382] A "case summary" is a document or data that succinctly summarizes important information or key points about a particular case.

[0383] "Specific keywords and phrases" refer to important words or short sentences that indicate specific information or content.

[0384] "Means for automatically counting the number of cases" refers to a device or program that has the function of calculating the number of cases based on specific conditions.

[0385] "Aggregated data" is data that shows overall trends and characteristics by organizing and statistically processing multiple data.

[0386] "Recognized emotion data" refers to information about an emotional state analyzed from data such as voice or text.

[0387] "Means for adding to the summary" refers to technical means for integrating additional information into the generated summary text.

[0388] "Reporting materials" refers to documents, graphs, charts, etc. created to report specific information.

[0389] "Automated means" refers to technology or devices that perform specific tasks automatically without human intervention.

[0390] A "data visualization tool" is software that converts data into a format that is easy to understand visually, such as a graph or table.

[0391] "Smart glasses" are wearable smart devices equipped with a display, sensors, communication functions, etc.

[0392] "Customer service" refers to all work related to providing products and services to customers and ensuring their satisfaction.

[0393] The present invention provides a system for improving the efficiency of customer service operations in brick-and-mortar stores and enhancing customer satisfaction. An embodiment of this system will be described in detail below.

[0394] System Configuration

[0395] This system mainly consists of four components: "smart glasses," "server," "terminal," and "user."

[0396] 1. Smart Glasses

[0397] Smart glasses are wearable devices equipped with a display, microphone, sensors, and communication functions. They are worn by store staff while serving customers and are used to capture conversations with customers. They convert speech into text in real time and perform sentiment analysis.

[0398] 2. Server

[0399] The server performs the main processes of receiving voice data, converting it into text, analyzing it, recognizing emotions, and compiling it. It uses the following software and technologies:

[0400] Speech recognition engine: Google Cloud Speech-to-Text API

[0401] Natural language processing tool: SpaCy

[0402] Emotion recognition engine: IBM Watson Tone Analyzer

[0403] Data visualization tool: Tableau

[0404] Automated Report Generation Tool: Google Docs API

[0405] 3. Terminal

[0406] Terminals are devices used by store managers and staff to check generated reporting materials, enter data, give instructions, etc. Data can be checked, corrected, and sent through an interface.

[0407] Specific examples of implementation

[0408] Step 1: Capture and transcribe customer interactions

[0409] The server receives voice data from the smart glasses in real time and converts it into text data using the Google Cloud Speech-to-Text API.

[0410] Step 2: Analyze requirements and generate a summary

[0411] The server then uses SpaCy to analyze the converted text data and generate a summary of the case. For example, if a customer asks, "Is this product in stock?", the server generates the summary "Product inventory inquiry."

[0412] Step 3: Recognizing and integrating emotional data

[0413] The server uses IBM Watson Tone Analyzer to analyze customer sentiment from voice and text data. For example, if a customer expresses impatience, it adds that information to the summary.

[0414] Step 4: Generate and visualize aggregate data

[0415] The server automatically counts the number of jobs related to specific keywords or phrases and generates aggregate data, which is then visualized in graphs and tables using Tableau.

[0416] Step 5: Prepare, review and distribute report materials

[0417] The server uses the Google Docs API to automatically create reporting materials based on the generated aggregated data. The user (store manager) can check the materials on their device, make any necessary corrections, and then send the materials via email or cloud drive.

[0418] Prompt sentence for specific implementation example

[0419] For example, when a system captures a customer question such as "Do you have this product in stock?" and performs response and sentiment analysis, the following prompt sentences are used:

[0420] "Please give us your feedback on the product."

[0421] "Do you have any questions about this product?"

[0422] Please let us know your thoughts on the service.

[0423] This will enable efficient collection of customer requests and feedback during customer service operations in brick-and-mortar stores, enabling detailed analysis based on emotional data, and providing useful information to improve staff efficiency and increase customer satisfaction.

[0424] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0425] Step 1:

[0426] The server receives voice data from the smart glasses in real time. The voice data is captured through the microphone of the smart glasses and sent to the server through the VoIP system. The input is voice data, which is analyzed by the voice recognition engine.

[0427] Step 2:

[0428] The server uses the Google Cloud Speech-to-Text API to convert the received voice data into text data. The input is voice data, and the output is text data that expresses the contents of the voice data in characters. The server analyzes and converts the voice data.

[0429] Step 3:

[0430] The server uses SpaCy to perform natural language processing on the converted text data. The input is text data, which undergoes processes such as tokenization, part-of-speech tagging, stemming, and lemmatization. The output is each element of the analyzed text data and a summary of the case.

[0431] Step 4:

[0432] The server uses IBM Watson Tone Analyzer to recognize emotions from the analyzed text and voice data. The input is the analyzed text and voice data, and the output is the recognized emotion data. Specifically, it detects the emotional state (e.g., impatience, anger, joy) from the tone and rate of the customer's voice.

[0433] Step 5:

[0434] The server integrates the emotion data into a case summary. The input is the summary data and emotion data, and the output is a case summary including emotion information. This makes the importance and urgency of the case clear.

[0435] Step 6:

[0436] The terminal provides an interface that accepts specific keywords or phrases entered by the administrator. The input is a keyword (e.g., "satisfied," "dissatisfied," "returned product"), and the number of cases is counted based on this.

[0437] Step 7:

[0438] The server searches the database using the entered keywords and counts the number of matching cases. The input is the keywords and text database, and the output is the number of matching cases. The search is performed quickly using a full-text search engine.

[0439] Step 8:

[0440] The server aggregates the number of cases and emotion data counted and visualizes the data using Tableau. The input is the number of cases and emotion data, and the output is easy-to-understand graphs and tables.

[0441] Step 9:

[0442] The server automatically creates report materials using the Google Docs API. The input is the aggregated data and visualization data, and the output is the report document. The server automatically generates the report and creates the document in the appropriate format.

[0443] Step 10:

[0444] Users can check and modify report materials on their terminals. The input is the automatically generated report material, and the output is the final, verified report material. The user checks the content and format of the material to ensure there are no errors.

[0445] Step 11:

[0446] The terminal sends the verified report materials to the administrator. They are saved by email or cloud drive and notified to the administrator. The input is the verified report materials, and the output is the report materials that can be accessed by the administrator. This ensures that the final report is distributed appropriately.

[0447] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0448] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0449] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0450] [Second embodiment]

[0451] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0452] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0453] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0454] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0455] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0456] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0457] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0458] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0459] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0460] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0461] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0462] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0463] This invention relates to a system for call center operations that converts customer interactions into text, generates case summaries, automatically counts the number of cases, and creates report materials. This system is composed of a combination of speech recognition technology, natural language processing technology, data visualization technology, and automatic report generation technology.

[0464] Program processing natural language explanation

[0465] Step 1: Transcribe your interactions with customers

[0466] The server receives the call voice data from the call center. The call voice is received from the VoIP system and can be processed in real time or in batches.

[0467] The server converts the received voice data into text using speech recognition technology, with the speech recognition engine utilizing a widely available cloud-based API.

[0468] Step 2: Generate a case summary

[0469] The server analyzes the converted text data using a natural language processing toolkit, which performs preprocessing of the text (tokenization, part-of-speech tagging, stemming, and lemmatization).

[0470] The server applies NLP models to extract key phrases and information, thereby organizing the case's main topics and issues and generating a case summary in a standardized format.

[0471] Step 3: Counting and data collection of cases

[0472] The terminals provide an interface for inputting specific keywords and phrases, in response to instructions from the administrator, who is interested in various topics that arise in the call center.

[0473] The server uses the entered keywords to search the text database, identify and count relevant cases. The use of a full-text search engine enables high-speed search processing.

[0474] The server aggregates the search results and compiles statistics on the number of cases, generating daily, weekly, and monthly aggregate data.

[0475] Step 4: Prepare your report

[0476] The server uses a data visualization tool to generate graphs and tables using the aggregated data, allowing the data to be presented in a visually understandable format.

[0477] The server uses an automated report generation tool to create reporting materials, including generated graphs and tables, which can be output as presentations, PDFs, or other formats.

[0478] Step 5: Final review and delivery

[0479] The user (person in charge) checks the created report materials on the terminal. The interface is designed to make it easy to check the contents of the materials.

[0480] The device sends the confirmed report materials to the administrator via the internal email system or saved on a shared drive.

[0481] Specific examples

[0482] Example 1: Counting returns

[0483] 1. The server receives the call audio data related to "returns" and converts it into text.

[0484] 2. The server analyzes the converted text data and generates a summary of the case, including important information such as the reason for return, the date of the request, and the status of the response.

[0485] 3. The administrator enters keywords related to "returns" on the terminal and issues an instruction to check the number of cases.

[0486] 4. The server searches for cases related to "returns" and counts the number of cases.

[0487] 5. The server aggregates the data based on the counting results and creates visual graphs and tables.

[0488] 6. The user checks the generated report materials on the terminal and sends them to the administrator.

[0489] Example 2: Counting cases related to delivery delays

[0490] 1. The server receives the call audio data related to "Delivery Delay" and converts it into text.

[0491] 2. The server analyzes the text data and generates a project summary.

[0492] 3. The administrator enters keywords related to "delivery delay" on the terminal and issues an instruction to check the number of cases.

[0493] 4. The server searches for the number of cases related to "Delivery Delay" and aggregates the results.

[0494] 5. The server creates graphs and tables based on the aggregated data and generates reporting materials.

[0495] 6. The user checks the materials and sends them to the administrator.

[0496] The introduction of this system will significantly improve the efficiency of call center operations, reduce the burden on staff, and enable the provision of accurate data.

[0497] The processing flow will be explained below.

[0498] Step 1:

[0499] The server acquires the call center's voice data, which is received in batches in real time from the VoIP system.

[0500] Step 2:

[0501] The server converts the acquired voice data into text data using speech recognition technology. The speech recognition engine uses a cloud-based API, and the converted text data is temporarily stored.

[0502] Step 3:

[0503] The server analyzes the stored text data using a natural language processing toolkit (e.g., spaCy or NLTK), which includes tokenization, part-of-speech tagging, stemming, and lemmatization.

[0504] Step 4:

[0505] The server applies an NLP model (e.g., BERT or GPT-3) to extract key phrases and information from the text data, generating a summary of the job, including information such as the problem, request, and status.

[0506] Step 5:

[0507] The terminal receives instructions from an administrator and provides an interface for inputting specific keywords or phrases, such as "returned goods" or "delayed delivery."

[0508] Step 6:

[0509] The server searches a text database using keywords entered from the terminal to identify relevant cases. High-speed search processing is performed using a full-text search engine (e.g., Elasticsearch).

[0510] Step 7:

[0511] The server aggregates the search results and counts the number of relevant cases. The counted data is compiled and organized as daily, weekly, and monthly statistical information.

[0512] Step 8:

[0513] The server uses a data visualization tool (e.g., Matplotlib or Tableau) to generate graphs and tables that are easy to understand visually based on the aggregated data.

[0514] Step 9:

[0515] The server uses automated report generation tools (e.g., LaTeX or Pandas DataFrames) to create reports, including generated graphs and tables.

[0516] Step 10:

[0517] The user (person in charge) checks the report materials on the terminal, checking the content and format of the materials through the interface for any errors.

[0518] Step 11:

[0519] The device sends the verified report materials to the administrator, who then saves them in the internal email system or on a shared drive and notifies the appropriate administrator.

[0520] Example 1

[0521] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0522] Call center operations require efficient management of a large volume of customer interactions, as well as the ability to quickly and accurately summarize cases and generate statistical data. However, conventional systems did not integrate voice recognition, natural language processing, data visualization, or automatic report generation, which placed a heavy burden on staff and posed challenges to the accuracy and efficiency of processing.

[0523] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0524] In this invention, the server includes means for receiving interactions with customers as voice data, means for converting the received voice data into text data using voice recognition technology, means for analyzing the converted text data using natural language processing technology and generating case summaries, means for automatically counting the number of cases related to specific keywords or phrases and generating aggregated data, means for automatically creating report materials based on the generated aggregated data, means for statistically compiling the aggregated data in daily, weekly, and monthly formats, means for supporting real-time and batch processing of voice data, and means for performing tokenization, part-of-speech tagging, stemming, and lemmatization during the analysis process. This makes it possible to improve the efficiency of call center operations, reduce the burden on staff, and provide accurate data.

[0525] "Voice Data" means digital audio files or live audio streams recorded during customer interactions in a call center.

[0526] "Speech recognition technology" is a technology for analyzing voice data and converting it into text data.

[0527] "Text data" refers to textual information converted using voice recognition technology.

[0528] "Natural language processing technology" is a technology for analyzing text data, understanding its content, and extracting specific information.

[0529] The "Case Summary" is the result of organizing key information and important phrases extracted using natural language processing technology.

[0530] "Keywords" are key words or phrases used to identify a particular subject.

[0531] "Aggregated data" refers to statistically compiled data on the number of jobs related to a particular keyword.

[0532] A "data visualization tool" is software that displays aggregated data in a visually easy-to-understand format, such as a graph or table.

[0533] "Reporting materials" are documents that are created based on aggregated data and contain project summaries and statistical data.

[0534] "Automated report generation tool" means software or a tool for automatically generating reports based on data.

[0535] The "daily, weekly, and monthly format" is a format in which data is classified and summarized by day, week, or month.

[0536] "Real-time processing" refers to the process of processing audio data and producing results in real time.

[0537] "Batch processing" is a process of processing a large amount of audio data at once at regular time intervals.

[0538] "Tokenization" is the process of dividing text data into words and phrases.

[0539] "Part-of-speech tagging" is the process of assigning part-of-speech labels to segmented words and phrases.

[0540] "Stemming" is the process of extracting the roots of words.

[0541] "Lemmatization" is the process of extracting the basic forms of words.

[0542] This invention is a system for call center operations that receives customer interactions as voice data, converts it into text, generates case summaries and aggregated data, and automatically creates report materials. This system is realized by combining the following hardware and software.

[0543] Hardware and software used

[0544] 1. The server is a device that receives call audio data through the call center's VoIP system. The server is equipped with a high-speed speech recognition engine (e.g., Google Cloud Speech-to-Text API or Amazon Transcribe).

[0545] 2. The server further has the function of converting the received voice data into text data in real time or batch processing using voice recognition technology.

[0546] 3. The server uses natural language processing techniques (e.g., spaCy, NLTK, BERT model) to extract key phrases and information from the converted text data, thereby organizing the main topics and issues of the case and generating a summary of the case in a standardized format.

[0547] 4. A terminal is a device that provides an interface for administrators and users to search for specific keywords or phrases. This interface may be in the form of a web form, where administrators input instructions about topics of interest.

[0548] 5. The server uses a full-text search engine such as Elasticsearch to search the text database based on the entered keywords, identifies relevant cases, and counts the number of cases. The search results are then statistically compiled by day, week, and month.

[0549] 6. The server uses a data visualization tool (e.g., Tableau, D3.js) to generate graphs and tables from the aggregated data, allowing the data to be presented in a visually understandable format.

[0550] 7. The server uses automated report generation tools (e.g., LaTeX, Google Docs API) to create a report containing the generated graphs and tables, which can be output as a presentation, PDF, or other format.

[0551] 8. The user checks the created report materials, confirms that the contents are accurate, and then sends them to the administrator. The interface on the terminal is designed to make it easy to check the contents of the materials.

[0552] Specific examples

[0553] Counting returns cases

[0554] 1. The server receives the call audio data related to "returns" from the VoIP system and converts it to text using the Google Cloud Speech-to-Text API. When converting the audio data to text, background noise is removed and the speaker is identified.

[0555] 2. The server analyzes the converted text data using spaCy, extracts important information such as the reason for return, request date, and response status, and generates a case summary.

[0556] 3. The administrator enters keywords related to "returns" on the terminal and issues an instruction to check the number of cases.

[0557] 4. The server uses Elasticsearch to search for cases related to "returns," lists the results, and performs further aggregation. The calculated data is saved in a database.

[0558] 5. The server uses Tableau to visualize the data and generate visual graphs and tables designed to show at a glance the increase or decrease in the number of returns.

[0559] 6. The user checks the generated report materials on the terminal and verifies that there are no problems with the contents. After checking, the materials are sent to the administrator.

[0560] Counting cases related to delivery delays

[0561] 1. The server receives the call audio data related to the "Delivery Delay" from the VoIP system and converts it to text using Amazon Transcribe. During the conversion process, the audio file format and sampling rate are adjusted.

[0562] 2. The server uses the BERT model to analyze the text data and extract important information such as the cause of the delay, the date and time of occurrence, and the response status.

[0563] 3. The administrator enters keywords related to "delivery delay" on the terminal and issues an instruction to check the number of cases.

[0564] 4. The server uses Elasticsearch to search for cases related to "Delivery Delays" and aggregates the results. The search results are then stored in an SQL database.

[0565] 5. The server uses D3.js to create graphs and tables based on the aggregated results and generate report materials. These materials are often output in PDF format and shared with all project stakeholders.

[0566] 6. The user checks the document and sends it to the administrator. The check is done on the screen, and the interface makes it easy to review and correct the document.

[0567] Examples of prompt statements

[0568] "Please transcribe the conversations about returns over the past week, tally the number of cases, and create a graph to generate a report."

[0569] "Count the number of inquiries about delivery delays and create a weekly report with monthly tallies and summaries."

[0570] The introduction of this system will significantly improve the efficiency of call center operations, ensuring the accuracy of information and reducing the burden on staff.

[0571] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0572] Step 1:

[0573] The server receives the call voice data through the call center's VoIP system. At this time, the server acquires the voice data in a format suitable for receiving the voice data in real time or batch processing. The input is the voice data from the VoIP system, and the output is the raw voice data stored in the server.

[0574] Step 2:

[0575] The server converts the received voice data into text data using the Google Cloud Speech-to-Text API. This process involves cleaning the voice data (noise removal and speaker identification). The input is voice data, and the output is text data.

[0576] Step 3:

[0577] The server analyzes the converted text data using a natural language processing toolkit (e.g., spaCy, NLTK). This analysis process involves tokenization, part-of-speech tagging, stemming, and lemmatization. The input is text data, and the output is preprocessed text data.

[0578] Step 4:

[0579] The server extracts important phrases and information from the preprocessed text data using the BERT model or GPT-3, and generates a summary of the case. The input is the preprocessed text data, and the output is the summary data of the case.

[0580] Step 5:

[0581] The terminal provides an interface for the administrator to input specific keywords or phrases. The administrator inputs topics of interest, such as "returns" or "delayed delivery," which in turn sends a search command to the server. The input is the administrator's keywords, and the output is a search request to the server.

[0582] Step 6:

[0583] The server uses Elasticsearch to search the text database based on keywords entered by the administrator. The search process quickly matches the text data with the target keywords to identify and count related jobs. The input is the keywords and the text database, and the output is a list of related jobs and their counts.

[0584] Step 7:

[0585] The server statistically compiles the list of cases and the counts it receives by day, week, and month. The results are saved in a database. The input is the list of cases and the counts, and the output is the statistically compiled aggregated data.

[0586] Step 8:

[0587] The server uses a data visualization tool (e.g., Tableau, D3.js) to generate graphs and tables based on the aggregated data. This allows the data to be displayed in a visually understandable format. The input is statistically compiled aggregated data, and the output is visual data such as graphs and tables.

[0588] Step 9:

[0589] The server uses automated report generation tools (e.g., LaTeX, Google Docs API) to create a report containing the generated visual data. The report is output in presentation or PDF format. The input is visual data such as graphs and tables, and the output is the completed report.

[0590] Step 10:

[0591] The user checks the report materials created on the terminal. The interface is designed to make it easy to check the contents of the materials. Once the materials have been checked, they are sent to the administrator. The input is the completed report materials, and the output is the checked materials and the results of their transmission.

[0592] (Application example 1)

[0593] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0594] In conventional call center operations, recording call content, summarizing cases, counting the number of cases, and creating reports are often done manually, resulting in inefficiency and the consumption of large amounts of human resources. Furthermore, manual data processing has a high error rate, making it difficult to quickly obtain accurate statistical data and reports. This makes it difficult for security services to respond quickly, leading to a decline in customer satisfaction. The present invention aims to solve these problems and improve the efficiency of call center operations and the accuracy of data.

[0595] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0596] In this invention, the server includes: means for receiving interactions with customers as voice data; means for converting the received voice data into text data using voice recognition technology; means for analyzing the converted text data using natural language processing technology and generating a summary of the cases; means for automatically counting the number of cases related to specific keywords or phrases and generating aggregated data; means for automatically creating report materials based on the generated aggregated data and distributing the generated report; means including technology for converting voice data into text in real time; and means for counting the number of summarized cases and visualizing the data based on that. This makes it possible to streamline call center operations, reduce the burden on staff, provide accurate data quickly, and improve the response speed and quality of security services.

[0597] "Customer" refers to general consumers, corporations, etc. who use the service.

[0598] "Interaction" refers to calls, conversations, and inquiries between customers and call center representatives.

[0599] "Voice data" refers to audio files and digital data that record the contents of conversations with customers.

[0600] "Means for receiving" refers to a device or a method for implementing a device for loading call center calls into a server.

[0601] "Speech recognition technology" refers to technology for converting voice data into text data.

[0602] "Text data" refers to data in the form of a string of characters converted using voice recognition technology.

[0603] "Natural language processing technology" refers to technology for analyzing text data and understanding its meaning and context.

[0604] "Means for generating a case summary" refers to a device or method that uses natural language processing technology to extract important information from text data and summarize it concisely.

[0605] "Specific keywords and phrases" refer to important words and expressions related to the subject or topic of your inquiry.

[0606] "Means for automatically counting the number of cases" refers to a method for automatically calculating within the system the number of cases that match a specific keyword or phrase.

[0607] "Aggregated data" refers to data that systematically compiles the number of counted cases and related information.

[0608] "Means for automatically creating report materials" refers to a device or method for automatically generating reports or presentation materials within a system based on aggregated data.

[0609] "Means for distributing reports" refers to devices and methods for transmitting generated report materials to interested parties.

[0610] "Technology for converting voice data into text in real time" refers to technology for converting voice data into text data in real time during a call.

[0611] "Means for visualizing data" refers to devices or methods that display aggregated data in a visually easy-to-understand format, such as graphs or tables.

[0612] The system required to realize this invention includes the following hardware and software: The main elements are speech recognition, natural language processing, data visualization, and automatic report generation technologies.

[0613] The server receives customer interactions as voice data. In this invention, calls at the call center are realized using a VoIP system. The voice data is sent to the server in real time and converted into text data using the Google Cloud Speech-to-Text API. The converted text data is analyzed on the server using the Python nltk (Natural Language Toolkit) library.

[0614] This is where analysis is performed using natural language processing techniques. First, the text data undergoes preprocessing, including tokenization, part-of-speech tagging, stemming, and lemmatization. Next, NLP models are applied to extract key phrases and information and generate a case summary, resulting in a summary that summarizes the key points of the call.

[0615] The server uses Elasticsearch (a full-text search engine) to count the number of jobs related to specific keywords or phrases. Administrators specify relevant keywords or phrases on the terminal and perform searches based on them. The search results are aggregated and compiled as statistical data. This statistical data is visualized as aggregate data by day, week, or month.

[0616] To visualize the aggregated data, the server generates graphs and tables using the Tableau (data visualization tool) API, which presents the data in a format that is easy to understand visually. Report materials containing the generated graphs and tables are then created using an automated report generation tool. The generated reports are then sent to the administrator via the internal email system API.

[0617] Examples of concrete examples and prompts

[0618] Example 1: Generating a report on phishing attacks

[0619] 1. Speech Recognition:

[0620] The server uses the Google Cloud Speech-to-Text API to convert the call audio into text data.

[0621] Example prompt: "I want to know about phishing attacks."

[0622] 2. Case summary generation:

[0623] The server analyzes the converted text data and generates a summary of the case using natural language processing technology.

[0624] Example prompt: "Summarize the details of the phishing attack contained in this text."

[0625] 3. Counting the number of cases:

[0626] The administrator enters keywords related to "phishing attacks" into the terminal, and the server uses Elasticsearch to count the number of cases.

[0627] Example prompt: "Count the number of incidents related to phishing attacks."

[0628] 4. Report material generation:

[0629] The server uses the Tableau API to visualize the data and create reporting materials.

[0630] Example prompt: "Graph the number of phishing attacks by month."

[0631] 5. Submit report:

[0632] The report is sent to the administrator via the internal email system.

[0633] Example prompt: "Please email this report to your administrator."

[0634] In this way, the present invention makes it possible to improve the efficiency of call center operations, reduce the burden on staff, and improve the response speed and quality of security services.

[0635] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0636] Step 1:

[0637] The server receives the call voice data from the call center. The input is real-time voice data from the VoIP system. The server sends this data to the Google Cloud Speech-to-Text API, which converts the voice data into text data. This converted text data is the output.

[0638] Step 2:

[0639] The server receives the converted text data and analyzes it using natural language processing techniques. The input is the text data obtained in step 1. Specifically, it uses the Python nltk library to perform preprocessing such as tokenization, part-of-speech tagging, stemming, and lemmatization. It then applies an NLP model to extract key phrases and information and generate a summary of the case. This summarized text is the output.

[0640] Step 3:

[0641] The administrator enters specific keywords or phrases into the terminal. The input is the keyword or phrase specified by the administrator and is used as a search condition. The server uses Elasticsearch to search the text database based on this keyword or phrase. The number of cases that match the entered conditions is counted and this becomes the output.

[0642] Step 4:

[0643] The server aggregates the search results and compiles them as statistical data. The input is the number of cases counted in step 3. The statistical data is compiled in an aggregated format by day, week, or month. This aggregated data is the output.

[0644] Step 5:

[0645] The server performs data visualization based on the aggregated data. The input is the aggregated data generated in step 4. The server uses the Tableau API to visualize the aggregated data as graphs and tables. This visualized data is the output.

[0646] Step 6:

[0647] The server automatically creates a report containing the visualized data. The input is the visualization data generated in step 5. An automatic report generation tool is used to generate a report. This report is the output.

[0648] Step 7:

[0649] The user (person in charge) checks the report generated on the terminal. The input is the report generated by the server. The user checks the contents and makes corrections if necessary. The checked and corrected report is the output.

[0650] Step 8:

[0651] The terminal sends a report to the administrator after confirmation is complete. The input is the report confirmed in step 7. The report is sent to the administrator via the internal mail system API or shared drive. This sent report becomes the final output.

[0652] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0653] This invention relates to a system for call center operations that converts customer interactions into text, generates case summaries, automatically counts the number of cases, and creates report materials while taking into account customer emotional information using an emotion engine. This system is composed of a combination of speech recognition technology, natural language processing technology, emotion recognition technology, data visualization technology, and automatic report generation technology.

[0654] Program processing natural language explanation

[0655] Step 1: Transcribe your interactions with customers

[0656] The server acquires the call voice data of the call center. The call voice is received in batches in real time from the VoIP system.

[0657] The server uses voice recognition technology to convert the received voice data into text data. The voice recognition engine utilizes a cloud-based API.

[0658] Step 2: Generate a case summary

[0659] The server analyzes the converted text data using a natural language processing toolkit, which includes tokenization, part-of-speech tagging, stemming, and lemmatization.

[0660] The server applies NLP models to extract key phrases and information from the text data, generating a summary of the case, including information such as the issue, request details, and response status.

[0661] Step 3: Recognizing and integrating emotional information

[0662] The server uses an emotion engine to recognize the customer's emotions during the call in real time, for example by analyzing the intonation, speed, and tone of the voice to identify the emotion.

[0663] The server adds the recognized emotional information to the case summary, which makes the importance and urgency of the case more clear.

[0664] Step 4: Counting and data collection of cases

[0665] The terminal receives instructions from an administrator and provides an interface for inputting specific keywords or phrases, such as "returned goods" or "delayed delivery."

[0666] The server uses the entered keywords to search a text database and identify relevant cases. High-speed search processing is performed using a full-text search engine.

[0667] The server aggregates the search results and counts the number of relevant cases. The counted data is compiled and organized as daily, weekly, and monthly statistical information.

[0668] Step 5: Integrate and analyze sentiment data

[0669] The server integrates the recognized emotion data into aggregate data and analyzes emotion trends, for example, analyzing the frequency of negative emotions when handling complaints.

[0670] Step 6: Prepare your report

[0671] The server uses a data visualization tool to generate graphs and tables that are easy to understand visually based on the aggregated data.

[0672] The server uses an automated report generation tool to create a report containing generated graphs and tables, including the emotion data.

[0673] Step 7: Final review and delivery

[0674] The user (person in charge) checks the report materials on the terminal, checking the content and format of the materials through the interface for any errors.

[0675] The device sends the verified report materials to the administrator, who then saves them in the internal email system or on a shared drive and notifies the appropriate administrator.

[0676] Specific examples

[0677] Example 1: Counting returns and recognizing emotions

[0678] 1. The server receives the call audio data related to "returns" and converts it into text.

[0679] 2. The server analyzes the converted text data and generates a summary of the case, including important information such as the reason for return, the date of the request, and the status of the response.

[0680] 3. The server uses an emotion engine to recognize the customer's emotions during the call and adds that information to the summary.

[0681] 4. The administrator enters keywords related to "returns" on the terminal and issues instructions to check the number of cases.

[0682] 5. The server searches for cases related to "returns" and counts the number of cases.

[0683] 6. The server aggregates the data based on the counting results, creates visual graphs and tables, and integrates the emotion data.

[0684] 7. The user checks the generated report materials on the terminal and sends them to the administrator.

[0685] Example 2: Counting delivery delay cases and recognizing emotions

[0686] 1. The server receives the call audio data related to "Delivery Delay" and converts it into text.

[0687] 2. The server analyzes the text data and generates a project summary.

[0688] 3. The server uses an emotion engine to recognize the customer's emotions during the call and adds that information to the summary.

[0689] 4. The administrator enters keywords related to "delivery delay" on the terminal and issues an instruction to check the number of cases.

[0690] 5. The server searches for the number of cases related to "Delivery Delay" and aggregates the results.

[0691] 6. The server creates graphs and tables based on the aggregated data and generates report materials including emotion data.

[0692] 7. The user checks the materials and sends them to the administrator.

[0693] The introduction of this system will significantly improve the efficiency of call center operations, reduce the burden on staff, and provide more accurate and useful data that takes into account customer emotional information.

[0694] The processing flow will be explained below.

[0695] Step 1:

[0696] The server acquires the call center's voice data, which is received in batches in real time from the VoIP system.

[0697] Step 2:

[0698] The server converts the acquired voice data into text data using speech recognition technology. The speech recognition engine uses a cloud-based API, and the converted text data is temporarily stored.

[0699] Step 3:

[0700] The server analyzes the stored text data using a natural language processing toolkit (e.g., spaCy or NLTK), which includes text preprocessing (tokenization, part-of-speech tagging, stemming, and lemmatization).

[0701] Step 4:

[0702] The server applies NLP models (e.g., BERT or GPT-3) to extract key phrases and information from the text data, summarizing information such as the problem, request details, and response status, and generating a case summary in a standardized format.

[0703] Step 5:

[0704] The server uses an emotion engine to recognize the customer's emotions in real time during a call. It analyzes the intonation, speed, and tone of the voice to identify emotions. For example, it determines whether the customer is angry or satisfied.

[0705] Step 6:

[0706] The server adds the recognized emotion information to the case summary, which more clearly indicates the importance and urgency of the case. For example, an emotion like "anger" will draw special attention.

[0707] Step 7:

[0708] The terminal provides an interface for administrators to input specific keywords or phrases of interest, such as "returned goods" or "delayed delivery."

[0709] Step 8:

[0710] The server searches a text database using keywords entered from the terminal to identify relevant cases. Using a full-text search engine (e.g., Elasticsearch) enables high-speed search processing.

[0711] Step 9:

[0712] The server aggregates the search results and counts the number of matching cases. The counted data is compiled and organized as daily, weekly, and monthly statistical information.

[0713] Step 10:

[0714] The server then aggregates the recognized emotion data and analyzes the trend of emotions, for example, analyzing the frequency and tendency of negative emotions in complaint handling.

[0715] Step 11:

[0716] The server uses a data visualization tool (e.g., Matplotlib or Tableau) to generate visually easy-to-understand graphs and tables based on the aggregated data.

[0717] Step 12:

[0718] The server uses automated report generation tools (e.g., LaTeX or Pandas DataFrame) to create a report containing generated graphs and tables, including the sentiment data.

[0719] Step 13:

[0720] The user (person in charge) checks the report materials on the terminal, checking the content and format of the materials through the interface for any errors.

[0721] Step 14:

[0722] The device sends the verified report materials to the administrator, who then saves them in the internal email system or on a shared drive and notifies the appropriate administrator.

[0723] Example 2

[0724] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0725] In call center operations, traditional methods require manual recording and analysis of call content, which is labor-intensive and time-consuming, making it difficult to provide efficient customer service. Additionally, the lack of data that reflects customer sentiment makes it difficult to develop specific measures to improve customer satisfaction and the quality of customer complaint handling.

[0726] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0727] In this invention, the server includes means for receiving interactions with customers as voice data, means for converting the received voice data into text data using voice recognition technology, means for analyzing the converted text data using natural language processing technology and generating a case summary, means for recognizing customer emotion information from the analyzed text data and integrating it into the case summary, means for automatically counting the number of cases related to specific keywords or phrases and generating aggregated data, and means for automatically creating reporting materials based on the generated aggregated data. This enables efficient recording and analysis of call content and enables data collection and reporting that takes customer emotion into consideration.

[0728] "Audio data" refers to data that has been recorded and stored in digital format.

[0729] "Speech recognition technology" is a technology in which a computer analyzes voice data and converts the content into text data.

[0730] "Text data" refers to text information converted using voice recognition technology.

[0731] "Natural language processing technology" refers to the technology that allows computers to understand and analyze human language.

[0732] A "case summary" is a summary of important information extracted from text data analyzed using natural language processing technology.

[0733] "Emotional information" refers to information that indicates the emotional state of the customer during a call.

[0734] "Specific keywords or phrases" are words or expressions that an administrator specifies they want to check.

[0735] "Job counting" refers to the process of automatically counting the number of jobs related to a particular keyword or phrase.

[0736] "Aggregated data" refers to data that compiles information such as the number of cases and emotional information.

[0737] "Reporting materials" refers to report-style materials that include graphs and tables visualized based on aggregated data.

[0738] MODE FOR CARRYING OUT THE INVENTION

[0739] This invention relates to a system that converts customer interactions into text in call center operations, organizes case summaries, and automatically creates reports that take emotional information into account. This system combines speech recognition technology, natural language processing technology, emotion recognition technology, and data visualization technology.

[0740] This system uses the following main hardware and software:

[0741] Server: Receives customer voice data from the VoIP system in real time and converts it into text using a speech recognition engine. For speech recognition, it utilizes a general cloud service API.

[0742] Natural language processing tools: Analyze text data, extract key information, and generate case summaries using techniques such as tokenization, part-of-speech tagging, stemming, and lemmatization, specifically using NLTK and BERT-based language models.

[0743] Sentiment engine: Recognizes customer sentiment from voice and text data and integrates it into case summaries. This includes sentiment analysis tools such as IBM Watson.

[0744] Database system: Stores and manages text data, case summaries, emotional information, etc.

[0745] Data visualization tools: Use tools such as Tableau to visualize the collected data and create reporting materials.

[0746] Automatic report generation tool: Uses Apache FOP to generate reports in PDF format based on various data.

[0747] For example, when a customer calls a call center regarding a return, the following process is performed:

[0748] 1. The server retrieves the call audio data related to the return from the VoIP system.

[0749] 2. The server calls a common cloud service API to convert the voice data into text and store it in a database.

[0750] 3. The server performs preprocessing using NLTK, and then uses a BERT-based model to extract important information such as the reason for return, request date, and response status, and generates a summary of the case.

[0751] 4. The server uses the IBM Watson emotion engine to recognize the customer's emotion and add it to the case summary.

[0752] 5. The terminal provides an interface for the administrator to enter keywords such as "return." The server uses Elasticsearch to search for matching cases and count the number of cases.

[0753] 6. The server uses Tableau to visually transform the aggregated data into graphs and tables, and uses Apache FOP to generate reporting materials.

[0754] 7. The user (person in charge) checks the report materials on the terminal and sends them to the administrator via the internal email system or shared drive.

[0755] Examples of prompts include:

[0756] "Analyze call data related to returns, generate case summaries, and add sentiment information. Include the specific reason for the return and the date of the request."

[0757] "Analyze trends in customer sentiment by compiling performance data on delivery delays. Create visual graphs and tables and generate reports."

[0758] As described above, the introduction of this system will improve the efficiency of call center operations and data accuracy.

[0759] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0760] Step 1:

[0761] The server receives real-time call audio data from the call center's VoIP system. The audio data is input, and an audio file is generated as output. Specifically, it establishes a connection with the VoIP system and digitally records the call audio.

[0762] Step 2:

[0763] The server sends the acquired voice data to the Google Cloud Speech-to-Text API and converts the voice data into text data. The input is voice data, and the output is the converted text data. Specifically, the server sends voice data to the API endpoint via an HTTP request and receives the converted text data as an HTTP response.

[0764] Step 3:

[0765] The server preprocesses the converted text data using NLTK. Preprocessing includes tokenization, part-of-speech tagging, stemming, and lemmatization. The input is text data, and the output is preprocessed text data. Specifically, the server breaks the text data down into words, tags each word with a part of speech, and converts it to its root form.

[0766] Step 4:

[0767] The server inputs the preprocessed text data into a BERT-based NLP model to extract important information. The input is the preprocessed text data, and the output is the extracted important information (such as "problem," "request content," and "response status"). Specifically, the server converts the text data into vectors, inputs them into the BERT model, and extracts important phrases that the model predicts.

[0768] Step 5:

[0769] The server generates a summary of the case based on the extracted important information and saves it in the database. The input is the important information and the output is the summary of the case. Specifically, the server applies the important information to a template, generates a summary, and saves it in the database.

[0770] Step 6:

[0771] The server sends the customer's emotional information from the voice and text data to the IBM Watson Tone Analyzer to recognize emotions. The input is voice and text data, and the output is emotional information. Specifically, it extracts features from the voice data and sends them along with the text data to the emotion analysis API to obtain emotional information.

[0772] Step 7:

[0773] The server integrates the acquired emotion information into the case summary and saves the updated summary in the database. The input is the emotion information and the existing case summary, and the output is the updated case summary. Specifically, the emotion information is added to the existing summary and re-saved as a new summary in the database.

[0774] Step 8:

[0775] The terminal provides an interface for the administrator to input specific keywords or phrases (e.g., "returned goods" or "delayed delivery"). The input is the administrator's keywords, and the output is the search criteria. Specifically, it provides a web form and retrieves the input keywords.

[0776] Step 9:

[0777] The server uses Elasticsearch to search a text database for the entered keywords and identify matching cases. The input is the search criteria, and the output is a list of the searched cases. Specifically, it generates and executes an Elasticsearch query to retrieve the search results.

[0778] Step 10:

[0779] The server aggregates the search results and counts the number of cases by day, week, month, etc. The input is the search results and the output is the aggregated data. Specifically, the server aggregates the search results, counts the cases for each specified period, and performs the aggregation.

[0780] Step 11:

[0781] The server then uses Tableau to visually transform the aggregated data into graphs and tables. The input is the aggregated data, and the output is the visualized graph or table. Specifically, the server loads the data into Tableau, selects the appropriate visualization, and generates it.

[0782] Step 12:

[0783] The server uses Apache FOP to automatically generate visualized data as report materials. The input is visualized graphs and tables, and the output is report materials (PDF format). Specifically, the data generated in XML format is input into Apache FOP, and a PDF file is generated.

[0784] Step 13:

[0785] The user (person in charge) checks the report materials on the terminal. The input is the report materials, and the output is the confirmation results. Specifically, the user opens the materials using a PDF viewer and checks for any errors in the content or format.

[0786] Step 14:

[0787] The terminal saves the confirmed report materials in the internal email system or shared drive and notifies the appropriate administrator. The input is the confirmed report materials, and the output is a notification to the administrator. Specifically, the terminal saves the materials in a specified folder and contacts the administrator via email or the notification system.

[0788] (Application example 2)

[0789] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0790] In customer service operations in brick-and-mortar stores, it is important to efficiently collect and analyze customer requests and feedback, and then respond appropriately based on the results. However, conventional methods have made it difficult to accurately record customer comments and analyze them in a timely manner, and there are also limited means of recognizing and reflecting customer emotions in real time. This has made it difficult to improve customer satisfaction and has increased the burden on staff.

[0791] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving interactions with customers as voice data, means for converting the received voice data into text data using voice recognition technology, means for analyzing the converted text data using natural language processing technology and generating a summary of the cases, means for automatically counting the number of cases related to specific keywords or phrases and generating aggregated data, means for adding recognized emotion data to the summary, and means for automatically creating reporting materials based on the generated aggregated data. This makes it possible to efficiently collect and analyze customer requests and feedback during customer service operations in physical stores and reflect the emotional information in real time.

[0792] "Interaction with customers" refers to communication such as dialogue and information exchange between service providers and customers.

[0793] "Audio data" means data for recording, storing, and processing audio information in digital format.

[0794] "Means of receiving" refers to the technical means or devices for acquiring data or information from the outside and incorporating it into the internal environment.

[0795] "Speech recognition technology" is a technology for analyzing voice data and converting it into text data.

[0796] "Text data" is data that represents character information in digital form.

[0797] "Natural language processing technology" is an artificial intelligence technology for analyzing, understanding, and generating human language.

[0798] A "case summary" is a document or data that succinctly summarizes important information or key points about a particular case.

[0799] "Specific keywords and phrases" refer to important words or short sentences that indicate specific information or content.

[0800] "Means for automatically counting the number of cases" refers to a device or program that has the function of calculating the number of cases based on specific conditions.

[0801] "Aggregated data" is data that shows overall trends and characteristics by organizing and statistically processing multiple data.

[0802] "Recognized emotion data" refers to information about an emotional state analyzed from data such as voice or text.

[0803] "Means for adding to the summary" refers to technical means for integrating additional information into the generated summary text.

[0804] "Reporting materials" refers to documents, graphs, charts, etc. created to report specific information.

[0805] "Automated means" refers to technology or devices that perform specific tasks automatically without human intervention.

[0806] A "data visualization tool" is software that converts data into a format that is easy to understand visually, such as a graph or table.

[0807] "Smart glasses" are wearable smart devices equipped with a display, sensors, communication functions, etc.

[0808] "Customer service" refers to all work related to providing products and services to customers and ensuring their satisfaction.

[0809] The present invention provides a system for improving the efficiency of customer service operations in brick-and-mortar stores and enhancing customer satisfaction. An embodiment of this system will be described in detail below.

[0810] System Configuration

[0811] This system mainly consists of four components: "smart glasses," "server," "terminal," and "user."

[0812] 1. Smart Glasses

[0813] Smart glasses are wearable devices equipped with a display, microphone, sensors, and communication functions. They are worn by store staff while serving customers and are used to capture conversations with customers. They convert speech into text in real time and perform sentiment analysis.

[0814] 2. Server

[0815] The server performs the main processes of receiving voice data, converting it into text, analyzing it, recognizing emotions, and compiling it. It uses the following software and technologies:

[0816] Speech recognition engine: Google Cloud Speech-to-Text API

[0817] Natural language processing tool: SpaCy

[0818] Emotion recognition engine: IBM Watson Tone Analyzer

[0819] Data visualization tool: Tableau

[0820] Automated Report Generation Tool: Google Docs API

[0821] 3. Terminal

[0822] Terminals are devices used by store managers and staff to check generated reporting materials, enter data, give instructions, etc. Data can be checked, corrected, and sent through an interface.

[0823] Specific examples of implementation

[0824] Step 1: Capture and transcribe customer interactions

[0825] The server receives voice data from the smart glasses in real time and converts it into text data using the Google Cloud Speech-to-Text API.

[0826] Step 2: Analyze requirements and generate a summary

[0827] The server then uses SpaCy to analyze the converted text data and generate a summary of the case. For example, if a customer asks, "Is this product in stock?", the server generates the summary "Product inventory inquiry."

[0828] Step 3: Recognizing and integrating emotional data

[0829] The server uses IBM Watson Tone Analyzer to analyze customer sentiment from voice and text data. For example, if a customer expresses impatience, it adds that information to the summary.

[0830] Step 4: Generate and visualize aggregate data

[0831] The server automatically counts the number of jobs related to specific keywords or phrases and generates aggregate data, which is then visualized in graphs and tables using Tableau.

[0832] Step 5: Prepare, review and distribute report materials

[0833] The server uses the Google Docs API to automatically create reporting materials based on the generated aggregated data. The user (store manager) can check the materials on their device, make any necessary corrections, and then send the materials via email or cloud drive.

[0834] Prompt sentence for specific implementation example

[0835] For example, when a system captures a customer question such as "Do you have this product in stock?" and performs response and sentiment analysis, the following prompt sentences are used:

[0836] "Please give us your feedback on the product."

[0837] "Do you have any questions about this product?"

[0838] Please let us know your thoughts on the service.

[0839] This will enable efficient collection of customer requests and feedback during customer service operations in brick-and-mortar stores, enabling detailed analysis based on emotional data, and providing useful information to improve staff efficiency and increase customer satisfaction.

[0840] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0841] Step 1:

[0842] The server receives voice data from the smart glasses in real time. The voice data is captured through the microphone of the smart glasses and sent to the server through the VoIP system. The input is voice data, which is analyzed by the voice recognition engine.

[0843] Step 2:

[0844] The server uses the Google Cloud Speech-to-Text API to convert the received voice data into text data. The input is voice data, and the output is text data that expresses the contents of the voice data in characters. The server analyzes and converts the voice data.

[0845] Step 3:

[0846] The server uses SpaCy to perform natural language processing on the converted text data. The input is text data, which undergoes processes such as tokenization, part-of-speech tagging, stemming, and lemmatization. The output is each element of the analyzed text data and a summary of the case.

[0847] Step 4:

[0848] The server uses IBM Watson Tone Analyzer to recognize emotions from the analyzed text and voice data. The input is the analyzed text and voice data, and the output is the recognized emotion data. Specifically, it detects the emotional state (e.g., impatience, anger, joy) from the tone and rate of the customer's voice.

[0849] Step 5:

[0850] The server integrates the emotion data into a case summary. The input is the summary data and emotion data, and the output is a case summary including emotion information. This makes the importance and urgency of the case clear.

[0851] Step 6:

[0852] The terminal provides an interface that accepts specific keywords or phrases entered by the administrator. The input is a keyword (e.g., "satisfied," "dissatisfied," "returned product"), and the number of cases is counted based on this.

[0853] Step 7:

[0854] The server searches the database using the entered keywords and counts the number of matching cases. The input is the keywords and text database, and the output is the number of matching cases. The search is performed quickly using a full-text search engine.

[0855] Step 8:

[0856] The server aggregates the number of cases and emotion data counted and visualizes the data using Tableau. The input is the number of cases and emotion data, and the output is easy-to-understand graphs and tables.

[0857] Step 9:

[0858] The server automatically creates report materials using the Google Docs API. The input is the aggregated data and visualization data, and the output is the report document. The server automatically generates the report and creates the document in the appropriate format.

[0859] Step 10:

[0860] Users can check and modify report materials on their terminals. The input is the automatically generated report material, and the output is the final, verified report material. The user checks the content and format of the material to ensure there are no errors.

[0861] Step 11:

[0862] The terminal sends the verified report materials to the administrator. They are saved by email or cloud drive and notified to the administrator. The input is the verified report materials, and the output is the report materials that can be accessed by the administrator. This ensures that the final report is distributed appropriately.

[0863] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0864] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0865] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0866] [Third embodiment]

[0867] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0868] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0869] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0870] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0871] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0872] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0873] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0874] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0875] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0876] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0877] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0878] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0879] This invention relates to a system for call center operations that converts customer interactions into text, generates case summaries, automatically counts the number of cases, and creates report materials. This system is composed of a combination of speech recognition technology, natural language processing technology, data visualization technology, and automatic report generation technology.

[0880] Program processing natural language explanation

[0881] Step 1: Transcribe your interactions with customers

[0882] The server receives the call voice data from the call center. The call voice is received from the VoIP system and can be processed in real time or in batches.

[0883] The server converts the received voice data into text using speech recognition technology, with the speech recognition engine utilizing a widely available cloud-based API.

[0884] Step 2: Generate a case summary

[0885] The server analyzes the converted text data using a natural language processing toolkit, which performs preprocessing of the text (tokenization, part-of-speech tagging, stemming, and lemmatization).

[0886] The server applies NLP models to extract key phrases and information, thereby organizing the case's main topics and issues and generating a case summary in a standardized format.

[0887] Step 3: Counting and data collection of cases

[0888] The terminals provide an interface for inputting specific keywords and phrases, in response to instructions from the administrator, who is interested in various topics that arise in the call center.

[0889] The server uses the entered keywords to search the text database, identify and count relevant cases. The use of a full-text search engine enables high-speed search processing.

[0890] The server aggregates the search results and compiles statistics on the number of cases, generating daily, weekly, and monthly aggregate data.

[0891] Step 4: Prepare your report

[0892] The server uses a data visualization tool to generate graphs and tables using the aggregated data, allowing the data to be presented in a visually understandable format.

[0893] The server uses an automated report generation tool to create reporting materials, including generated graphs and tables, which can be output as presentations, PDFs, or other formats.

[0894] Step 5: Final review and delivery

[0895] The user (person in charge) checks the created report materials on the terminal. The interface is designed to make it easy to check the contents of the materials.

[0896] The device sends the confirmed report materials to the administrator via the internal email system or saved on a shared drive.

[0897] Specific examples

[0898] Example 1: Counting returns

[0899] 1. The server receives the call audio data related to "returns" and converts it into text.

[0900] 2. The server analyzes the converted text data and generates a summary of the case, including important information such as the reason for return, the date of the request, and the status of the response.

[0901] 3. The administrator enters keywords related to "returns" on the terminal and issues an instruction to check the number of cases.

[0902] 4. The server searches for cases related to "returns" and counts the number of cases.

[0903] 5. The server aggregates the data based on the counting results and creates visual graphs and tables.

[0904] 6. The user checks the generated report materials on the terminal and sends them to the administrator.

[0905] Example 2: Counting cases related to delivery delays

[0906] 1. The server receives the call audio data related to "Delivery Delay" and converts it into text.

[0907] 2. The server analyzes the text data and generates a project summary.

[0908] 3. The administrator enters keywords related to "delivery delay" on the terminal and issues an instruction to check the number of cases.

[0909] 4. The server searches for the number of cases related to "Delivery Delay" and aggregates the results.

[0910] 5. The server creates graphs and tables based on the aggregated data and generates reporting materials.

[0911] 6. The user checks the materials and sends them to the administrator.

[0912] The introduction of this system will significantly improve the efficiency of call center operations, reduce the burden on staff, and enable the provision of accurate data.

[0913] The processing flow will be explained below.

[0914] Step 1:

[0915] The server acquires the call center's voice data, which is received in batches in real time from the VoIP system.

[0916] Step 2:

[0917] The server converts the acquired voice data into text data using speech recognition technology. The speech recognition engine uses a cloud-based API, and the converted text data is temporarily stored.

[0918] Step 3:

[0919] The server analyzes the stored text data using a natural language processing toolkit (e.g., spaCy or NLTK), which includes tokenization, part-of-speech tagging, stemming, and lemmatization.

[0920] Step 4:

[0921] The server applies an NLP model (e.g., BERT or GPT-3) to extract key phrases and information from the text data, generating a summary of the job, including information such as the problem, request, and status.

[0922] Step 5:

[0923] The terminal receives instructions from an administrator and provides an interface for inputting specific keywords or phrases, such as "returned goods" or "delayed delivery."

[0924] Step 6:

[0925] The server searches a text database using keywords entered from the terminal to identify relevant cases. High-speed search processing is performed using a full-text search engine (e.g., Elasticsearch).

[0926] Step 7:

[0927] The server aggregates the search results and counts the number of relevant cases. The counted data is compiled and organized as daily, weekly, and monthly statistical information.

[0928] Step 8:

[0929] The server uses a data visualization tool (e.g., Matplotlib or Tableau) to generate graphs and tables that are easy to understand visually based on the aggregated data.

[0930] Step 9:

[0931] The server uses automated report generation tools (e.g., LaTeX or Pandas DataFrames) to create reports, including generated graphs and tables.

[0932] Step 10:

[0933] The user (person in charge) checks the report materials on the terminal, checking the content and format of the materials through the interface for any errors.

[0934] Step 11:

[0935] The device sends the verified report materials to the administrator, who then saves them in the internal email system or on a shared drive and notifies the appropriate administrator.

[0936] Example 1

[0937] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0938] Call center operations require efficient management of a large volume of customer interactions, as well as the ability to quickly and accurately summarize cases and generate statistical data. However, conventional systems did not integrate voice recognition, natural language processing, data visualization, or automatic report generation, which placed a heavy burden on staff and posed challenges to the accuracy and efficiency of processing.

[0939] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0940] In this invention, the server includes means for receiving interactions with customers as voice data, means for converting the received voice data into text data using voice recognition technology, means for analyzing the converted text data using natural language processing technology and generating case summaries, means for automatically counting the number of cases related to specific keywords or phrases and generating aggregated data, means for automatically creating report materials based on the generated aggregated data, means for statistically compiling the aggregated data in daily, weekly, and monthly formats, means for supporting real-time and batch processing of voice data, and means for performing tokenization, part-of-speech tagging, stemming, and lemmatization during the analysis process. This makes it possible to improve the efficiency of call center operations, reduce the burden on staff, and provide accurate data.

[0941] "Voice Data" means digital audio files or live audio streams recorded during customer interactions in a call center.

[0942] "Speech recognition technology" is a technology for analyzing voice data and converting it into text data.

[0943] "Text data" refers to textual information converted using voice recognition technology.

[0944] "Natural language processing technology" is a technology for analyzing text data, understanding its content, and extracting specific information.

[0945] The "Case Summary" is the result of organizing key information and important phrases extracted using natural language processing technology.

[0946] "Keywords" are key words or phrases used to identify a particular subject.

[0947] "Aggregated data" refers to statistically compiled data on the number of jobs related to a particular keyword.

[0948] A "data visualization tool" is software that displays aggregated data in a visually easy-to-understand format, such as a graph or table.

[0949] "Reporting materials" are documents that are created based on aggregated data and contain project summaries and statistical data.

[0950] "Automated report generation tool" means software or a tool for automatically generating reports based on data.

[0951] The "daily, weekly, and monthly format" is a format in which data is classified and summarized by day, week, or month.

[0952] "Real-time processing" refers to the process of processing audio data and producing results in real time.

[0953] "Batch processing" is a process of processing a large amount of audio data at once at regular time intervals.

[0954] "Tokenization" is the process of dividing text data into words and phrases.

[0955] "Part-of-speech tagging" is the process of assigning part-of-speech labels to segmented words and phrases.

[0956] "Stemming" is the process of extracting the roots of words.

[0957] "Lemmatization" is the process of extracting the basic forms of words.

[0958] This invention is a system for call center operations that receives customer interactions as voice data, converts it into text, generates case summaries and aggregated data, and automatically creates report materials. This system is realized by combining the following hardware and software.

[0959] Hardware and software used

[0960] 1. The server is a device that receives call audio data through the call center's VoIP system. The server is equipped with a high-speed speech recognition engine (e.g., Google Cloud Speech-to-Text API or Amazon Transcribe).

[0961] 2. The server further has the function of converting the received voice data into text data in real time or batch processing using voice recognition technology.

[0962] 3. The server uses natural language processing techniques (e.g., spaCy, NLTK, BERT model) to extract key phrases and information from the converted text data, thereby organizing the main topics and issues of the case and generating a summary of the case in a standardized format.

[0963] 4. A terminal is a device that provides an interface for administrators and users to search for specific keywords or phrases. This interface may be in the form of a web form, where administrators input instructions about topics of interest.

[0964] 5. The server uses a full-text search engine such as Elasticsearch to search the text database based on the entered keywords, identifies relevant cases, and counts the number of cases. The search results are then statistically compiled by day, week, and month.

[0965] 6. The server uses a data visualization tool (e.g., Tableau, D3.js) to generate graphs and tables from the aggregated data, allowing the data to be presented in a visually understandable format.

[0966] 7. The server uses automated report generation tools (e.g., LaTeX, Google Docs API) to create a report containing the generated graphs and tables, which can be output as a presentation, PDF, or other format.

[0967] 8. The user checks the created report materials, confirms that the contents are accurate, and then sends them to the administrator. The interface on the terminal is designed to make it easy to check the contents of the materials.

[0968] Specific examples

[0969] Counting returns cases

[0970] 1. The server receives the call audio data related to "returns" from the VoIP system and converts it to text using the Google Cloud Speech-to-Text API. When converting the audio data to text, background noise is removed and the speaker is identified.

[0971] 2. The server analyzes the converted text data using spaCy, extracts important information such as the reason for return, request date, and response status, and generates a case summary.

[0972] 3. The administrator enters keywords related to "returns" on the terminal and issues an instruction to check the number of cases.

[0973] 4. The server uses Elasticsearch to search for cases related to "returns," lists the results, and performs further aggregation. The calculated data is saved in a database.

[0974] 5. The server uses Tableau to visualize the data and generate visual graphs and tables designed to show at a glance the increase or decrease in the number of returns.

[0975] 6. The user checks the generated report materials on the terminal and verifies that there are no problems with the contents. After checking, the materials are sent to the administrator.

[0976] Counting cases related to delivery delays

[0977] 1. The server receives the call audio data related to the "Delivery Delay" from the VoIP system and converts it to text using Amazon Transcribe. During the conversion process, the audio file format and sampling rate are adjusted.

[0978] 2. The server uses the BERT model to analyze the text data and extract important information such as the cause of the delay, the date and time of occurrence, and the response status.

[0979] 3. The administrator enters keywords related to "delivery delay" on the terminal and issues an instruction to check the number of cases.

[0980] 4. The server uses Elasticsearch to search for cases related to "Delivery Delays" and aggregates the results. The search results are then stored in an SQL database.

[0981] 5. The server uses D3.js to create graphs and tables based on the aggregated results and generate report materials. These materials are often output in PDF format and shared with all project stakeholders.

[0982] 6. The user checks the document and sends it to the administrator. The check is done on the screen, and the interface makes it easy to review and correct the document.

[0983] Examples of prompt statements

[0984] "Please transcribe the conversations about returns over the past week, tally the number of cases, and create a graph to generate a report."

[0985] "Count the number of inquiries about delivery delays and create a weekly report with monthly tallies and summaries."

[0986] The introduction of this system will significantly improve the efficiency of call center operations, ensuring the accuracy of information and reducing the burden on staff.

[0987] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0988] Step 1:

[0989] The server receives the call voice data through the call center's VoIP system. At this time, the server acquires the voice data in a format suitable for receiving the voice data in real time or batch processing. The input is the voice data from the VoIP system, and the output is the raw voice data stored in the server.

[0990] Step 2:

[0991] The server converts the received voice data into text data using the Google Cloud Speech-to-Text API. This process involves cleaning the voice data (noise removal and speaker identification). The input is voice data, and the output is text data.

[0992] Step 3:

[0993] The server analyzes the converted text data using a natural language processing toolkit (e.g., spaCy, NLTK). This analysis process involves tokenization, part-of-speech tagging, stemming, and lemmatization. The input is text data, and the output is preprocessed text data.

[0994] Step 4:

[0995] The server extracts important phrases and information from the preprocessed text data using the BERT model or GPT-3, and generates a summary of the case. The input is the preprocessed text data, and the output is the summary data of the case.

[0996] Step 5:

[0997] The terminal provides an interface for the administrator to input specific keywords or phrases. The administrator inputs topics of interest, such as "returns" or "delayed delivery," which in turn sends a search command to the server. The input is the administrator's keywords, and the output is a search request to the server.

[0998] Step 6:

[0999] The server uses Elasticsearch to search the text database based on keywords entered by the administrator. The search process quickly matches the text data with the target keywords to identify and count related jobs. The input is the keywords and the text database, and the output is a list of related jobs and their counts.

[1000] Step 7:

[1001] The server statistically compiles the list of cases and the counts it receives by day, week, and month. The results are saved in a database. The input is the list of cases and the counts, and the output is the statistically compiled aggregated data.

[1002] Step 8:

[1003] The server uses a data visualization tool (e.g., Tableau, D3.js) to generate graphs and tables based on the aggregated data. This allows the data to be displayed in a visually understandable format. The input is statistically compiled aggregated data, and the output is visual data such as graphs and tables.

[1004] Step 9:

[1005] The server uses automated report generation tools (e.g., LaTeX, Google Docs API) to create a report containing the generated visual data. The report is output in presentation or PDF format. The input is visual data such as graphs and tables, and the output is the completed report.

[1006] Step 10:

[1007] The user checks the report materials created on the terminal. The interface is designed to make it easy to check the contents of the materials. Once the materials have been checked, they are sent to the administrator. The input is the completed report materials, and the output is the checked materials and the results of their transmission.

[1008] (Application example 1)

[1009] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1010] In conventional call center operations, recording call content, summarizing cases, counting the number of cases, and creating reports are often done manually, resulting in inefficiency and the consumption of large amounts of human resources. Furthermore, manual data processing has a high error rate, making it difficult to quickly obtain accurate statistical data and reports. This makes it difficult for security services to respond quickly, leading to a decline in customer satisfaction. The present invention aims to solve these problems and improve the efficiency of call center operations and the accuracy of data.

[1011] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1012] In this invention, the server includes: means for receiving interactions with customers as voice data; means for converting the received voice data into text data using voice recognition technology; means for analyzing the converted text data using natural language processing technology and generating a summary of the cases; means for automatically counting the number of cases related to specific keywords or phrases and generating aggregated data; means for automatically creating report materials based on the generated aggregated data and distributing the generated report; means including technology for converting voice data into text in real time; and means for counting the number of summarized cases and visualizing the data based on that. This makes it possible to streamline call center operations, reduce the burden on staff, provide accurate data quickly, and improve the response speed and quality of security services.

[1013] "Customer" refers to general consumers, corporations, etc. who use the service.

[1014] "Interaction" refers to calls, conversations, and inquiries between customers and call center representatives.

[1015] "Voice data" refers to audio files and digital data that record the contents of conversations with customers.

[1016] "Means for receiving" refers to a device or a method for implementing a device for loading call center calls into a server.

[1017] "Speech recognition technology" refers to technology for converting voice data into text data.

[1018] "Text data" refers to data in the form of a string of characters converted using voice recognition technology.

[1019] "Natural language processing technology" refers to technology for analyzing text data and understanding its meaning and context.

[1020] "Means for generating a case summary" refers to a device or method that uses natural language processing technology to extract important information from text data and summarize it concisely.

[1021] "Specific keywords and phrases" refer to important words and expressions related to the subject or topic of your inquiry.

[1022] "Means for automatically counting the number of cases" refers to a method for automatically calculating within the system the number of cases that match a specific keyword or phrase.

[1023] "Aggregated data" refers to data that systematically compiles the number of counted cases and related information.

[1024] "Means for automatically creating report materials" refers to a device or method for automatically generating reports or presentation materials within a system based on aggregated data.

[1025] "Means for distributing reports" refers to devices and methods for transmitting generated report materials to interested parties.

[1026] "Technology for converting voice data into text in real time" refers to technology for converting voice data into text data in real time during a call.

[1027] "Means for visualizing data" refers to devices or methods that display aggregated data in a visually easy-to-understand format, such as graphs or tables.

[1028] The system required to realize this invention includes the following hardware and software: The main elements are speech recognition, natural language processing, data visualization, and automatic report generation technologies.

[1029] The server receives customer interactions as voice data. In this invention, calls at the call center are realized using a VoIP system. The voice data is sent to the server in real time and converted into text data using the Google Cloud Speech-to-Text API. The converted text data is analyzed on the server using the Python nltk (Natural Language Toolkit) library.

[1030] This is where analysis is performed using natural language processing techniques. First, the text data undergoes preprocessing, including tokenization, part-of-speech tagging, stemming, and lemmatization. Next, NLP models are applied to extract key phrases and information and generate a case summary, resulting in a summary that summarizes the key points of the call.

[1031] The server uses Elasticsearch (a full-text search engine) to count the number of jobs related to specific keywords or phrases. Administrators specify relevant keywords or phrases on the terminal and perform searches based on them. The search results are aggregated and compiled as statistical data. This statistical data is visualized as aggregate data by day, week, or month.

[1032] To visualize the aggregated data, the server generates graphs and tables using the Tableau (data visualization tool) API, which presents the data in a format that is easy to understand visually. Report materials containing the generated graphs and tables are then created using an automated report generation tool. The generated reports are then sent to the administrator via the internal email system API.

[1033] Examples of concrete examples and prompts

[1034] Example 1: Generating a report on phishing attacks

[1035] 1. Speech Recognition:

[1036] The server uses the Google Cloud Speech-to-Text API to convert the call audio into text data.

[1037] Example prompt: "I want to know about phishing attacks."

[1038] 2. Case summary generation:

[1039] The server analyzes the converted text data and generates a summary of the case using natural language processing technology.

[1040] Example prompt: "Summarize the details of the phishing attack contained in this text."

[1041] 3. Counting the number of cases:

[1042] The administrator enters keywords related to "phishing attacks" into the terminal, and the server uses Elasticsearch to count the number of cases.

[1043] Example prompt: "Count the number of incidents related to phishing attacks."

[1044] 4. Report material generation:

[1045] The server uses the Tableau API to visualize the data and create reporting materials.

[1046] Example prompt: "Graph the number of phishing attacks by month."

[1047] 5. Submit report:

[1048] The report is sent to the administrator via the internal email system.

[1049] Example prompt: "Please email this report to your administrator."

[1050] In this way, the present invention makes it possible to improve the efficiency of call center operations, reduce the burden on staff, and improve the response speed and quality of security services.

[1051] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1052] Step 1:

[1053] The server receives the call voice data from the call center. The input is real-time voice data from the VoIP system. The server sends this data to the Google Cloud Speech-to-Text API, which converts the voice data into text data. This converted text data is the output.

[1054] Step 2:

[1055] The server receives the converted text data and analyzes it using natural language processing techniques. The input is the text data obtained in step 1. Specifically, it uses the Python nltk library to perform preprocessing such as tokenization, part-of-speech tagging, stemming, and lemmatization. It then applies an NLP model to extract key phrases and information and generate a summary of the case. This summarized text is the output.

[1056] Step 3:

[1057] The administrator enters specific keywords or phrases into the terminal. The input is the keyword or phrase specified by the administrator and is used as a search condition. The server uses Elasticsearch to search the text database based on this keyword or phrase. The number of cases that match the entered conditions is counted and this becomes the output.

[1058] Step 4:

[1059] The server aggregates the search results and compiles them as statistical data. The input is the number of cases counted in step 3. The statistical data is compiled in an aggregated format by day, week, or month. This aggregated data is the output.

[1060] Step 5:

[1061] The server performs data visualization based on the aggregated data. The input is the aggregated data generated in step 4. The server uses the Tableau API to visualize the aggregated data as graphs and tables. This visualized data is the output.

[1062] Step 6:

[1063] The server automatically creates a report containing the visualized data. The input is the visualization data generated in step 5. An automatic report generation tool is used to generate a report. This report is the output.

[1064] Step 7:

[1065] The user (person in charge) checks the report generated on the terminal. The input is the report generated by the server. The user checks the contents and makes corrections if necessary. The checked and corrected report is the output.

[1066] Step 8:

[1067] The terminal sends a report to the administrator after confirmation is complete. The input is the report confirmed in step 7. The report is sent to the administrator via the internal mail system API or shared drive. This sent report becomes the final output.

[1068] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1069] This invention relates to a system for call center operations that converts customer interactions into text, generates case summaries, automatically counts the number of cases, and creates report materials while taking into account customer emotional information using an emotion engine. This system is composed of a combination of speech recognition technology, natural language processing technology, emotion recognition technology, data visualization technology, and automatic report generation technology.

[1070] Program processing natural language explanation

[1071] Step 1: Transcribe your interactions with customers

[1072] The server acquires the call voice data of the call center. The call voice is received in batches in real time from the VoIP system.

[1073] The server uses voice recognition technology to convert the received voice data into text data. The voice recognition engine utilizes a cloud-based API.

[1074] Step 2: Generate a case summary

[1075] The server analyzes the converted text data using a natural language processing toolkit, which includes tokenization, part-of-speech tagging, stemming, and lemmatization.

[1076] The server applies NLP models to extract key phrases and information from the text data, generating a summary of the case, including information such as the issue, request details, and response status.

[1077] Step 3: Recognizing and integrating emotional information

[1078] The server uses an emotion engine to recognize the customer's emotions during the call in real time, for example by analyzing the intonation, speed, and tone of the voice to identify the emotion.

[1079] The server adds the recognized emotional information to the case summary, which makes the importance and urgency of the case more clear.

[1080] Step 4: Counting and data collection of cases

[1081] The terminal receives instructions from an administrator and provides an interface for inputting specific keywords or phrases, such as "returned goods" or "delayed delivery."

[1082] The server uses the entered keywords to search a text database and identify relevant cases. High-speed search processing is performed using a full-text search engine.

[1083] The server aggregates the search results and counts the number of relevant cases. The counted data is compiled and organized as daily, weekly, and monthly statistical information.

[1084] Step 5: Integrate and analyze sentiment data

[1085] The server integrates the recognized emotion data into aggregate data and analyzes emotion trends, for example, analyzing the frequency of negative emotions when handling complaints.

[1086] Step 6: Prepare your report

[1087] The server uses a data visualization tool to generate graphs and tables that are easy to understand visually based on the aggregated data.

[1088] The server uses an automated report generation tool to create a report containing generated graphs and tables, including the emotion data.

[1089] Step 7: Final review and delivery

[1090] The user (person in charge) checks the report materials on the terminal, checking the content and format of the materials through the interface for any errors.

[1091] The device sends the verified report materials to the administrator, who then saves them in the internal email system or on a shared drive and notifies the appropriate administrator.

[1092] Specific examples

[1093] Example 1: Counting returns and recognizing emotions

[1094] 1. The server receives the call audio data related to "returns" and converts it into text.

[1095] 2. The server analyzes the converted text data and generates a summary of the case, including important information such as the reason for return, the date of the request, and the status of the response.

[1096] 3. The server uses an emotion engine to recognize the customer's emotions during the call and adds that information to the summary.

[1097] 4. The administrator enters keywords related to "returns" on the terminal and issues instructions to check the number of cases.

[1098] 5. The server searches for cases related to "returns" and counts the number of cases.

[1099] 6. The server aggregates the data based on the counting results, creates visual graphs and tables, and integrates the emotion data.

[1100] 7. The user checks the generated report materials on the terminal and sends them to the administrator.

[1101] Example 2: Counting delivery delay cases and recognizing emotions

[1102] 1. The server receives the call audio data related to "Delivery Delay" and converts it into text.

[1103] 2. The server analyzes the text data and generates a project summary.

[1104] 3. The server uses an emotion engine to recognize the customer's emotions during the call and adds that information to the summary.

[1105] 4. The administrator enters keywords related to "delivery delay" on the terminal and issues an instruction to check the number of cases.

[1106] 5. The server searches for the number of cases related to "Delivery Delay" and aggregates the results.

[1107] 6. The server creates graphs and tables based on the aggregated data and generates report materials including emotion data.

[1108] 7. The user checks the materials and sends them to the administrator.

[1109] The introduction of this system will significantly improve the efficiency of call center operations, reduce the burden on staff, and provide more accurate and useful data that takes into account customer emotional information.

[1110] The processing flow will be explained below.

[1111] Step 1:

[1112] The server acquires the call center's voice data, which is received in batches in real time from the VoIP system.

[1113] Step 2:

[1114] The server converts the acquired voice data into text data using speech recognition technology. The speech recognition engine uses a cloud-based API, and the converted text data is temporarily stored.

[1115] Step 3:

[1116] The server analyzes the stored text data using a natural language processing toolkit (e.g., spaCy or NLTK), which includes text preprocessing (tokenization, part-of-speech tagging, stemming, and lemmatization).

[1117] Step 4:

[1118] The server applies NLP models (e.g., BERT or GPT-3) to extract key phrases and information from the text data, summarizing information such as the problem, request details, and response status, and generating a case summary in a standardized format.

[1119] Step 5:

[1120] The server uses an emotion engine to recognize the customer's emotions in real time during a call. It analyzes the intonation, speed, and tone of the voice to identify emotions. For example, it determines whether the customer is angry or satisfied.

[1121] Step 6:

[1122] The server adds the recognized emotion information to the case summary, which more clearly indicates the importance and urgency of the case. For example, an emotion like "anger" will draw special attention.

[1123] Step 7:

[1124] The terminal provides an interface for administrators to input specific keywords or phrases of interest, such as "returned goods" or "delayed delivery."

[1125] Step 8:

[1126] The server searches a text database using keywords entered from the terminal to identify relevant cases. Using a full-text search engine (e.g., Elasticsearch) enables high-speed search processing.

[1127] Step 9:

[1128] The server aggregates the search results and counts the number of matching cases. The counted data is compiled and organized as daily, weekly, and monthly statistical information.

[1129] Step 10:

[1130] The server then aggregates the recognized emotion data and analyzes the trend of emotions, for example, analyzing the frequency and tendency of negative emotions in complaint handling.

[1131] Step 11:

[1132] The server uses a data visualization tool (e.g., Matplotlib or Tableau) to generate visually easy-to-understand graphs and tables based on the aggregated data.

[1133] Step 12:

[1134] The server uses automated report generation tools (e.g., LaTeX or Pandas DataFrame) to create a report containing generated graphs and tables, including the sentiment data.

[1135] Step 13:

[1136] The user (person in charge) checks the report materials on the terminal, checking the content and format of the materials through the interface for any errors.

[1137] Step 14:

[1138] The device sends the verified report materials to the administrator, who then saves them in the internal email system or on a shared drive and notifies the appropriate administrator.

[1139] Example 2

[1140] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1141] In call center operations, traditional methods require manual recording and analysis of call content, which is labor-intensive and time-consuming, making it difficult to provide efficient customer service. Additionally, the lack of data that reflects customer sentiment makes it difficult to develop specific measures to improve customer satisfaction and the quality of customer complaint handling.

[1142] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1143] In this invention, the server includes means for receiving interactions with customers as voice data, means for converting the received voice data into text data using voice recognition technology, means for analyzing the converted text data using natural language processing technology and generating a case summary, means for recognizing customer emotion information from the analyzed text data and integrating it into the case summary, means for automatically counting the number of cases related to specific keywords or phrases and generating aggregated data, and means for automatically creating reporting materials based on the generated aggregated data. This enables efficient recording and analysis of call content and enables data collection and reporting that takes customer emotion into consideration.

[1144] "Audio data" refers to data that has been recorded and stored in digital format.

[1145] "Speech recognition technology" is a technology in which a computer analyzes voice data and converts the content into text data.

[1146] "Text data" refers to text information converted using voice recognition technology.

[1147] "Natural language processing technology" refers to the technology that allows computers to understand and analyze human language.

[1148] A "case summary" is a summary of important information extracted from text data analyzed using natural language processing technology.

[1149] "Emotional information" refers to information that indicates the emotional state of the customer during a call.

[1150] "Specific keywords or phrases" are words or expressions that an administrator specifies they want to check.

[1151] "Job counting" refers to the process of automatically counting the number of jobs related to a particular keyword or phrase.

[1152] "Aggregated data" refers to data that compiles information such as the number of cases and emotional information.

[1153] "Reporting materials" refers to report-style materials that include graphs and tables visualized based on aggregated data.

[1154] MODE FOR CARRYING OUT THE INVENTION

[1155] This invention relates to a system that converts customer interactions into text in call center operations, organizes case summaries, and automatically creates reports that take emotional information into account. This system combines speech recognition technology, natural language processing technology, emotion recognition technology, and data visualization technology.

[1156] This system uses the following main hardware and software:

[1157] Server: Receives customer voice data from the VoIP system in real time and converts it into text using a speech recognition engine. For speech recognition, it utilizes a general cloud service API.

[1158] Natural language processing tools: Analyze text data, extract key information, and generate case summaries using techniques such as tokenization, part-of-speech tagging, stemming, and lemmatization, specifically using NLTK and BERT-based language models.

[1159] Sentiment engine: Recognizes customer sentiment from voice and text data and integrates it into case summaries. This includes sentiment analysis tools such as IBM Watson.

[1160] Database system: Stores and manages text data, case summaries, emotional information, etc.

[1161] Data visualization tools: Use tools such as Tableau to visualize the collected data and create reporting materials.

[1162] Automatic report generation tool: Uses Apache FOP to generate reports in PDF format based on various data.

[1163] For example, when a customer calls a call center regarding a return, the following process is performed:

[1164] 1. The server retrieves the call audio data related to the return from the VoIP system.

[1165] 2. The server calls a common cloud service API to convert the voice data into text and store it in a database.

[1166] 3. The server performs preprocessing using NLTK, and then uses a BERT-based model to extract important information such as the reason for return, request date, and response status, and generates a summary of the case.

[1167] 4. The server uses the IBM Watson emotion engine to recognize the customer's emotion and add it to the case summary.

[1168] 5. The terminal provides an interface for the administrator to enter keywords such as "return." The server uses Elasticsearch to search for matching cases and count the number of cases.

[1169] 6. The server uses Tableau to visually transform the aggregated data into graphs and tables, and uses Apache FOP to generate reporting materials.

[1170] 7. The user (person in charge) checks the report materials on the terminal and sends them to the administrator via the internal email system or shared drive.

[1171] Examples of prompts include:

[1172] "Analyze call data related to returns, generate case summaries, and add sentiment information. Include the specific reason for the return and the date of the request."

[1173] "Analyze trends in customer sentiment by compiling performance data on delivery delays. Create visual graphs and tables and generate reports."

[1174] As described above, the introduction of this system will improve the efficiency of call center operations and data accuracy.

[1175] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1176] Step 1:

[1177] The server receives real-time call audio data from the call center's VoIP system. The audio data is input, and an audio file is generated as output. Specifically, it establishes a connection with the VoIP system and digitally records the call audio.

[1178] Step 2:

[1179] The server sends the acquired voice data to the Google Cloud Speech-to-Text API and converts the voice data into text data. The input is voice data, and the output is the converted text data. Specifically, the server sends voice data to the API endpoint via an HTTP request and receives the converted text data as an HTTP response.

[1180] Step 3:

[1181] The server preprocesses the converted text data using NLTK. Preprocessing includes tokenization, part-of-speech tagging, stemming, and lemmatization. The input is text data, and the output is preprocessed text data. Specifically, the server breaks the text data down into words, tags each word with a part of speech, and converts it to its root form.

[1182] Step 4:

[1183] The server inputs the preprocessed text data into a BERT-based NLP model to extract important information. The input is the preprocessed text data, and the output is the extracted important information (such as "problem," "request content," and "response status"). Specifically, the server converts the text data into vectors, inputs them into the BERT model, and extracts important phrases that the model predicts.

[1184] Step 5:

[1185] The server generates a summary of the case based on the extracted important information and saves it in the database. The input is the important information and the output is the summary of the case. Specifically, the server applies the important information to a template, generates a summary, and saves it in the database.

[1186] Step 6:

[1187] The server sends the customer's emotional information from the voice and text data to the IBM Watson Tone Analyzer to recognize emotions. The input is voice and text data, and the output is emotional information. Specifically, it extracts features from the voice data and sends them along with the text data to the emotion analysis API to obtain emotional information.

[1188] Step 7:

[1189] The server integrates the acquired emotion information into the case summary and saves the updated summary in the database. The input is the emotion information and the existing case summary, and the output is the updated case summary. Specifically, the emotion information is added to the existing summary and re-saved as a new summary in the database.

[1190] Step 8:

[1191] The terminal provides an interface for the administrator to input specific keywords or phrases (e.g., "returned goods" or "delayed delivery"). The input is the administrator's keywords, and the output is the search criteria. Specifically, it provides a web form and retrieves the input keywords.

[1192] Step 9:

[1193] The server uses Elasticsearch to search a text database for the entered keywords and identify matching cases. The input is the search criteria, and the output is a list of the searched cases. Specifically, it generates and executes an Elasticsearch query to retrieve the search results.

[1194] Step 10:

[1195] The server aggregates the search results and counts the number of cases by day, week, month, etc. The input is the search results and the output is the aggregated data. Specifically, the server aggregates the search results, counts the cases for each specified period, and performs the aggregation.

[1196] Step 11:

[1197] The server then uses Tableau to visually transform the aggregated data into graphs and tables. The input is the aggregated data, and the output is the visualized graph or table. Specifically, the server loads the data into Tableau, selects the appropriate visualization, and generates it.

[1198] Step 12:

[1199] The server uses Apache FOP to automatically generate visualized data as report materials. The input is visualized graphs and tables, and the output is report materials (PDF format). Specifically, the data generated in XML format is input into Apache FOP, and a PDF file is generated.

[1200] Step 13:

[1201] The user (person in charge) checks the report materials on the terminal. The input is the report materials, and the output is the confirmation results. Specifically, the user opens the materials using a PDF viewer and checks for any errors in the content or format.

[1202] Step 14:

[1203] The terminal saves the confirmed report materials in the internal email system or shared drive and notifies the appropriate administrator. The input is the confirmed report materials, and the output is a notification to the administrator. Specifically, the terminal saves the materials in a specified folder and contacts the administrator via email or the notification system.

[1204] (Application example 2)

[1205] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1206] In customer service operations in brick-and-mortar stores, it is important to efficiently collect and analyze customer requests and feedback, and then respond appropriately based on the results. However, conventional methods have made it difficult to accurately record customer comments and analyze them in a timely manner, and there are also limited means of recognizing and reflecting customer emotions in real time. This has made it difficult to improve customer satisfaction and has increased the burden on staff.

[1207] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving interactions with customers as voice data, means for converting the received voice data into text data using voice recognition technology, means for analyzing the converted text data using natural language processing technology and generating a summary of the cases, means for automatically counting the number of cases related to specific keywords or phrases and generating aggregated data, means for adding recognized emotion data to the summary, and means for automatically creating reporting materials based on the generated aggregated data. This makes it possible to efficiently collect and analyze customer requests and feedback during customer service operations in physical stores and reflect the emotional information in real time.

[1208] "Interaction with customers" refers to communication such as dialogue and information exchange between service providers and customers.

[1209] "Audio data" means data for recording, storing, and processing audio information in digital format.

[1210] "Means of receiving" refers to the technical means or devices for acquiring data or information from the outside and incorporating it into the internal environment.

[1211] "Speech recognition technology" is a technology for analyzing voice data and converting it into text data.

[1212] "Text data" is data that represents character information in digital form.

[1213] "Natural language processing technology" is an artificial intelligence technology for analyzing, understanding, and generating human language.

[1214] A "case summary" is a document or data that succinctly summarizes important information or key points about a particular case.

[1215] "Specific keywords and phrases" refer to important words or short sentences that indicate specific information or content.

[1216] "Means for automatically counting the number of cases" refers to a device or program that has the function of calculating the number of cases based on specific conditions.

[1217] "Aggregated data" is data that shows overall trends and characteristics by organizing and statistically processing multiple data.

[1218] "Recognized emotion data" refers to information about an emotional state analyzed from data such as voice or text.

[1219] "Means for adding to the summary" refers to technical means for integrating additional information into the generated summary text.

[1220] "Reporting materials" refers to documents, graphs, charts, etc. created to report specific information.

[1221] "Automated means" refers to technology or devices that perform specific tasks automatically without human intervention.

[1222] A "data visualization tool" is software that converts data into a format that is easy to understand visually, such as a graph or table.

[1223] "Smart glasses" are wearable smart devices equipped with a display, sensors, communication functions, etc.

[1224] "Customer service" refers to all work related to providing products and services to customers and ensuring their satisfaction.

[1225] The present invention provides a system for improving the efficiency of customer service operations in brick-and-mortar stores and enhancing customer satisfaction. An embodiment of this system will be described in detail below.

[1226] System Configuration

[1227] This system mainly consists of four components: "smart glasses," "server," "terminal," and "user."

[1228] 1. Smart Glasses

[1229] Smart glasses are wearable devices equipped with a display, microphone, sensors, and communication functions. They are worn by store staff while serving customers and are used to capture conversations with customers. They convert speech into text in real time and perform sentiment analysis.

[1230] 2. Server

[1231] The server performs the main processes of receiving voice data, converting it into text, analyzing it, recognizing emotions, and compiling it. It uses the following software and technologies:

[1232] Speech recognition engine: Google Cloud Speech-to-Text API

[1233] Natural language processing tool: SpaCy

[1234] Emotion recognition engine: IBM Watson Tone Analyzer

[1235] Data visualization tool: Tableau

[1236] Automated Report Generation Tool: Google Docs API

[1237] 3. Terminal

[1238] Terminals are devices used by store managers and staff to check generated reporting materials, enter data, give instructions, etc. Data can be checked, corrected, and sent through an interface.

[1239] Specific examples of implementation

[1240] Step 1: Capture and transcribe customer interactions

[1241] The server receives voice data from the smart glasses in real time and converts it into text data using the Google Cloud Speech-to-Text API.

[1242] Step 2: Analyze requirements and generate a summary

[1243] The server then uses SpaCy to analyze the converted text data and generate a summary of the case. For example, if a customer asks, "Is this product in stock?", the server generates the summary "Product inventory inquiry."

[1244] Step 3: Recognizing and integrating emotional data

[1245] The server uses IBM Watson Tone Analyzer to analyze customer sentiment from voice and text data. For example, if a customer expresses impatience, it adds that information to the summary.

[1246] Step 4: Generate and visualize aggregate data

[1247] The server automatically counts the number of jobs related to specific keywords or phrases and generates aggregate data, which is then visualized in graphs and tables using Tableau.

[1248] Step 5: Prepare, review and distribute report materials

[1249] The server uses the Google Docs API to automatically create reporting materials based on the generated aggregated data. The user (store manager) can check the materials on their device, make any necessary corrections, and then send the materials via email or cloud drive.

[1250] Prompt sentence for specific implementation example

[1251] For example, when a system captures a customer question such as "Do you have this product in stock?" and performs response and sentiment analysis, the following prompt sentences are used:

[1252] "Please give us your feedback on the product."

[1253] "Do you have any questions about this product?"

[1254] Please let us know your thoughts on the service.

[1255] This will enable efficient collection of customer requests and feedback during customer service operations in brick-and-mortar stores, enabling detailed analysis based on emotional data, and providing useful information to improve staff efficiency and increase customer satisfaction.

[1256] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1257] Step 1:

[1258] The server receives voice data from the smart glasses in real time. The voice data is captured through the microphone of the smart glasses and sent to the server through the VoIP system. The input is voice data, which is analyzed by the voice recognition engine.

[1259] Step 2:

[1260] The server uses the Google Cloud Speech-to-Text API to convert the received voice data into text data. The input is voice data, and the output is text data that expresses the contents of the voice data in characters. The server analyzes and converts the voice data.

[1261] Step 3:

[1262] The server uses SpaCy to perform natural language processing on the converted text data. The input is text data, which undergoes processes such as tokenization, part-of-speech tagging, stemming, and lemmatization. The output is each element of the analyzed text data and a summary of the case.

[1263] Step 4:

[1264] The server uses IBM Watson Tone Analyzer to recognize emotions from the analyzed text and voice data. The input is the analyzed text and voice data, and the output is the recognized emotion data. Specifically, it detects the emotional state (e.g., impatience, anger, joy) from the tone and rate of the customer's voice.

[1265] Step 5:

[1266] The server integrates the emotion data into a case summary. The input is the summary data and emotion data, and the output is a case summary including emotion information. This makes the importance and urgency of the case clear.

[1267] Step 6:

[1268] The terminal provides an interface that accepts specific keywords or phrases entered by the administrator. The input is a keyword (e.g., "satisfied," "dissatisfied," "returned product"), and the number of cases is counted based on this.

[1269] Step 7:

[1270] The server searches the database using the entered keywords and counts the number of matching cases. The input is the keywords and text database, and the output is the number of matching cases. The search is performed quickly using a full-text search engine.

[1271] Step 8:

[1272] The server aggregates the number of cases and emotion data counted and visualizes the data using Tableau. The input is the number of cases and emotion data, and the output is easy-to-understand graphs and tables.

[1273] Step 9:

[1274] The server automatically creates report materials using the Google Docs API. The input is the aggregated data and visualization data, and the output is the report document. The server automatically generates the report and creates the document in the appropriate format.

[1275] Step 10:

[1276] Users can check and modify report materials on their terminals. The input is the automatically generated report material, and the output is the final, verified report material. The user checks the content and format of the material to ensure there are no errors.

[1277] Step 11:

[1278] The terminal sends the verified report materials to the administrator. They are saved by email or cloud drive and notified to the administrator. The input is the verified report materials, and the output is the report materials that can be accessed by the administrator. This ensures that the final report is distributed appropriately.

[1279] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1280] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1281] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1282] [Fourth embodiment]

[1283] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1284] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1285] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1286] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1287] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1288] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1289] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1290] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1291] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1292] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1293] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1294] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1295] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1296] This invention relates to a system for call center operations that converts customer interactions into text, generates case summaries, automatically counts the number of cases, and creates report materials. This system is composed of a combination of speech recognition technology, natural language processing technology, data visualization technology, and automatic report generation technology.

[1297] Program processing natural language explanation

[1298] Step 1: Transcribe your interactions with customers

[1299] The server receives the call voice data from the call center. The call voice is received from the VoIP system and can be processed in real time or in batches.

[1300] The server converts the received voice data into text using speech recognition technology, with the speech recognition engine utilizing a widely available cloud-based API.

[1301] Step 2: Generate a case summary

[1302] The server analyzes the converted text data using a natural language processing toolkit, which performs preprocessing of the text (tokenization, part-of-speech tagging, stemming, and lemmatization).

[1303] The server applies NLP models to extract key phrases and information, thereby organizing the case's main topics and issues and generating a case summary in a standardized format.

[1304] Step 3: Counting and data collection of cases

[1305] The terminals provide an interface for inputting specific keywords and phrases, in response to instructions from the administrator, who is interested in various topics that arise in the call center.

[1306] The server uses the entered keywords to search the text database, identify and count relevant cases. The use of a full-text search engine enables high-speed search processing.

[1307] The server aggregates the search results and compiles statistics on the number of cases, generating daily, weekly, and monthly aggregate data.

[1308] Step 4: Prepare your report

[1309] The server uses a data visualization tool to generate graphs and tables using the aggregated data, allowing the data to be presented in a visually understandable format.

[1310] The server uses an automated report generation tool to create reporting materials, including generated graphs and tables, which can be output as presentations, PDFs, or other formats.

[1311] Step 5: Final review and delivery

[1312] The user (person in charge) checks the created report materials on the terminal. The interface is designed to make it easy to check the contents of the materials.

[1313] The device sends the confirmed report materials to the administrator via the internal email system or saved on a shared drive.

[1314] Specific examples

[1315] Example 1: Counting returns

[1316] 1. The server receives the call audio data related to "returns" and converts it into text.

[1317] 2. The server analyzes the converted text data and generates a summary of the case, including important information such as the reason for return, the date of the request, and the status of the response.

[1318] 3. The administrator enters keywords related to "returns" on the terminal and issues an instruction to check the number of cases.

[1319] 4. The server searches for cases related to "returns" and counts the number of cases.

[1320] 5. The server aggregates the data based on the counting results and creates visual graphs and tables.

[1321] 6. The user checks the generated report materials on the terminal and sends them to the administrator.

[1322] Example 2: Counting cases related to delivery delays

[1323] 1. The server receives the call audio data related to "Delivery Delay" and converts it into text.

[1324] 2. The server analyzes the text data and generates a project summary.

[1325] 3. The administrator enters keywords related to "delivery delay" on the terminal and issues an instruction to check the number of cases.

[1326] 4. The server searches for the number of cases related to "Delivery Delay" and aggregates the results.

[1327] 5. The server creates graphs and tables based on the aggregated data and generates reporting materials.

[1328] 6. The user checks the materials and sends them to the administrator.

[1329] The introduction of this system will significantly improve the efficiency of call center operations, reduce the burden on staff, and enable the provision of accurate data.

[1330] The processing flow will be explained below.

[1331] Step 1:

[1332] The server acquires the call center's voice data, which is received in batches in real time from the VoIP system.

[1333] Step 2:

[1334] The server converts the acquired voice data into text data using speech recognition technology. The speech recognition engine uses a cloud-based API, and the converted text data is temporarily stored.

[1335] Step 3:

[1336] The server analyzes the stored text data using a natural language processing toolkit (e.g., spaCy or NLTK), which includes tokenization, part-of-speech tagging, stemming, and lemmatization.

[1337] Step 4:

[1338] The server applies an NLP model (e.g., BERT or GPT-3) to extract key phrases and information from the text data, generating a summary of the job, including information such as the problem, request, and status.

[1339] Step 5:

[1340] The terminal receives instructions from an administrator and provides an interface for inputting specific keywords or phrases, such as "returned goods" or "delayed delivery."

[1341] Step 6:

[1342] The server searches a text database using keywords entered from the terminal to identify relevant cases. High-speed search processing is performed using a full-text search engine (e.g., Elasticsearch).

[1343] Step 7:

[1344] The server aggregates the search results and counts the number of relevant cases. The counted data is compiled and organized as daily, weekly, and monthly statistical information.

[1345] Step 8:

[1346] The server uses a data visualization tool (e.g., Matplotlib or Tableau) to generate graphs and tables that are easy to understand visually based on the aggregated data.

[1347] Step 9:

[1348] The server uses automated report generation tools (e.g., LaTeX or Pandas DataFrames) to create reports, including generated graphs and tables.

[1349] Step 10:

[1350] The user (person in charge) checks the report materials on the terminal, checking the content and format of the materials through the interface for any errors.

[1351] Step 11:

[1352] The device sends the verified report materials to the administrator, who then saves them in the internal email system or on a shared drive and notifies the appropriate administrator.

[1353] Example 1

[1354] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1355] Call center operations require efficient management of a large volume of customer interactions, as well as the ability to quickly and accurately summarize cases and generate statistical data. However, conventional systems did not integrate voice recognition, natural language processing, data visualization, or automatic report generation, which placed a heavy burden on staff and posed challenges to the accuracy and efficiency of processing.

[1356] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1357] In this invention, the server includes means for receiving interactions with customers as voice data, means for converting the received voice data into text data using voice recognition technology, means for analyzing the converted text data using natural language processing technology and generating case summaries, means for automatically counting the number of cases related to specific keywords or phrases and generating aggregated data, means for automatically creating report materials based on the generated aggregated data, means for statistically compiling the aggregated data in daily, weekly, and monthly formats, means for supporting real-time and batch processing of voice data, and means for performing tokenization, part-of-speech tagging, stemming, and lemmatization during the analysis process. This makes it possible to improve the efficiency of call center operations, reduce the burden on staff, and provide accurate data.

[1358] "Voice Data" means digital audio files or live audio streams recorded during customer interactions in a call center.

[1359] "Speech recognition technology" is a technology for analyzing voice data and converting it into text data.

[1360] "Text data" refers to textual information converted using voice recognition technology.

[1361] "Natural language processing technology" is a technology for analyzing text data, understanding its content, and extracting specific information.

[1362] The "Case Summary" is the result of organizing key information and important phrases extracted using natural language processing technology.

[1363] "Keywords" are key words or phrases used to identify a particular subject.

[1364] "Aggregated data" refers to statistically compiled data on the number of jobs related to a particular keyword.

[1365] A "data visualization tool" is software that displays aggregated data in a visually easy-to-understand format, such as a graph or table.

[1366] "Reporting materials" are documents that are created based on aggregated data and contain project summaries and statistical data.

[1367] "Automated report generation tool" means software or a tool for automatically generating reports based on data.

[1368] The "daily, weekly, and monthly format" is a format in which data is classified and summarized by day, week, or month.

[1369] "Real-time processing" refers to the process of processing audio data and producing results in real time.

[1370] "Batch processing" is a process of processing a large amount of audio data at once at regular time intervals.

[1371] "Tokenization" is the process of dividing text data into words and phrases.

[1372] "Part-of-speech tagging" is the process of assigning part-of-speech labels to segmented words and phrases.

[1373] "Stemming" is the process of extracting the roots of words.

[1374] "Lemmatization" is the process of extracting the basic forms of words.

[1375] This invention is a system for call center operations that receives customer interactions as voice data, converts it into text, generates case summaries and aggregated data, and automatically creates report materials. This system is realized by combining the following hardware and software.

[1376] Hardware and software used

[1377] 1. The server is a device that receives call audio data through the call center's VoIP system. The server is equipped with a high-speed speech recognition engine (e.g., Google Cloud Speech-to-Text API or Amazon Transcribe).

[1378] 2. The server further has the function of converting the received voice data into text data in real time or batch processing using voice recognition technology.

[1379] 3. The server uses natural language processing techniques (e.g., spaCy, NLTK, BERT model) to extract key phrases and information from the converted text data, thereby organizing the main topics and issues of the case and generating a summary of the case in a standardized format.

[1380] 4. A terminal is a device that provides an interface for administrators and users to search for specific keywords or phrases. This interface may be in the form of a web form, where administrators input instructions about topics of interest.

[1381] 5. The server uses a full-text search engine such as Elasticsearch to search the text database based on the entered keywords, identifies relevant cases, and counts the number of cases. The search results are then statistically compiled by day, week, and month.

[1382] 6. The server uses a data visualization tool (e.g., Tableau, D3.js) to generate graphs and tables from the aggregated data, allowing the data to be presented in a visually understandable format.

[1383] 7. The server uses automated report generation tools (e.g., LaTeX, Google Docs API) to create a report containing the generated graphs and tables, which can be output as a presentation, PDF, or other format.

[1384] 8. The user checks the created report materials, confirms that the contents are accurate, and then sends them to the administrator. The interface on the terminal is designed to make it easy to check the contents of the materials.

[1385] Specific examples

[1386] Counting returns cases

[1387] 1. The server receives the call audio data related to "returns" from the VoIP system and converts it to text using the Google Cloud Speech-to-Text API. When converting the audio data to text, background noise is removed and the speaker is identified.

[1388] 2. The server analyzes the converted text data using spaCy, extracts important information such as the reason for return, request date, and response status, and generates a case summary.

[1389] 3. The administrator enters keywords related to "returns" on the terminal and issues an instruction to check the number of cases.

[1390] 4. The server uses Elasticsearch to search for cases related to "returns," lists the results, and performs further aggregation. The calculated data is saved in a database.

[1391] 5. The server uses Tableau to visualize the data and generate visual graphs and tables designed to show at a glance the increase or decrease in the number of returns.

[1392] 6. The user checks the generated report materials on the terminal and verifies that there are no problems with the contents. After checking, the materials are sent to the administrator.

[1393] Counting cases related to delivery delays

[1394] 1. The server receives the call audio data related to the "Delivery Delay" from the VoIP system and converts it to text using Amazon Transcribe. During the conversion process, the audio file format and sampling rate are adjusted.

[1395] 2. The server uses the BERT model to analyze the text data and extract important information such as the cause of the delay, the date and time of occurrence, and the response status.

[1396] 3. The administrator enters keywords related to "delivery delay" on the terminal and issues an instruction to check the number of cases.

[1397] 4. The server uses Elasticsearch to search for cases related to "Delivery Delays" and aggregates the results. The search results are then stored in an SQL database.

[1398] 5. The server uses D3.js to create graphs and tables based on the aggregated results and generate report materials. These materials are often output in PDF format and shared with all project stakeholders.

[1399] 6. The user checks the document and sends it to the administrator. The check is done on the screen, and the interface makes it easy to review and correct the document.

[1400] Examples of prompt statements

[1401] "Please transcribe the conversations about returns over the past week, tally the number of cases, and create a graph to generate a report."

[1402] "Count the number of inquiries about delivery delays and create a weekly report with monthly tallies and summaries."

[1403] The introduction of this system will significantly improve the efficiency of call center operations, ensuring the accuracy of information and reducing the burden on staff.

[1404] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1405] Step 1:

[1406] The server receives the call voice data through the call center's VoIP system. At this time, the server acquires the voice data in a format suitable for receiving the voice data in real time or batch processing. The input is the voice data from the VoIP system, and the output is the raw voice data stored in the server.

[1407] Step 2:

[1408] The server converts the received voice data into text data using the Google Cloud Speech-to-Text API. This process involves cleaning the voice data (noise removal and speaker identification). The input is voice data, and the output is text data.

[1409] Step 3:

[1410] The server analyzes the converted text data using a natural language processing toolkit (e.g., spaCy, NLTK). This analysis process involves tokenization, part-of-speech tagging, stemming, and lemmatization. The input is text data, and the output is preprocessed text data.

[1411] Step 4:

[1412] The server extracts important phrases and information from the preprocessed text data using the BERT model or GPT-3, and generates a summary of the case. The input is the preprocessed text data, and the output is the summary data of the case.

[1413] Step 5:

[1414] The terminal provides an interface for the administrator to input specific keywords or phrases. The administrator inputs topics of interest, such as "returns" or "delayed delivery," which in turn sends a search command to the server. The input is the administrator's keywords, and the output is a search request to the server.

[1415] Step 6:

[1416] The server uses Elasticsearch to search the text database based on keywords entered by the administrator. The search process quickly matches the text data with the target keywords to identify and count related jobs. The input is the keywords and the text database, and the output is a list of related jobs and their counts.

[1417] Step 7:

[1418] The server statistically compiles the list of cases and the counts it receives by day, week, and month. The results are saved in a database. The input is the list of cases and the counts, and the output is the statistically compiled aggregated data.

[1419] Step 8:

[1420] The server uses a data visualization tool (e.g., Tableau, D3.js) to generate graphs and tables based on the aggregated data. This allows the data to be displayed in a visually understandable format. The input is statistically compiled aggregated data, and the output is visual data such as graphs and tables.

[1421] Step 9:

[1422] The server uses automated report generation tools (e.g., LaTeX, Google Docs API) to create a report containing the generated visual data. The report is output in presentation or PDF format. The input is visual data such as graphs and tables, and the output is the completed report.

[1423] Step 10:

[1424] The user checks the report materials created on the terminal. The interface is designed to make it easy to check the contents of the materials. Once the materials have been checked, they are sent to the administrator. The input is the completed report materials, and the output is the checked materials and the results of their transmission.

[1425] (Application example 1)

[1426] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1427] In conventional call center operations, recording call content, summarizing cases, counting the number of cases, and creating reports are often done manually, resulting in inefficiency and the consumption of large amounts of human resources. Furthermore, manual data processing has a high error rate, making it difficult to quickly obtain accurate statistical data and reports. This makes it difficult for security services to respond quickly, leading to a decline in customer satisfaction. The present invention aims to solve these problems and improve the efficiency of call center operations and the accuracy of data.

[1428] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1429] In this invention, the server includes: means for receiving interactions with customers as voice data; means for converting the received voice data into text data using voice recognition technology; means for analyzing the converted text data using natural language processing technology and generating a summary of the cases; means for automatically counting the number of cases related to specific keywords or phrases and generating aggregated data; means for automatically creating report materials based on the generated aggregated data and distributing the generated report; means including technology for converting voice data into text in real time; and means for counting the number of summarized cases and visualizing the data based on that. This makes it possible to streamline call center operations, reduce the burden on staff, provide accurate data quickly, and improve the response speed and quality of security services.

[1430] "Customer" refers to general consumers, corporations, etc. who use the service.

[1431] "Interaction" refers to calls, conversations, and inquiries between customers and call center representatives.

[1432] "Voice data" refers to audio files and digital data that record the contents of conversations with customers.

[1433] "Means for receiving" refers to a device or a method for implementing a device for loading call center calls into a server.

[1434] "Speech recognition technology" refers to technology for converting voice data into text data.

[1435] "Text data" refers to data in the form of a string of characters converted using voice recognition technology.

[1436] "Natural language processing technology" refers to technology for analyzing text data and understanding its meaning and context.

[1437] "Means for generating a case summary" refers to a device or method that uses natural language processing technology to extract important information from text data and summarize it concisely.

[1438] "Specific keywords and phrases" refer to important words and expressions related to the subject or topic of your inquiry.

[1439] "Means for automatically counting the number of cases" refers to a method for automatically calculating within the system the number of cases that match a specific keyword or phrase.

[1440] "Aggregated data" refers to data that systematically compiles the number of counted cases and related information.

[1441] "Means for automatically creating report materials" refers to a device or method for automatically generating reports or presentation materials within a system based on aggregated data.

[1442] "Means for distributing reports" refers to devices and methods for transmitting generated report materials to interested parties.

[1443] "Technology for converting voice data into text in real time" refers to technology for converting voice data into text data in real time during a call.

[1444] "Means for visualizing data" refers to devices or methods that display aggregated data in a visually easy-to-understand format, such as graphs or tables.

[1445] The system required to realize this invention includes the following hardware and software: The main elements are speech recognition, natural language processing, data visualization, and automatic report generation technologies.

[1446] The server receives customer interactions as voice data. In this invention, calls at the call center are realized using a VoIP system. The voice data is sent to the server in real time and converted into text data using the Google Cloud Speech-to-Text API. The converted text data is analyzed on the server using the Python nltk (Natural Language Toolkit) library.

[1447] This is where analysis is performed using natural language processing techniques. First, the text data undergoes preprocessing, including tokenization, part-of-speech tagging, stemming, and lemmatization. Next, NLP models are applied to extract key phrases and information and generate a case summary, resulting in a summary that summarizes the key points of the call.

[1448] The server uses Elasticsearch (a full-text search engine) to count the number of jobs related to specific keywords or phrases. Administrators specify relevant keywords or phrases on the terminal and perform searches based on them. The search results are aggregated and compiled as statistical data. This statistical data is visualized as aggregate data by day, week, or month.

[1449] To visualize the aggregated data, the server generates graphs and tables using the Tableau (data visualization tool) API, which presents the data in a format that is easy to understand visually. Report materials containing the generated graphs and tables are then created using an automated report generation tool. The generated reports are then sent to the administrator via the internal email system API.

[1450] Examples of concrete examples and prompts

[1451] Example 1: Generating a report on phishing attacks

[1452] 1. Speech Recognition:

[1453] The server uses the Google Cloud Speech-to-Text API to convert the call audio into text data.

[1454] Example prompt: "I want to know about phishing attacks."

[1455] 2. Case summary generation:

[1456] The server analyzes the converted text data and generates a summary of the case using natural language processing technology.

[1457] Example prompt: "Summarize the details of the phishing attack contained in this text."

[1458] 3. Counting the number of cases:

[1459] The administrator enters keywords related to "phishing attacks" into the terminal, and the server uses Elasticsearch to count the number of cases.

[1460] Example prompt: "Count the number of incidents related to phishing attacks."

[1461] 4. Report material generation:

[1462] The server uses the Tableau API to visualize the data and create reporting materials.

[1463] Example prompt: "Graph the number of phishing attacks by month."

[1464] 5. Submit report:

[1465] The report is sent to the administrator via the internal email system.

[1466] Example prompt: "Please email this report to your administrator."

[1467] In this way, the present invention makes it possible to improve the efficiency of call center operations, reduce the burden on staff, and improve the response speed and quality of security services.

[1468] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1469] Step 1:

[1470] The server receives the call voice data from the call center. The input is real-time voice data from the VoIP system. The server sends this data to the Google Cloud Speech-to-Text API, which converts the voice data into text data. This converted text data is the output.

[1471] Step 2:

[1472] The server receives the converted text data and analyzes it using natural language processing techniques. The input is the text data obtained in step 1. Specifically, it uses the Python nltk library to perform preprocessing such as tokenization, part-of-speech tagging, stemming, and lemmatization. It then applies an NLP model to extract key phrases and information and generate a summary of the case. This summarized text is the output.

[1473] Step 3:

[1474] The administrator enters specific keywords or phrases into the terminal. The input is the keyword or phrase specified by the administrator and is used as a search condition. The server uses Elasticsearch to search the text database based on this keyword or phrase. The number of cases that match the entered conditions is counted and this becomes the output.

[1475] Step 4:

[1476] The server aggregates the search results and compiles them as statistical data. The input is the number of cases counted in step 3. The statistical data is compiled in an aggregated format by day, week, or month. This aggregated data is the output.

[1477] Step 5:

[1478] The server performs data visualization based on the aggregated data. The input is the aggregated data generated in step 4. The server uses the Tableau API to visualize the aggregated data as graphs and tables. This visualized data is the output.

[1479] Step 6:

[1480] The server automatically creates a report containing the visualized data. The input is the visualization data generated in step 5. An automatic report generation tool is used to generate a report. This report is the output.

[1481] Step 7:

[1482] The user (person in charge) checks the report generated on the terminal. The input is the report generated by the server. The user checks the contents and makes corrections if necessary. The checked and corrected report is the output.

[1483] Step 8:

[1484] The terminal sends a report to the administrator after confirmation is complete. The input is the report confirmed in step 7. The report is sent to the administrator via the internal mail system API or shared drive. This sent report becomes the final output.

[1485] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1486] This invention relates to a system for call center operations that converts customer interactions into text, generates case summaries, automatically counts the number of cases, and creates report materials while taking into account customer emotional information using an emotion engine. This system is composed of a combination of speech recognition technology, natural language processing technology, emotion recognition technology, data visualization technology, and automatic report generation technology.

[1487] Program processing natural language explanation

[1488] Step 1: Transcribe your interactions with customers

[1489] The server acquires the call voice data of the call center. The call voice is received in batches in real time from the VoIP system.

[1490] The server uses voice recognition technology to convert the received voice data into text data. The voice recognition engine utilizes a cloud-based API.

[1491] Step 2: Generate a case summary

[1492] The server analyzes the converted text data using a natural language processing toolkit, which includes tokenization, part-of-speech tagging, stemming, and lemmatization.

[1493] The server applies NLP models to extract key phrases and information from the text data, generating a summary of the case, including information such as the issue, request details, and response status.

[1494] Step 3: Recognizing and integrating emotional information

[1495] The server uses an emotion engine to recognize the customer's emotions during the call in real time, for example by analyzing the intonation, speed, and tone of the voice to identify the emotion.

[1496] The server adds the recognized emotional information to the case summary, which makes the importance and urgency of the case more clear.

[1497] Step 4: Counting and data collection of cases

[1498] The terminal receives instructions from an administrator and provides an interface for inputting specific keywords or phrases, such as "returned goods" or "delayed delivery."

[1499] The server uses the entered keywords to search a text database and identify relevant cases. High-speed search processing is performed using a full-text search engine.

[1500] The server aggregates the search results and counts the number of relevant cases. The counted data is compiled and organized as daily, weekly, and monthly statistical information.

[1501] Step 5: Integrate and analyze sentiment data

[1502] The server integrates the recognized emotion data into aggregate data and analyzes emotion trends, for example, analyzing the frequency of negative emotions when handling complaints.

[1503] Step 6: Prepare your report

[1504] The server uses a data visualization tool to generate graphs and tables that are easy to understand visually based on the aggregated data.

[1505] The server uses an automated report generation tool to create a report containing generated graphs and tables, including the emotion data.

[1506] Step 7: Final review and delivery

[1507] The user (person in charge) checks the report materials on the terminal, checking the content and format of the materials through the interface for any errors.

[1508] The device sends the verified report materials to the administrator, who then saves them in the internal email system or on a shared drive and notifies the appropriate administrator.

[1509] Specific examples

[1510] Example 1: Counting returns and recognizing emotions

[1511] 1. The server receives the call audio data related to "returns" and converts it into text.

[1512] 2. The server analyzes the converted text data and generates a summary of the case, including important information such as the reason for return, the date of the request, and the status of the response.

[1513] 3. The server uses an emotion engine to recognize the customer's emotions during the call and adds that information to the summary.

[1514] 4. The administrator enters keywords related to "returns" on the terminal and issues instructions to check the number of cases.

[1515] 5. The server searches for cases related to "returns" and counts the number of cases.

[1516] 6. The server aggregates the data based on the counting results, creates visual graphs and tables, and integrates the emotion data.

[1517] 7. The user checks the generated report materials on the terminal and sends them to the administrator.

[1518] Example 2: Counting delivery delay cases and recognizing emotions

[1519] 1. The server receives the call audio data related to "Delivery Delay" and converts it into text.

[1520] 2. The server analyzes the text data and generates a project summary.

[1521] 3. The server uses an emotion engine to recognize the customer's emotions during the call and adds that information to the summary.

[1522] 4. The administrator enters keywords related to "delivery delay" on the terminal and issues an instruction to check the number of cases.

[1523] 5. The server searches for the number of cases related to "Delivery Delay" and aggregates the results.

[1524] 6. The server creates graphs and tables based on the aggregated data and generates report materials including emotion data.

[1525] 7. The user checks the materials and sends them to the administrator.

[1526] The introduction of this system will significantly improve the efficiency of call center operations, reduce the burden on staff, and provide more accurate and useful data that takes into account customer emotional information.

[1527] The processing flow will be explained below.

[1528] Step 1:

[1529] The server acquires the call center's voice data, which is received in batches in real time from the VoIP system.

[1530] Step 2:

[1531] The server converts the acquired voice data into text data using speech recognition technology. The speech recognition engine uses a cloud-based API, and the converted text data is temporarily stored.

[1532] Step 3:

[1533] The server analyzes the stored text data using a natural language processing toolkit (e.g., spaCy or NLTK), which includes text preprocessing (tokenization, part-of-speech tagging, stemming, and lemmatization).

[1534] Step 4:

[1535] The server applies NLP models (e.g., BERT or GPT-3) to extract key phrases and information from the text data, summarizing information such as the problem, request details, and response status, and generating a case summary in a standardized format.

[1536] Step 5:

[1537] The server uses an emotion engine to recognize the customer's emotions in real time during a call. It analyzes the intonation, speed, and tone of the voice to identify emotions. For example, it determines whether the customer is angry or satisfied.

[1538] Step 6:

[1539] The server adds the recognized emotion information to the case summary, which more clearly indicates the importance and urgency of the case. For example, an emotion like "anger" will draw special attention.

[1540] Step 7:

[1541] The terminal provides an interface for administrators to input specific keywords or phrases of interest, such as "returned goods" or "delayed delivery."

[1542] Step 8:

[1543] The server searches a text database using keywords entered from the terminal to identify relevant cases. Using a full-text search engine (e.g., Elasticsearch) enables high-speed search processing.

[1544] Step 9:

[1545] The server aggregates the search results and counts the number of matching cases. The counted data is compiled and organized as daily, weekly, and monthly statistical information.

[1546] Step 10:

[1547] The server then aggregates the recognized emotion data and analyzes the trend of emotions, for example, analyzing the frequency and tendency of negative emotions in complaint handling.

[1548] Step 11:

[1549] The server uses a data visualization tool (e.g., Matplotlib or Tableau) to generate visually easy-to-understand graphs and tables based on the aggregated data.

[1550] Step 12:

[1551] The server uses automated report generation tools (e.g., LaTeX or Pandas DataFrame) to create a report containing generated graphs and tables, including the sentiment data.

[1552] Step 13:

[1553] The user (person in charge) checks the report materials on the terminal, checking the content and format of the materials through the interface for any errors.

[1554] Step 14:

[1555] The device sends the verified report materials to the administrator, who then saves them in the internal email system or on a shared drive and notifies the appropriate administrator.

[1556] Example 2

[1557] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1558] In call center operations, traditional methods require manual recording and analysis of call content, which is labor-intensive and time-consuming, making it difficult to provide efficient customer service. Additionally, the lack of data that reflects customer sentiment makes it difficult to develop specific measures to improve customer satisfaction and the quality of customer complaint handling.

[1559] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1560] In this invention, the server includes means for receiving interactions with customers as voice data, means for converting the received voice data into text data using voice recognition technology, means for analyzing the converted text data using natural language processing technology and generating a case summary, means for recognizing customer emotion information from the analyzed text data and integrating it into the case summary, means for automatically counting the number of cases related to specific keywords or phrases and generating aggregated data, and means for automatically creating reporting materials based on the generated aggregated data. This enables efficient recording and analysis of call content and enables data collection and reporting that takes customer emotion into consideration.

[1561] "Audio data" refers to data that has been recorded and stored in digital format.

[1562] "Speech recognition technology" is a technology in which a computer analyzes voice data and converts the content into text data.

[1563] "Text data" refers to text information converted using voice recognition technology.

[1564] "Natural language processing technology" refers to the technology that allows computers to understand and analyze human language.

[1565] A "case summary" is a summary of important information extracted from text data analyzed using natural language processing technology.

[1566] "Emotional information" refers to information that indicates the emotional state of the customer during a call.

[1567] "Specific keywords or phrases" are words or expressions that an administrator specifies they want to check.

[1568] "Job counting" refers to the process of automatically counting the number of jobs related to a particular keyword or phrase.

[1569] "Aggregated data" refers to data that compiles information such as the number of cases and emotional information.

[1570] "Reporting materials" refers to report-style materials that include graphs and tables visualized based on aggregated data.

[1571] MODE FOR CARRYING OUT THE INVENTION

[1572] This invention relates to a system that converts customer interactions into text in call center operations, organizes case summaries, and automatically creates reports that take emotional information into account. This system combines speech recognition technology, natural language processing technology, emotion recognition technology, and data visualization technology.

[1573] This system uses the following main hardware and software:

[1574] Server: Receives customer voice data from the VoIP system in real time and converts it into text using a speech recognition engine. For speech recognition, it utilizes a general cloud service API.

[1575] Natural language processing tools: Analyze text data, extract key information, and generate case summaries using techniques such as tokenization, part-of-speech tagging, stemming, and lemmatization, specifically using NLTK and BERT-based language models.

[1576] Sentiment engine: Recognizes customer sentiment from voice and text data and integrates it into case summaries. This includes sentiment analysis tools such as IBM Watson.

[1577] Database system: Stores and manages text data, case summaries, emotional information, etc.

[1578] Data visualization tools: Use tools such as Tableau to visualize the collected data and create reporting materials.

[1579] Automatic report generation tool: Uses Apache FOP to generate reports in PDF format based on various data.

[1580] For example, when a customer calls a call center regarding a return, the following process is performed:

[1581] 1. The server retrieves the call audio data related to the return from the VoIP system.

[1582] 2. The server calls a common cloud service API to convert the voice data into text and store it in a database.

[1583] 3. The server performs preprocessing using NLTK, and then uses a BERT-based model to extract important information such as the reason for return, request date, and response status, and generates a summary of the case.

[1584] 4. The server uses the IBM Watson emotion engine to recognize the customer's emotion and add it to the case summary.

[1585] 5. The terminal provides an interface for the administrator to enter keywords such as "return." The server uses Elasticsearch to search for matching cases and count the number of cases.

[1586] 6. The server uses Tableau to visually transform the aggregated data into graphs and tables, and uses Apache FOP to generate reporting materials.

[1587] 7. The user (person in charge) checks the report materials on the terminal and sends them to the administrator via the internal email system or shared drive.

[1588] Examples of prompts include:

[1589] "Analyze call data related to returns, generate case summaries, and add sentiment information. Include the specific reason for the return and the date of the request."

[1590] "Analyze trends in customer sentiment by compiling performance data on delivery delays. Create visual graphs and tables and generate reports."

[1591] As described above, the introduction of this system will improve the efficiency of call center operations and data accuracy.

[1592] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1593] Step 1:

[1594] The server receives call voice data from the call center in real time from the VoIP system. The voice data is the input, and an audio file is generated as the output. Specifically, it establishes a connection with the VoIP system and records the call voice in digital format.

[1595] Step 2:

[1596] The server sends the acquired voice data to the Google Cloud Speech-to-Text API and converts the voice data into text data. The input is voice data, and the output is the converted text data. Specifically, the server sends voice data to the API endpoint via an HTTP request and receives the converted text data as an HTTP response.

[1597] Step 3:

[1598] The server preprocesses the converted text data using NLTK. Preprocessing includes tokenization, part-of-speech tagging, stemming, and lemmatization. The input is text data, and the output is preprocessed text data. Specifically, the server breaks the text data down into words, tags each word with a part of speech, and converts it to its root form.

[1599] Step 4:

[1600] The server inputs the preprocessed text data into a BERT-based NLP model to extract important information. The input is the preprocessed text data, and the output is the extracted important information (such as "problem," "request content," and "response status"). Specifically, the server converts the text data into vectors, inputs them into the BERT model, and extracts important phrases that the model predicts.

[1601] Step 5:

[1602] The server generates a summary of the case based on the extracted important information and saves it in the database. The input is the important information and the output is the summary of the case. Specifically, the server applies the important information to a template, generates a summary, and saves it in the database.

[1603] Step 6:

[1604] The server sends the customer's emotional information from the voice and text data to the IBM Watson Tone Analyzer to recognize emotions. The input is voice and text data, and the output is emotional information. Specifically, it extracts features from the voice data and sends them along with the text data to the emotion analysis API to obtain emotional information.

[1605] Step 7:

[1606] The server integrates the acquired emotion information into the case summary and saves the updated summary in the database. The input is the emotion information and the existing case summary, and the output is the updated case summary. Specifically, the emotion information is added to the existing summary and re-saved as a new summary in the database.

[1607] Step 8:

[1608] The terminal provides an interface for the administrator to input specific keywords or phrases (e.g., "returned goods" or "delayed delivery"). The input is the administrator's keywords, and the output is the search criteria. Specifically, it provides a web form and retrieves the input keywords.

[1609] Step 9:

[1610] The server uses Elasticsearch to search a text database for the entered keywords and identify matching cases. The input is the search criteria, and the output is a list of the searched cases. Specifically, it generates and executes an Elasticsearch query to retrieve the search results.

[1611] Step 10:

[1612] The server aggregates the search results and counts the number of cases by day, week, month, etc. The input is the search results and the output is the aggregated data. Specifically, the server aggregates the search results, counts the cases for each specified period, and performs the aggregation.

[1613] Step 11:

[1614] The server then uses Tableau to visually transform the aggregated data into graphs and tables. The input is the aggregated data, and the output is the visualized graph or table. Specifically, the server loads the data into Tableau, selects the appropriate visualization, and generates it.

[1615] Step 12:

[1616] The server uses Apache FOP to automatically generate visualized data as report materials. The input is visualized graphs and tables, and the output is report materials (PDF format). Specifically, the data generated in XML format is input into Apache FOP, and a PDF file is generated.

[1617] Step 13:

[1618] The user (person in charge) checks the report materials on the terminal. The input is the report materials, and the output is the confirmation results. Specifically, the user opens the materials using a PDF viewer and checks for any errors in the content or format.

[1619] Step 14:

[1620] The terminal saves the confirmed report materials in the internal email system or shared drive and notifies the appropriate administrator. The input is the confirmed report materials, and the output is a notification to the administrator. Specifically, the terminal saves the materials in a specified folder and contacts the administrator via email or the notification system.

[1621] (Application example 2)

[1622] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1623] In customer service operations in brick-and-mortar stores, it is important to efficiently collect and analyze customer requests and feedback, and then respond appropriately based on the results. However, conventional methods have made it difficult to accurately record customer comments and analyze them in a timely manner, and there are also limited means of recognizing and reflecting customer emotions in real time. This has made it difficult to improve customer satisfaction and has increased the burden on staff.

[1624] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving interactions with customers as voice data, means for converting the received voice data into text data using voice recognition technology, means for analyzing the converted text data using natural language processing technology and generating a summary of the cases, means for automatically counting the number of cases related to specific keywords or phrases and generating aggregated data, means for adding recognized emotion data to the summary, and means for automatically creating reporting materials based on the generated aggregated data. This makes it possible to efficiently collect and analyze customer requests and feedback during customer service operations in physical stores and reflect the emotional information in real time.

[1625] "Interaction with customers" refers to communication such as dialogue and information exchange between service providers and customers.

[1626] "Audio data" means data for recording, storing, and processing audio information in digital format.

[1627] "Means of receiving" refers to the technical means or devices for acquiring data or information from the outside and incorporating it into the internal environment.

[1628] "Speech recognition technology" is a technology for analyzing voice data and converting it into text data.

[1629] "Text data" is data that represents character information in digital form.

[1630] "Natural language processing technology" is an artificial intelligence technology for analyzing, understanding, and generating human language.

[1631] A "case summary" is a document or data that succinctly summarizes important information or key points about a particular case.

[1632] "Specific keywords and phrases" refer to important words or short sentences that indicate specific information or content.

[1633] "Means for automatically counting the number of cases" refers to a device or program that has the function of calculating the number of cases based on specific conditions.

[1634] "Aggregated data" is data that shows overall trends and characteristics by organizing and statistically processing multiple data.

[1635] "Recognized emotion data" refers to information about an emotional state analyzed from data such as voice or text.

[1636] "Means for adding to the summary" refers to technical means for integrating additional information into the generated summary text.

[1637] "Reporting materials" refers to documents, graphs, charts, etc. created to report specific information.

[1638] "Automated means" refers to technology or devices that perform specific tasks automatically without human intervention.

[1639] A "data visualization tool" is software that converts data into a format that is easy to understand visually, such as a graph or table.

[1640] "Smart glasses" are wearable smart devices equipped with a display, sensors, communication functions, etc.

[1641] "Customer service" refers to all work related to providing products and services to customers and ensuring their satisfaction.

[1642] The present invention provides a system for improving the efficiency of customer service operations in brick-and-mortar stores and enhancing customer satisfaction. An embodiment of this system will be described in detail below.

[1643] System Configuration

[1644] This system mainly consists of four components: "smart glasses," "server," "terminal," and "user."

[1645] 1. Smart Glasses

[1646] Smart glasses are wearable devices equipped with a display, microphone, sensors, and communication functions. They are worn by store staff while serving customers and are used to capture conversations with customers. They convert speech into text in real time and perform sentiment analysis.

[1647] 2. Server

[1648] The server performs the main processes of receiving voice data, converting it into text, analyzing it, recognizing emotions, and compiling it. It uses the following software and technologies:

[1649] Speech recognition engine: Google Cloud Speech-to-Text API

[1650] Natural language processing tool: SpaCy

[1651] Emotion recognition engine: IBM Watson Tone Analyzer

[1652] Data visualization tool: Tableau

[1653] Automated Report Generation Tool: Google Docs API

[1654] 3. Terminal

[1655] Terminals are devices used by store managers and staff to check generated reporting materials, enter data, give instructions, etc. Data can be checked, corrected, and sent through an interface.

[1656] Specific examples of implementation

[1657] Step 1: Capture and transcribe customer interactions

[1658] The server receives voice data from the smart glasses in real time and converts it into text data using the Google Cloud Speech-to-Text API.

[1659] Step 2: Analyze requirements and generate a summary

[1660] The server then uses SpaCy to analyze the converted text data and generate a summary of the case. For example, if a customer asks, "Is this product in stock?", the server generates the summary "Product inventory inquiry."

[1661] Step 3: Recognizing and integrating emotional data

[1662] The server uses IBM Watson Tone Analyzer to analyze customer sentiment from voice and text data. For example, if a customer expresses impatience, it adds that information to the summary.

[1663] Step 4: Generate and visualize aggregate data

[1664] The server automatically counts the number of jobs related to specific keywords or phrases and generates aggregate data, which is then visualized in graphs and tables using Tableau.

[1665] Step 5: Prepare, review and distribute report materials

[1666] The server uses the Google Docs API to automatically create reporting materials based on the generated aggregated data. The user (store manager) can check the materials on their device, make any necessary corrections, and then send the materials via email or cloud drive.

[1667] Prompt sentence for specific implementation example

[1668] For example, when a system captures a customer question such as "Do you have this product in stock?" and performs response and sentiment analysis, the following prompt sentences are used:

[1669] "Please give us your feedback on the product."

[1670] "Do you have any questions about this product?"

[1671] Please let us know your thoughts on the service.

[1672] This will enable efficient collection of customer requests and feedback during customer service operations in brick-and-mortar stores, enabling detailed analysis based on emotional data, and providing useful information to improve staff efficiency and increase customer satisfaction.

[1673] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1674] Step 1:

[1675] The server receives voice data from the smart glasses in real time. The voice data is captured through the microphone of the smart glasses and sent to the server through the VoIP system. The input is voice data, which is analyzed by the voice recognition engine.

[1676] Step 2:

[1677] The server uses the Google Cloud Speech-to-Text API to convert the received voice data into text data. The input is voice data, and the output is text data that expresses the contents of the voice data in characters. The server analyzes and converts the voice data.

[1678] Step 3:

[1679] The server uses SpaCy to perform natural language processing on the converted text data. The input is text data, which undergoes processes such as tokenization, part-of-speech tagging, stemming, and lemmatization. The output is each element of the analyzed text data and a summary of the case.

[1680] Step 4:

[1681] The server uses IBM Watson Tone Analyzer to recognize emotions from the analyzed text and voice data. The input is the analyzed text and voice data, and the output is the recognized emotion data. Specifically, it detects the emotional state (e.g., impatience, anger, joy) from the tone and rate of the customer's voice.

[1682] Step 5:

[1683] The server integrates the emotion data into a case summary. The input is the summary data and emotion data, and the output is a case summary including emotion information. This makes the importance and urgency of the case clear.

[1684] Step 6:

[1685] The terminal provides an interface that accepts specific keywords or phrases entered by the administrator. The input is a keyword (e.g., "satisfied," "dissatisfied," "returned product"), and the number of cases is counted based on this.

[1686] Step 7:

[1687] The server searches the database using the entered keywords and counts the number of matching cases. The input is the keywords and text database, and the output is the number of matching cases. The search is performed quickly using a full-text search engine.

[1688] Step 8:

[1689] The server aggregates the number of cases and emotion data counted and visualizes the data using Tableau. The input is the number of cases and emotion data, and the output is easy-to-understand graphs and tables.

[1690] Step 9:

[1691] The server automatically creates report materials using the Google Docs API. The input is the aggregated data and visualization data, and the output is the report document. The server automatically generates the report and creates the document in the appropriate format.

[1692] Step 10:

[1693] Users can check and modify report materials on their terminals. The input is the automatically generated report material, and the output is the final, verified report material. The user checks the content and format of the material to ensure there are no errors.

[1694] Step 11:

[1695] The terminal sends the verified report materials to the administrator. They are saved by email or cloud drive and notified to the administrator. The input is the verified report materials, and the output is the report materials that can be accessed by the administrator. This ensures that the final report is distributed appropriately.

[1696] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1697] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1698] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1699] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1700] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1701] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1702] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1703] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1704] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1705] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1706] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1707] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1708] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1709] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1710] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1711] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1712] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1713] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1714] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1715] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1716] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1717] The following is further disclosed regarding the above embodiment.

[1718] (Claim 1)

[1719] A means of receiving communication with customers as voice data;

[1720] A means for converting the received voice data into text data using voice recognition technology;

[1721] a means for analyzing the converted text data using natural language processing technology and generating a summary of the case;

[1722] A means to automatically count the number of jobs related to specific keywords or phrases and generate aggregate data;

[1723] A means for automatically creating report materials based on the generated aggregated data;

[1724] A system including:

[1725] (Claim 2)

[1726] The system of claim 1, further comprising means for generating the aggregated data as a graph or table using a data visualization tool.

[1727] (Claim 3)

[1728] 10. The system of claim 1, further comprising means for generating said report material using an automated report generation tool.

[1729] "Example 1"

[1730] (Claim 1)

[1731] A means of receiving communication with customers as voice data;

[1732] A means for converting the received voice data into text data using voice recognition technology;

[1733] a means for analyzing the converted text data using natural language processing technology and generating a summary of the case;

[1734] A means to automatically count the number of jobs related to specific keywords or phrases and generate aggregate data;

[1735] A means for automatically creating report materials based on the generated aggregated data;

[1736] A means of statistically summarizing aggregate data in daily, weekly, and monthly formats;

[1737] means for supporting real-time and batch processing of audio data;

[1738] means of tokenization, part-of-speech tagging, stemming, and lemmatization during the analysis process;

[1739] A system including:

[1740] (Claim 2)

[1741] The system of claim 1, further comprising means for generating the aggregated data as a graph or table using a data visualization tool.

[1742] (Claim 3)

[1743] 10. The system of claim 1, further comprising means for generating said report material using an automated report generation tool.

[1744] "Application Example 1"

[1745] (Claim 1)

[1746] A means of receiving communication with customers as voice data;

[1747] A means for converting the received voice data into text data using voice recognition technology;

[1748] a means for analyzing the converted text data using natural language processing technology and generating a summary of the case;

[1749] A means to automatically count the number of jobs related to specific keywords or phrases and generate aggregate data;

[1750] A means for automatically creating report materials based on the generated aggregated data and distributing the generated report;

[1751] A means including a technology for converting voice data into text in real time;

[1752] A means to count the number of summarized cases and visualize the data based on that;

[1753] A system including:

[1754] (Claim 2)

[1755] 2. The system according to claim 1, further comprising means for generating the aggregated data as a graph or table using a data visualization tool and outputting the graph or table in a format that is visually easy to understand.

[1756] (Claim 3)

[1757] 2. The system according to claim 1, further comprising means for generating materials for the report using an automatic report generation tool and automatically transmitting the generated report to an administrator.

[1758] "Example 2: Combining Emotion Engines"

[1759] (Claim 1)

[1760] A means of receiving communication with customers as voice data;

[1761] A means for converting the received voice data into text data using voice recognition technology;

[1762] a means for analyzing the converted text data using natural language processing technology and generating a summary of the case;

[1763] A method for recognizing customer sentiment information from the analyzed text data and integrating it into a case summary;

[1764] A means to automatically count the number of jobs related to specific keywords or phrases and generate aggregate data;

[1765] A means for automatically creating report materials based on the generated aggregated data;

[1766] A system including:

[1767] (Claim 2)

[1768] 10. The system of claim 1, further comprising means for generating the generated aggregated data as a graph or table using a data visualization tool.

[1769] (Claim 3)

[1770] 10. The system of claim 1, further comprising means for generating the report material using an automated report generation tool.

[1771] "Application example 2 when combining emotion engines"

[1772] (Claim 1)

[1773] A means of receiving communication with customers as voice data;

[1774] A means for converting the received voice data into text data using voice recognition technology;

[1775] a means for analyzing the converted text data using natural language processing technology and generating a summary of the case;

[1776] A means to automatically count the number of jobs related to specific keywords or phrases and generate aggregate data;

[1777] a means for adding recognized emotion data to the summary;

[1778] A means for automatically creating report materials based on the generated aggregated data;

[1779] A system including:

[1780] (Claim 2)

[1781] The system of claim 1, further comprising means for generating the aggregated data as a graph or table using a data visualization tool.

[1782] (Claim 3)

[1783] 10. The system of claim 1, further comprising means for generating said report material using an automated report generation tool.

[1784] (Claim 4)

[1785] The system of claim 1, further comprising means for combining the speech recognition technology, natural language processing technology, and emotion recognition technology to collect data related to customer service operations in a physical store and analyze it in real time.

[1786] (Claim 5)

[1787] The system of claim 4, further comprising means for capturing interactions with customers during service using the smart glasses device and performing text conversion and sentiment analysis. [Explanation of symbols]

[1788] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of receiving communication with customers as voice data; A means for converting the received voice data into text data using voice recognition technology; a means for analyzing the converted text data using natural language processing technology and generating a summary of the case; A means to automatically count the number of jobs related to specific keywords or phrases and generate aggregate data; A means for automatically creating report materials based on the generated aggregated data; A system including:

2. The system of claim 1 , further comprising means for generating the aggregated data as a graph or table using a data visualization tool.

3. The system of claim 1 further comprising means for generating said report material using an automated report generation tool.

Images