system

Abstract

We provide the system. [Solution] A means for converting audio data from meetings, etc., into text data, A method for extracting frequently occurring words from converted text data, A method to search for related documents from the company's database based on the extracted frequently occurring words and sort them in order of synchronicity, A means of generating links to the arranged materials, A system that includes this.

Description

Technical Field

[0005] ,

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor and includes steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the modern business environment, there is a demand for efficiently managing information such as meetings and consultations, and quickly searching for and providing related materials. However, there are problems that the content of meetings is not effectively recorded, related materials are often scattered, the excess or deficiency of information and the difficulty of searching increase. In addition, when employees lack understanding of technical terms and the latest technologies, there is a possibility that work efficiency will decrease. There is a demand for a system that comprehensively solves these problems.

Means for Solving the Problems

[0005] This invention proposes a system that converts audio data from meetings and other events into text data, extracts frequently occurring words from that text, and then searches for and provides related materials from an internal company database based on those frequently occurring words. Specifically, the system provides means for converting audio data from meetings and other events into text data, means for extracting frequently occurring words from the converted text data, means for searching for related materials from an internal company database based on the extracted frequently occurring words and arranging them in order of synchronization rate, and means for generating links to the arranged materials. Furthermore, the system includes means for organizing materials in the internal company database by category based on frequently occurring words and related materials, presenting the latest hot topics related to the frequently occurring words, and means for identifying terms that users may not understand from the extracted frequently occurring words, generating explanations for the identified terms, and presenting links to related materials, thereby improving the efficiency of information management and enhancing employee skills.

[0006] "Audio data" refers to digital data that records the content of speech spoken during meetings or discussions.

[0007] "Text data" refers to string information converted from audio data, in a format that can be read by humans or analyzed by computers.

[0008] "Frequently occurring words" refer to keywords or terms that appear many times within text data.

[0009] An "internal database" is a data storage system operated within a company, where documents, materials, and other data are managed.

[0010] "Synchronization rate" refers to the percentage of the extracted frequently occurring words that match the content of documents in the company's internal database.

[0011] A "link" is a hypertext address that allows users to directly access related materials or web pages by clicking on it.

[0012] A "category" is a system for classifying information or materials based on specific themes or attributes.

[0013] A "hot topic" refers to a topic or piece of information that is receiving particular attention or is considered important at a given time.

[0014] A "glossary" is a text or piece of information that clearly explains the meaning of specific technical terms or difficult words. [Brief explanation of the drawing]

[0015] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when the emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when the emotion engine is combined.

Mode for Carrying Out the Invention

[0016] Hereinafter, an example of an embodiment of the system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be described.

[0018] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be one arithmetic unit or a combination of a plurality of arithmetic units. Also, the processor may be one type of arithmetic unit or a combination of a plurality of types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0019] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0020] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0021] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0023] [First Embodiment]

[0024] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0025] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0028] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0031] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0035] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0036] This invention relates to a system that converts audio data, such as from meetings, into text data, extracts frequently occurring words from that text data, and searches for and provides related materials from an internal company database based on those words. It also provides explanations of the latest hot topics related to the frequently occurring words, as well as terms that employees may not be familiar with.

[0037] Acquisition of audio data and text conversion

[0038] The user uploads meeting audio data (e.g., a .wav file) to the device. The device sends this audio data to a speech recognition API, which converts the audio data into text data. The converted text data is then formatted into a specific format within the device.

[0039] Extraction of frequently occurring words

[0040] The terminal uses natural language processing tools to extract frequently occurring words from the converted text data. For example, if the meeting content was "discussing future market strategies," keywords such as "market" and "strategy" would be extracted.

[0041] Searching for and providing related materials

[0042] Next, the terminal sends the extracted list of frequently occurring words to the server. The server searches for documents in the company's database based on the received list of frequently occurring words. The search results are sorted by calculating the synchronization rate between the content of each document and the frequently occurring words, and then sorting them in descending order of synchronization rate. As a result, documents such as "Strategic Plan 2023.pdf" and "Market Analysis Report 2023.pptx" are listed in reverse chronological order. The server generates links to these sorted documents and provides them to the user.

[0043] Categorizing materials and providing hot topics

[0044] The server also has the function of organizing all documents in the company's internal database into categories based on themes and attributes. This allows for efficient management of documents by category, such as "Market Analysis" and "Strategy Formulation." Furthermore, it selects and presents the latest documents related to frequently used keywords as hot topics. For example, within the "Market Analysis" category, the latest "September Updated Market Report.pdf" is presented as a hot topic.

[0045] Explanation of terminology and improvement of employee skills

[0046] The device identifies terms that the user may not understand from a list of frequently used words. For example, if the frequently used word "digital transformation" is included, the device will determine, based on past data and the user's skill level, that this term is unfamiliar. The device will generate an easy-to-understand explanation of this term and also display a link to related materials (e.g., "DX Introduction Guide.pdf").

[0047] Explanation with specific examples

[0048] The following is a concrete example. An audio file, "Meetings_aug30.wav," from a meeting held on August 30th is uploaded to the terminal. The terminal uses a speech recognition API to convert this audio file into text, and the content is rephrased as "Discussion on future market strategy policies." A natural language processing tool is used to extract frequently occurring words such as "market" and "strategy," and this is sent to the server. The server searches the company's internal database for related documents such as "Strategic Plan 2023.pdf" and "Market Analysis Report 2023.pptx," sorts them in reverse chronological order, and generates links. Furthermore, it presents the latest "September Market Report.pdf" as a hot topic related to "market analysis," and provides the user with an explanation of "digital transformation" and related materials.

[0049] Thus, the present invention provides a system that efficiently searches for and provides relevant materials based on meeting content, and contributes to improving employees' knowledge.

[0050] The following describes the processing flow.

[0051] Step 1:

[0052] The user uploads the meeting audio data (e.g., a .wav file) to their device. The audio data is in a format such as "Meetings_aug30.wav".

[0053] Step 2:

[0054] The device sends the uploaded audio data to a speech recognition API, which converts the audio data into text data. This is where text such as "Strategic Meeting August 30, 2023" is generated.

[0055] Step 3:

[0056] The terminal receives the converted text data and formats it by removing unnecessary information (such as noise or irrelevant conversation). For example, unnecessary words like "um" or "well" are removed.

[0057] Step 4:

[0058] The device uses natural language processing tools to extract frequently occurring words from the formatted text. For example, keywords such as "strategy," "market," and "analysis" might be extracted.

[0059] Step 5:

[0060] The terminal sends the extracted list of frequently occurring words to the server. This list includes not only the frequently occurring words but also their frequency of occurrence.

[0061] Step 6:

[0062] Based on the frequently occurring word list received by the server, all documents in the company database are searched. The degree of agreement between the document content and the frequently occurring words, i.e., the synchronization rate, is calculated.

[0063] Step 7:

[0064] The server sorts the documents with the highest synchronization rate by newest first. For example, "Strategic Plan 2023.pdf" and "Latest Market Analysis Report 2023.pptx" are output.

[0065] Step 8:

[0066] The server generates links to a sorted list of materials and provides them to the user via the terminal. This allows the user to easily access relevant materials.

[0067] Step 9:

[0068] The server organizes documents within the company's internal database by category. For example, they might be divided into categories such as "Strategy," "Market," and "Analysis."

[0069] Step 10:

[0070] The server searches for and presents the latest hot topics related to frequently occurring keywords. For example, "September Market Report.pdf" might be presented as a hot topic within the "Market Analysis" category.

[0071] Step 11:

[0072] The device identifies terms that the user may not understand from among frequently used words. For example, "digital transformation" might be identified in this way.

[0073] Step 12:

[0074] The device provides explanations of terms it has identified and links to related materials. For example, it might display an explanation such as, "Digital transformation is the process by which companies use digital technologies to transform their business models," along with a link to "DX Introduction Guide.pdf".

[0075] (Example 1)

[0076] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0077] Traditional methods of recording and analyzing meetings made it difficult to efficiently utilize audio data, requiring significant time and effort for manual text conversion and document retrieval. Furthermore, the accuracy and speed of searching for and providing relevant materials were limited, and explanations of unfamiliar terminology were insufficient. Therefore, a system was needed that could simultaneously achieve effective utilization of meeting data and improve knowledge.

[0078] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0079] In this invention, the server includes means for converting audio data into text data, means for extracting frequently occurring words from the converted text data, means for searching a database for related information based on the extracted frequently occurring words and arranging it in order of decreasing synchronization rate, means for generating links to the arranged information, means for using an external API for recognizing audio data, and means for using a natural language processing tool for extracting frequently occurring words. This makes it possible to efficiently transcribe meeting audio data into text, provide related materials quickly and accurately, and simultaneously improve employees' knowledge.

[0080] "Audio data" refers to digital data files containing recordings of audio from meetings, conversations, and other similar events.

[0081] "Text data" refers to a digital data file created by converting audio data into text.

[0082] "Frequently occurring words" refer to words or expressions that appear with a specific frequency within text data.

[0083] An "external API" refers to an application programming interface provided by an external service or platform.

[0084] "Natural language processing tools" refer to software and libraries used to analyze text data, understand its meaning and grammar, and process it.

[0085] A "database" refers to a collection of information that allows for the efficient management and retrieval of large amounts of data.

[0086] A "link" is a hypertext element that allows a user to access a specific webpage or file by clicking on it.

[0087] "Synchronization rate" refers to an indicator that shows the degree of agreement between frequently occurring words and the content of the materials in the database.

[0088] "Explanation" refers to a text or explanation that clearly describes unfamiliar or unknown terms.

[0089] This invention is a system that converts audio data, such as from meetings, into text data, extracts frequently occurring words from that text data, and searches and provides related materials from a database based on those words. Furthermore, it also provides the latest topics related to the frequently occurring words and explanations of terms that the user may not be familiar with. Embodiments of this system are described below.

[0090] First, the user uploads the meeting audio data (e.g., a .wav file) to their device. The device then sends this audio data to a speech recognition API (e.g., Google® Speech-to-Text or Amazon Transcribe) to convert it into text data. The converted text data is then formatted into a specific format (e.g., date and time, speaker, content of speech) and processed.

[0091] Next, the terminal inputs the converted text data into a natural language processing tool (e.g., NLTK or spaCy) to extract frequently occurring words. For example, if the meeting topic is "discussing future market strategies," frequently occurring words such as "market" and "strategy" will be extracted. The extraction results are compiled into a list of frequently occurring words.

[0092] The terminal sends the extracted list of frequently occurring words to the server. The server searches the database using the ElasticSearch® engine or similar based on the list. The search results are sorted by calculating the synchronization rate between the content of each document and the frequently occurring words, and then sorting them in descending order of synchronization rate. This results in a list of documents such as "Strategic Plan 2023.pdf" and "Market Analysis Report 2023.pptx," and links to the sorted documents are generated.

[0093] Furthermore, the server organizes all documents in the company database by theme (e.g., "Market Analysis" or "Strategy Formulation"), selects the latest information related to frequently used keywords, and presents it to the user. For example, within the "Market Analysis" category, the latest "September Updated Market Report.pdf" might be presented as a hot topic.

[0094] Furthermore, the device identifies terms from a list of frequently used words that the user may not understand, based on past learning data. For example, if the frequently used word "digital transformation" is identified as unfamiliar, it generates an easy-to-understand explanation and provides a link to related materials (e.g., "DX Introduction Guide.pdf").

[0095] Specific example:

[0096] An audio file, "Meetings_aug30.wav," from a meeting held on August 30th is uploaded to the user's device. This audio file is converted into text data via a speech recognition API on the device. The content is translated to something like "Discussion on future market strategy." Next, a natural language processing tool is used to extract frequently occurring words such as "market" and "strategy," and this is sent to the server. The server searches its database for related documents such as "Strategic Plan 2023.pdf" and "Market Analysis Report 2023.pptx," sorts them in reverse chronological order, and generates links. Furthermore, the latest "September Market Report.pdf" is presented as a hot topic related to "market analysis," and the user is provided with an explanation of "digital transformation" and related materials.

[0097] Example of a prompt

[0098] The following are specific examples of prompt statements that will be used as input to the generative AI model.

[0099] "Please describe a program to search for and provide materials related to the topics of 'market' and 'strategy' discussed at the August 30th meeting. Additionally, please explain the term 'digital transformation' and add a function to provide related materials."

[0100] This system enables efficient transcription of meeting content, prompt and accurate provision of related materials, and contributes to improving employees' knowledge.

[0101] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0102] Step 1: Acquire audio data and convert it to text.

[0103] The user uploads meeting audio data (e.g., a .wav file) to their device. The input is an audio data file. The device sends this audio data to the Google Speech-to-Text API and retrieves text data. The output is the converted text data. Specifically, the audio data is passed to the API, and the text data is received as the API response.

[0104] Step 2: Formatting the text data

[0105] The terminal formats the acquired text data into a specific format (e.g., date and time, speaker, content of the message). The input is text data obtained from the API. The output is the formatted text data. Specifically, the terminal organizes the text data using methods such as JSON or CSV format to make it suitable for subsequent processing.

[0106] Step 3: Extraction of frequently occurring words

[0107] The terminal inputs formatted text data into a natural language processing tool (e.g., NLTK or spaCy) and extracts frequently occurring words. The input is formatted text data. The output is a list of frequently occurring words. Specifically, the text data is tokenized, the frequency of each word is counted, and words exceeding a certain frequency are extracted.

[0108] Step 4: Send the list of frequently used words

[0109] The terminal sends the extracted list of frequently occurring words to the server. The input is the list of frequently occurring words. The output is the data sent to the server. Specifically, the process involves encoding the list of frequently occurring words in JSON format and sending it to the server using an HTTP request.

[0110] Step 5: Search for related materials

[0111] The server uses the Elasticsearch engine to search for materials in the database based on the received list of frequently occurring words. The input is the list of frequently occurring words. The output is the search results. Specifically, it generates an Elasticsearch query and executes the query against the database.

[0112] Step 6: Calculate and sort the data synchronization rate.

[0113] The server calculates the synchronization rate between the content of each document and its frequently occurring words, and sorts them in descending order of synchronization rate. The input is the search results. The output is the sorted list of documents. Specifically, it scores how well the frequently occurring words match the content of each document and sorts the documents based on that score.

[0114] Step 7: Generate and provide links

[0115] The server generates and provides links to sorted documents to the user. The input is a sorted list of documents. The output is links to the documents. Specifically, the server generates the links in HTML format, making them accessible to the user.

[0116] Step 8: Categorize materials and provide hot topics

[0117] The server organizes all documents in the company database by theme, selects the latest information related to frequently used keywords, and presents it to the user. Inputs are a list of documents and frequently used keywords. Outputs are documents organized by category and the latest hot topics. Specifically, it uses a classification algorithm to group documents by theme and selects the most recent documents.

[0118] Step 9: Terminology identification and explanation generation

[0119] The device identifies terms that the user may not understand from a list of frequently used words, generates easy-to-understand explanations for the unfamiliar terms, and displays links to related materials. Input consists of the list of frequently used words and the user's past learning data. Output is the term explanation and links. Specifically, it executes a discrimination algorithm based on the learning data, generates explanatory text, and attaches links to related materials.

[0120] (Application Example 1)

[0121] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0122] While it is crucial to respond quickly and accurately to customer questions and requests during customer service in stores, delays can occur when staff are busy or when there is inconsistency in product knowledge. Furthermore, insufficient information provided to customers can lead to decreased customer satisfaction. Additionally, it is difficult for new or part-time staff to fully grasp specialized terminology and detailed product information, which can result in a decline in the quality of customer service. A system is needed to address these challenges and improve customer satisfaction.

[0123] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0124] This invention includes a server that transcribes audio data of customer conversations into text in real time and searches for and provides related product information based on frequently occurring words; a server that searches for related materials from a database based on the extracted frequently occurring words and arranges them in order of synchronization rate; and a server that generates explanations for the identified terms and presents links to related materials. This makes it possible to convert conversations with customers into text data and quickly provide related product information and detailed explanations based on frequently occurring questions and requests.

[0125] "Audio data" refers to sound information, such as conversations and music, recorded and stored in digital format.

[0126] "Text data" refers to a digital representation of character information.

[0127] "Frequently occurring words" refer to words or phrases that appear particularly often within the text data.

[0128] A "database" is a system for efficiently managing and retrieving data, and is a collection of structured information.

[0129] "Synchronization rate" is an indicator that shows the degree of agreement between frequently occurring words and the data in the database.

[0130] A "link" is reference information that allows direct access to specific data or materials.

[0131] "Customer service" refers to the activities of interacting with and providing services to customers in stores and service industries.

[0132] "Real-time text conversion" is the process of instantly converting audio data into text data.

[0133] "Related product information" refers to detailed data and descriptions of products found through searches based on frequently occurring keywords.

[0134] "Category organization" is the process of classifying and managing data and materials based on common attributes or themes.

[0135] "Hot topics" refer to the latest information or topics that have recently become popular.

[0136] An "explanation" is a detailed explanation of a specific term or concept.

[0137] System program

[0138] The apparatus for carrying out this invention consists of a system in which a server, a terminal, and a user work together.

[0139] The terminal has the functionality to acquire voice data during customer service and convert it into text data. To achieve this, the Google Cloud Speech-to-Text API and Python's speech_recognition library are used. In addition, frequently occurring words are extracted from the converted text data using natural language processing tools (e.g., spaCy, NLTK).

[0140] The server has the functionality to search for related materials and product information from existing databases (e.g., Firebase) based on a list of frequently used words sent from the terminal, sort them in order of synchronicity, and generate links. Furthermore, it also has the functionality to present hot topics related to frequently used words.

[0141] Program Processing Description

[0142] 1. Acquisition and transcription of audio data

[0143] The system records conversations during customer service using a terminal and sends the recordings to the Google Cloud Speech-to-Text API, converting the audio data into text data. This makes the audio information easily searchable text data.

[0144] 2. Extraction of frequently occurring words

[0145] The converted text data is analyzed using natural language processing tools (NLTK, spaCy) to extract frequently occurring words. This allows for a quick understanding of customer interests and question trends.

[0146] 3. Searching for and providing related information

[0147] The device sends the extracted list of frequently occurring words to the server. The server searches its database (Firebase) for relevant product information and materials, sorts them in the optimal order based on the synchronization rate, and generates links.

[0148] 4. Explanation of hot topics and terminology

[0149] The server generates and presents detailed explanations, along with links, of the latest hot topics related to frequently used words and terms that customers may not understand.

[0150] Examples of specific cases and prompt statements

[0151] Specific example:

[0152] For example, if frequently occurring keywords such as "new product" and "promotion" are extracted from an audio file recorded during customer service on August 30th, the server will search for detailed information on products related to these keywords and the latest campaign information, and present them in a list sorted by the degree of synchronization. Furthermore, if it is determined that the term "promotion" is not well understood by new staff, an explanation of the term and links to related materials will also be provided.

[0153] Example of a prompt:

[0154] "We want to develop an application that transcribes audio data of customers being served on August 30th into text, extracts frequently occurring words, and then searches for and provides related product information and the latest promotional information."

[0155] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0156] Step 1:

[0157] The terminal acquires audio data during customer service. The user (store clerk) uses a smartphone or dedicated device to record the audio data of the conversation. This audio data becomes the input data, and in this step, an audio file is specifically generated.

[0158] Step 2:

[0159] The device sends the acquired audio data to the Google Cloud Speech-to-Text API, where it is converted into text data. The input here is the audio file obtained in step 1, and the API is used to convert that audio into text format and output it. Specifically, the audio data is sent to the API, and the converted text is returned from the API.

[0160] Step 3:

[0161] The terminal analyzes the converted text data using a natural language processing tool (e.g., spaCy or NLTK) and extracts frequently occurring words. The input is the text data obtained in step 2, and data processing such as word segmentation and part-of-speech analysis is performed using NLTK or spaCy to extract and output frequently occurring words. Specifically, the text is broken down, and the words that appear most frequently are listed.

[0162] Step 4:

[0163] The terminal sends the extracted list of frequently occurring words to the server. The input is the list of frequently occurring words obtained in step 3, and the terminal sends data to the server based on this. Specifically, the operation to send the list of frequently occurring words to the API endpoint is performed.

[0164] Step 5:

[0165] The server searches for relevant documents and product information from a database (such as Firebase) based on the list of frequently occurring words. The input for this step is the list of frequently occurring words submitted in step 4. The server searches the database to retrieve relevant documents and product information and outputs it as a list sorted by synchronization rate. Specifically, it forms a search query using frequently occurring words as keywords and extracts the corresponding data from the database.

[0166] Step 6:

[0167] The server generates links to relevant documents and product information from the search results and sends them to the terminal. The input is the search results obtained in step 5, and it generates and outputs links to the detail pages for each document and product. Specifically, it uses a link generation algorithm to form URLs, lists them, and sends them.

[0168] Step 7:

[0169] The server identifies hot topics and terms that customers may not understand, and generates explanations and related materials. The input is the search results obtained in step 5 and the user profile, and it generates and outputs explanatory text and links to related materials. Specifically, it identifies unfamiliar terms from frequently occurring words and historical data, and provides explanatory text and links to related materials based on those terms.

[0170] Step 8:

[0171] The terminal displays the received link and explanation to the user (store clerk) and provides it to the customer. The input consists of the link and explanation generated in steps 6 and 7, and is output by being displayed on a screen such as a smartphone. Specifically, the link is displayed in a clickable format, and the explanation is displayed in an information box.

[0172] Through these steps, users can respond quickly and accurately to customer questions and requests, thereby improving customer satisfaction.

[0173] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0174] This invention relates to a system that converts audio data, such as from meetings, into text data, extracts frequently occurring words from that text data, and searches for and provides related materials from an internal company database based on those words. Furthermore, by combining this with an emotion engine that recognizes the user's emotions, the system can also provide materials and change their priority according to the user's emotional state.

[0175] Acquisition of audio data and text conversion

[0176] The user uploads meeting audio data (e.g., a .wav file) to the device. The device sends this audio data to a speech recognition API, which converts the audio data into text data. The converted text data is then formatted into a specific format within the device.

[0177] Extraction of frequently occurring words

[0178] The terminal uses natural language processing tools to extract frequently occurring words from the converted text data. For example, if the meeting content was "discussing future market strategies," keywords such as "market" and "strategy" would be extracted.

[0179] Utilizing the Emotion Engine

[0180] The device utilizes an emotion engine to recognize the user's emotions in real time. For example, the emotion engine analyzes the user's facial expressions and voice tone while they are viewing documents and generates emotion data.

[0181] Searching for and providing related materials

[0182] Next, the terminal sends the extracted list of frequently occurring words and sentiment data to the server. The server searches for documents in the company's database based on the received list of frequently occurring words and sentiment data. The search results are sorted using a new criterion that integrates synchronization rate and sentiment influence, taking into account the degree of match between the content of each document and the frequently occurring words, as well as importance based on the user's sentiment data.

[0183] For example, if a user expresses positive feelings towards "marketing strategy," materials related to that topic will be displayed at the top of the search results. On the other hand, if negative feelings are detected, materials with a higher priority for mitigating those feelings will be presented.

[0184] Categorizing materials and providing hot topics

[0185] The server also has the function of organizing all documents in the company's internal database into categories based on themes and attributes. This allows for efficient management of documents by category, such as "Market Analysis" and "Strategy Formulation." Furthermore, it selects and presents the latest hot topics based on sentiment data. For example, within the "Market Analysis" category, the latest "September Updated Market Report.pdf" is presented as a hot topic.

[0186] Explanation of terminology and improvement of employee skills

[0187] The device identifies terms that the user may not understand from a list of frequently used words. For example, if the frequently used word "digital transformation" is included, the device will determine, based on past data and the user's skill level, that this term is unfamiliar. The device will generate an easy-to-understand explanation of this term and also display a link to related materials (e.g., "DX Introduction Guide.pdf").

[0188] Explanation with specific examples

[0189] The following is a concrete example. An audio file, "Meetings_aug30.wav," from a meeting held on August 30th is uploaded to the terminal. The terminal uses a speech recognition API to transcribe this audio file into text, and the content is translated into something like "Discussion on future market strategy." A natural language processing tool is used to extract frequently occurring words such as "market" and "strategy," and an emotion engine analyzes the user's emotional state (e.g., excitement, interest, negative emotion) in real time. The extracted list of frequently occurring words and emotion data are sent to the server.

[0190] The server searches the internal database for relevant documents such as "Strategic Plan 2023.pdf" and "Market Analysis Report 2023.pptx," and re-evaluates their priority based on synchronization rate and sentiment data. The documents are then sorted in the new order, and links are generated. Furthermore, the latest "September Market Report.pdf" is presented as a hot topic, and the user is provided with an explanation of "Digital Transformation" and related materials (e.g., "DX Introduction Guide.pdf").

[0191] Thus, the present invention provides a system that efficiently searches for and provides relevant materials based on meeting content, and realizes information provision that also takes into account the user's emotional state.

[0192] The following describes the processing flow.

[0193] Step 1:

[0194] The user uploads the meeting audio data (e.g., a .wav file) to their device. The audio data is, for example, named "Meetings_aug30.wav".

[0195] Step 2:

[0196] The device sends the uploaded audio data to a speech recognition API, which converts the audio data into text data. For example, this might result in "Meeting content: Market strategy and cost reduction."

[0197] Step 3:

[0198] The terminal receives the converted text data and performs preprocessing to remove unnecessary characters and noise, and then formats it. For example, unnecessary parts such as "um" or "well" are removed.

[0199] Step 4:

[0200] The device uses natural language processing tools to extract frequently occurring words from pre-processed text data. For example, "market," "strategy," and "cost reduction" might be extracted.

[0201] Step 5:

[0202] The terminal sends the extracted list of frequently occurring words to the server. This list includes each frequently occurring word and its frequency of occurrence.

[0203] Step 6:

[0204] The server searches for documents within the company database based on the list of frequently occurring words it receives. It then calculates the synchronization rate between the content of each document and the list of frequently occurring words.

[0205] Step 7:

[0206] The server sorts the documents with the highest synchronization rate by newest first. For example, "Strategic Plan 2023.pdf" and "Latest Market Analysis Report 2023.pptx" are output.

[0207] Step 8:

[0208] The server generates links to a sorted list of materials and provides them to the user via the terminal. This allows the user to easily access relevant materials.

[0209] Step 9:

[0210] The device uses an emotion engine to analyze the user's emotions in real time. For example, it performs facial recognition and voice tone analysis to collect data on the emotional state the user exhibits while viewing materials.

[0211] Step 10:

[0212] The device sends emotional data to the server. For example, if a user expresses positive emotions towards "marketing strategy," that data is sent to the server.

[0213] Step 11:

[0214] The server re-evaluates the priority of previously searched materials based on sentiment data. Materials related to topics showing positive sentiment will appear higher in the search results.

[0215] Step 12:

[0216] If a server is showing negative emotions, resources to alleviate those emotions will be recommended. For example, resources that help reduce stress or solve problems will be selected and presented preferentially.

[0217] Step 13:

[0218] The server organizes all documents in the company's internal database by category. For example, they might be classified into categories such as "Strategy," "Market," and "Analysis."

[0219] Step 14:

[0220] The server searches for and presents the latest hot topics related to frequently occurring keywords. For example, "September Market Report.pdf" might be presented as a hot topic.

[0221] Step 15:

[0222] The device identifies terms that the user may not understand from among frequently used words. For example, "digital transformation" might be identified in this way.

[0223] Step 16:

[0224] The device provides explanations of terms it has identified and displays links to related materials. For example, it might provide an explanation such as, "Digital transformation is the process by which companies use digital technologies to transform their business models," along with a link to "DX Introduction Guide.pdf."

[0225] (Example 2)

[0226] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0227] Conventional systems that convert meeting audio into text data and provide related materials only utilize speech recognition and extraction of frequently occurring words, failing to provide information that takes user emotions into consideration. Therefore, there were issues such as users not receiving the information they needed appropriately and being unable to respond to changes in their emotions during use. Furthermore, features to deepen understanding of meeting content, such as category organization and glossary, were insufficient.

[0228] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0229] In this invention, the server includes means for converting audio data such as meetings into text data, means for extracting frequently occurring words from the converted text data, and means for searching for relevant materials from the company's internal database based on the extracted frequently occurring words and user sentiment data, and sorting them by synchronization rate and priority based on sentiment data. This makes it possible to search for and provide relevant materials that take into account the user's emotional state. Furthermore, by adding means for presenting the latest hot topics related to frequently occurring words and sentiment data, and means for identifying terms that the user may not understand and generating and providing explanations, more efficient and user-friendly information provision can be achieved.

[0230] 1. "Audio data" refers to data in digital file format that records audio, including oral communication such as meetings and interviews.

[0231] 2. "Text data" refers to character information converted from audio data, and is data expressed in a readable format.

[0232] 3. "Frequently occurring words" refer to words or phrases that appear particularly often in the text data and are extracted as important keywords.

[0233] 4. "Emotional data" refers to digital data generated as a result of analyzing a user's emotional state (e.g., positive, negative, excited, etc.).

[0234] 5. An "internal database" is a digital storage system used to accumulate and manage various documents and data within a company.

[0235] 6. "Related materials" refer to highly relevant documents and digital content searched from the company's internal database based on the extracted frequently occurring words and sentiment data.

[0236] 7. "Synchronization rate" is an indicator that shows how well the extracted frequently occurring words match the content of related materials.

[0237] 8. "Priority" is an indicator that influences the display order of search results and is determined based on the match rate of frequently occurring words and sentiment data.

[0238] 9. A "link" is a hypertext-based reference that allows a user to navigate to a specific document or webpage by clicking on it.

[0239] 10. A "category" is a classification of related materials based on subject matter or attributes, and is a concept designed to enable users to efficiently search for and utilize information.

[0240] 11. "Hot topics" refer to the latest information and topics considered to be of high interest, and are determined based on user interest and sentiment data.

[0241] 12. A "glossary" is a document that clearly explains the meaning and background of specific terms that users may not understand.

[0242] This invention is a system that converts audio data from meetings and other events into text data, extracts frequently occurring words from that text data, and searches for and provides related materials from the company's internal database based on those words. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it can also provide materials and change their priority according to the user's emotional state.

[0243] Hardware and software to be used

[0244] Terminal: A device on which users upload audio data and perform tasks such as displaying materials and emotion recognition. Examples include PCs and smartphones.

[0245] Server: A central server that performs tasks such as converting audio data to text, extracting frequently occurring words, searching for related materials, and prioritizing the order of materials.

[0246] software:

[0247] Speech-to-Text API: Google Cloud Speech-to-Text

[0248] Natural language processing tools: NLTK, spaCy

[0249] Emotion engine: Microsoft® Azure® Emotion API

[0250] Database search engine: Elasticsearch

[0251] Acquisition of audio data and text conversion

[0252] The user uploads meeting audio data (e.g., a .wav file) to their device. The device sends this audio data to a speech recognition API, which converts the audio data into text data. The converted text data is then formatted within the device into a specific format (e.g., JSON format).

[0253] Specific example:

[0254] The user clicks the "Select audio file" button, selects "Meetings_aug30.wav" from their local file system, and uploads it.

[0255] The device sends this audio file to the Google Cloud Speech-to-Text API, where it is converted into text data. For example, the converted text data might read, "Discussing future market strategy guidelines."

[0256] Extraction of frequently occurring words

[0257] The device uses natural language processing tools to extract frequently occurring words from the converted text data. For example, keywords such as "market" and "strategy" are extracted. The extracted keywords are then compiled into a list.

[0258] Specific example:

[0259] The terminal analyzes text data using NLTK and spaCy to extract frequently occurring words such as "market" and "strategy."

[0260] The extracted keywords are listed. For example, the list might look like this: ["market", "strategy"].

[0261] Utilizing the Emotion Engine

[0262] The device utilizes an emotion engine to recognize the user's emotions in real time. For example, it analyzes the user's facial expressions and voice tone while they are viewing documents and generates emotion data.

[0263] Specific example:

[0264] The device captures the user's facial expressions with its camera and acquires their voice tone with its microphone.

[0265] This data is sent to the Microsoft Azure Emotion API, and the generated emotion data is received. For example, the data might look like this: {"emotion": "positive", "confidence": 0.92}.

[0266] Searching for and providing related materials

[0267] Next, the terminal sends the extracted list of frequently occurring words and sentiment data to the server. The server searches for documents in the company's database based on the received list of frequently occurring words and sentiment data. The search results are sorted considering the degree of match between the content of each document and the frequently occurring words, as well as their importance based on the user's sentiment data.

[0268] Specific example:

[0269] The server uses Elasticsearch to search for materials that match "market" and "strategy".

[0270] For example, "Strategic Plan 2023.pdf" and "Market Analysis Report 2023.pptx" are searched, and their priority is re-evaluated based on relevance and sentiment data.

[0271] Categorizing materials and providing hot topics

[0272] The server also has the function of organizing all documents in the company's internal database into categories based on themes and attributes. This allows for efficient management of documents by category, such as "market analysis" and "strategy formulation." Furthermore, it selects and presents the latest hot topics to users based on sentiment data.

[0273] Specific example:

[0274] Based on sentiment data, the latest "September Updated Market Report.pdf" is presented as a hot topic.

[0275] Explanation of terminology and improvement of employee skills

[0276] The terminal discriminates terms that the user may not understand from the listed frequently-occurring words. For example, if the frequently-occurring word "digital transformation" is included, based on past data and the user's skill level, this term is discriminated as unclear. The terminal generates an easy-to-understand explanation regarding this term and also displays a link to related materials (e.g., "DX Introduction Guide.pdf").

[0277] Specific example:

[0278] The terminal discriminates the keyword "digital transformation" and displays a link to the related explanatory text and materials (e.g., "DX Introduction Guide.pdf").

[0279] Example of a prompt sentence:

[0280] The audio data of the meeting has been uploaded. Frequently-occurring words such as "market strategy" and "future direction" have been extracted. The user's sentiment is positive, and the user shows interest in materials related to the market strategy. Please search for and sort the related materials. Also, please provide an explanation and related materials about "digital transformation".

[0281] The flow of the specific process in Example 2 will be described using FIG. 13.

[0282] Step 1: Upload of audio data

[0283] The user uploads the audio data of the meeting (e.g.,.wav file) to the terminal. The input is an audio file, and the output is a notification of successful upload to the terminal. The terminal saves the input audio data in temporary storage and prepares to send it to the speech recognition API in the next step.

[0284] Specific operations:

[0285] The user clicks the "Select Audio File" button and selects "Meetings_aug30.wav" from the local file system.

[0286] The terminal accepts the file and displays a notification indicating that the upload is complete.

[0287] Step 2: Text Conversion of Audio Data

[0288] The terminal sends the audio data to an audio recognition API (e.g., Google Cloud Speech-to-Text) for conversion to text data. The input is audio data, and the output is the converted text data (e.g., in JSON format). The terminal receives the texturized data and formats it into a specific format.

[0289] Specific Operations:

[0290] The terminal sends "Meetings_aug30.wav" to the Google Cloud Speech-to-Text API.

[0291] The terminal receives the text data returned from the API and formats the data in a form such as {"text": "Discuss the future market strategy guidelines"}.

[0292] Step 3: Extraction of Frequently Occurring Words

[0293] The terminal extracts frequently occurring words from the converted text data using natural language processing tools (e.g., NLTK, spaCy). The input is text data, and the output is a list of frequently occurring words. The terminal analyzes the text data to extract the main keywords of the content discussed by the user.

[0294] Specific Operations:

[0295] The terminal analyzes the text data using NLTK or spaCy and extracts frequently occurring words such as "market" and "strategy".

[0296] The terminal lists the extracted keywords. For example, it generates a list such as ["market", "strategy"].

[0297] Step 4: Analysis using the emotion engine

[0298] The device uses an emotion engine (e.g., Microsoft Azure Emotion API) to recognize the user's emotions in real time. The input is the user's facial expressions and voice tone, and the output is emotion data. The device analyzes the collected data to confirm the user's emotional state.

[0299] Specific actions:

[0300] The device captures the user's facial expressions with its camera and records their voice tone with its microphone while they are viewing the document.

[0301] The device sends the captured data to the Microsoft Azure Emotion API and receives emotion data. For example, it might receive data such as {"emotion": "positive", "confidence": 0.92}.

[0302] Step 5: Search and sort related materials

[0303] The terminal sends the extracted list of frequently occurring words and sentiment data to the server. The input is the list of frequently occurring words and sentiment data, and the output is a sorted list of related documents. The server accesses the internal database, searches for the relevant documents, and sorts them based on priority.

[0304] Specific actions:

[0305] The device sends a list of frequently occurring words and sentiment data to the server.

[0306] The server uses Elasticsearch to search for materials that match the terms "market" and "strategy".

[0307] The server sorts the search results based on the degree of match of frequently occurring words and sentiment data. For example, "Strategic Plan 2023.pdf" and "Market Analysis Report 2023.pptx" are displayed at the top.

[0308] Step 6: Display and provide materials

[0309] The server sends the sorted list of materials to the terminal. The input is the sorted list of materials, and the output is the link to the materials displayed to the user. The terminal displays the link to the materials to the user based on the search results.

[0310] Specific operations:

[0311] The server generates a sorted list of materials and sends it to the terminal.

[0312] The terminal displays the link to the materials on the user's screen. For example, "Strategic Plan 2023.pdf", "Market Analysis Report 2023.pptx", and "September Updated Market Report.pdf" are displayed.

[0313] Step 7: Explanation of terms and provision of related materials

[0314] The terminal discriminates terms that the user may not understand from the frequently occurring words and generates explanations. The input is the frequently occurring words, and the output is the link to the term explanation and related materials. Generate an explanatory text for the discriminated terms and display the link to the related materials to the user.

[0315] Specific operations:

[0316] The terminal discriminates keywords such as "digital transformation".

[0317] The terminal generates an explanatory text for "digital transformation" and a link to related materials (e.g., "DX Introduction Guide.pdf").

[0318] The user's screen displays explanatory text and links to related resources.

[0319] (Application Example 2)

[0320] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0321] Current autonomous vehicles offer limited information tailored to the interests and emotions of passengers and users, and lack systems that provide relevant information in real time based on conversation. This makes it difficult to improve passenger convenience and satisfaction. Furthermore, there is a lack of appropriate means to provide information about terminology that passengers do not understand.

[0322] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for converting audio data such as meetings into text data, means for extracting frequently occurring words from the converted text data, means for searching for related materials from a database based on the extracted frequently occurring words and arranging them in order of synchronization rate, means for generating links to the arranged materials, means for converting conversations of passengers or users into text data, extracting frequently occurring words from the text data and recognizing the emotional state of passengers in real time, and means for searching for related content based on the extracted frequently occurring words and emotional data, and displaying it with priority. As a result, relevant information can be obtained in real time from conversations that passengers have in an autonomous vehicle, and information can be provided while also taking into account the emotions of passengers. In addition, by appropriately providing explanations of terms that passengers do not understand, passenger satisfaction and convenience can be greatly improved.

[0323] "Audio data from meetings, etc." refers to audio information recorded during meetings or other similar gatherings.

[0324] "Means of converting to text data" refers to methods or techniques for converting audio data into text information.

[0325] "Frequently occurring words" refer to words or phrases that appear frequently in text data.

[0326] "Emotional state" refers to data that indicates the user's emotions, such as positive or negative, excited or relieved.

[0327] A "database" is an electronic storage facility for systematically saving and managing related materials and information.

[0328] "Related materials" refer to information and documents that are related to the frequently extracted words.

[0329] "Synchronization rate" is an indicator that shows the degree of agreement or relevance between extracted frequently occurring words and related materials.

[0330] An "emotion engine" is a technology or software that analyzes a user's emotions in real time and generates data based on those emotions.

[0331] "Means of prioritizing display" refers to methods and techniques for sorting and displaying related information or content according to specific criteria.

[0332] "Conversation" is the process by which multiple people communicate with each other through sound.

[0333] This invention relates to a system primarily applied in autonomous vehicles that converts passenger and user conversations into text data in real time, extracts frequently occurring words, recognizes emotional states, and provides relevant information based on the results. The system is implemented based on the following procedure and configuration.

[0334] System Configuration

[0335] 1. Acquisition of audio data and conversion to text:

[0336] The server uses microphones inside the autonomous vehicle to acquire audio data of conversations between passengers and users. The acquired audio data is then converted into text data using the Google Speech-to-Text API.

[0337] 2. Extraction of frequently occurring words:

[0338] The server extracts frequently occurring words from text data using the NLTK library (Python), a natural language processing (NLP) tool.

[0339] 3. Recognition of emotional states:

[0340] The server uses Microsoft Azure's Emotion API to recognize the emotional state of passengers and users in real time and generate emotional data.

[0341] 4. Searching for and providing related information:

[0342] Based on the extracted frequent words and sentiment data, the server searches the Elasticsearch database to find relevant content (e.g., music, news, tourist information, advertisements, etc.), prioritizes it, and displays it.

[0343] Program Processing Overview

[0344] 1. Processing of audio data:

[0345] The audio data is sent from the microphone to the server and converted into text data using the Google Speech-to-Text API. The following is an example of a specific prompt.

[0346] Example prompt:

[0347] Based on the audio data, transcribe the content into text, extract frequently occurring words, obtain sentiment data using a sentiment engine, search for related content using Elasticsearch, and display it according to priority.

[0348] 2. Processing text data:

[0349] The NLTK library is used to extract frequently occurring words from the converted text data. The extracted keywords are important words that reflect the conversation content and are used in the search query.

[0350] 3. Processing of emotional data:

[0351] Text data is sent to the emotion engine, and emotion data is generated using Azure's Emotion API. This emotion data will play a crucial role in the subsequent information provision process.

[0352] 4. Searching for and displaying related content:

[0353] Based on the extracted frequent words and sentiment data, the server searches the Elasticsearch database to find relevant content. The content is sorted according to priority based on sentiment state and displayed on the smart display. For example, if it is recognized that a passenger is "talking about restaurants nearby," restaurant information will be displayed preferentially.

[0354] Specific example

[0355] Let's say a passenger is talking on the train and says, "I want to know about tourist attractions in this area." The server captures this conversation audio and uses the Google Speech-to-Text API to convert "tourist attractions in this area" into text. Using the NLTK library, "tourist attractions" is extracted as a key keyword, and the passenger's level of interest is further analyzed by an emotion engine. The Elasticsearch database is searched, and information based on the keyword "tourist attractions" is displayed preferentially. For example, "popular tourist attractions in this area" or "recommended tourist destinations list" might be displayed on the smart display.

[0356] Thus, this invention realizes a system that can improve passenger convenience and satisfaction in autonomous vehicles by providing information in real time that corresponds to the content of passengers' or users' conversations and emotional states.

[0357] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0358] Step 1:

[0359] The server acquires audio data using microphones inside the autonomous vehicle. The acquired audio data is saved as is. Specifically, it is recorded in an audio file format (e.g., a .wav file).

[0360] Input: Audio of conversations between passengers and users inside the vehicle.

[0361] Output: Audio data file (.wav)

[0362] Step 2:

[0363] The server sends the audio data to the Google Speech-to-Text API, where it is converted into text data. The converted text data is then stored for subsequent processing and formatted into a concise form.

[0364] Input: Audio data file (.wav)

[0365] Output: Text data

[0366] Step 3:

[0367] The server uses the NLTK library to extract frequently occurring words from text data. It tokenizes (segments) the text data, calculates the frequency of each word, and picks out the most frequent words.

[0368] Input: Text data

[0369] Output: List of frequently occurring words

[0370] Step 4:

[0371] The server sends the extracted text data to the emotion engine, which uses Microsoft Azure's Emotion API to generate emotion data. This emotion data provides a detailed description of the user's emotional state, such as positive, negative, or excited.

[0372] Input: Text data

[0373] Output: Sentiment data

[0374] Step 5:

[0375] The server searches the Elasticsearch database based on the extracted list of frequent words and sentiment data. The extracted frequent words are used in the search queries, and the search results are prioritized based on the sentiment data.

[0376] Input: List of frequently occurring words, sentiment data

[0377] Output: Related Content List

[0378] Step 6:

[0379] The server selects relevant content from the search results in order of priority and displays it on the smart display. For example, if a passenger is talking about "restaurants," information about nearby restaurants will be displayed.

[0380] Input: Related Content List

[0381] Output: Content displayed on the smart display

[0382] Step 7:

[0383] The system uses a terminal to identify terms that passengers may not understand and generates and displays explanations of those terms along with links to related resources. For example, if the term "digital transformation" is included, an explanation and a link to more detailed resources will be displayed.

[0384] Input: List of frequently occurring words

[0385] Output: Glossary of terms, links to related resources

[0386] The above outlines the specific processing steps for implementing this invention. This enables the real-time provision of information tailored to the conversation content and emotional state of passengers and users.

[0387] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0388] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0389] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0390] [Second Embodiment]

[0391] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0392] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0393] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0394] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0395] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0396] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0397] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0398] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0399] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0400] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0401] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0402] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0403] This invention relates to a system that converts audio data, such as from meetings, into text data, extracts frequently occurring words from that text data, and searches for and provides related materials from an internal company database based on those words. It also provides explanations of the latest hot topics related to the frequently occurring words, as well as terms that employees may not be familiar with.

[0404] Acquisition of audio data and text conversion

[0405] The user uploads meeting audio data (e.g., a .wav file) to the device. The device sends this audio data to a speech recognition API, which converts the audio data into text data. The converted text data is then formatted into a specific format within the device.

[0406] Extraction of frequently occurring words

[0407] The terminal uses natural language processing tools to extract frequently occurring words from the converted text data. For example, if the meeting content was "discussing future market strategies," keywords such as "market" and "strategy" would be extracted.

[0408] Searching for and providing related materials

[0409] Next, the terminal sends the extracted list of frequently occurring words to the server. The server searches for documents in the company's database based on the received list of frequently occurring words. The search results are sorted by calculating the synchronization rate between the content of each document and the frequently occurring words, and then sorting them in descending order of synchronization rate. As a result, documents such as "Strategic Plan 2023.pdf" and "Market Analysis Report 2023.pptx" are listed in reverse chronological order. The server generates links to these sorted documents and provides them to the user.

[0410] Categorizing materials and providing hot topics

[0411] The server also has the function of organizing all documents in the company's internal database into categories based on themes and attributes. This allows for efficient management of documents by category, such as "Market Analysis" and "Strategy Formulation." Furthermore, it selects and presents the latest documents related to frequently used keywords as hot topics. For example, within the "Market Analysis" category, the latest "September Updated Market Report.pdf" is presented as a hot topic.

[0412] Explanation of terminology and improvement of employee skills

[0413] The device identifies terms that the user may not understand from a list of frequently used words. For example, if the frequently used word "digital transformation" is included, the device will determine, based on past data and the user's skill level, that this term is unfamiliar. The device will generate an easy-to-understand explanation of this term and also display a link to related materials (e.g., "DX Introduction Guide.pdf").

[0414] Explanation with specific examples

[0415] The following is a concrete example. An audio file, "Meetings_aug30.wav," from a meeting held on August 30th is uploaded to the terminal. The terminal uses a speech recognition API to convert this audio file into text, and the content is rephrased as "Discussion on future market strategy policies." A natural language processing tool is used to extract frequently occurring words such as "market" and "strategy," and this is sent to the server. The server searches the company's internal database for related documents such as "Strategic Plan 2023.pdf" and "Market Analysis Report 2023.pptx," sorts them in reverse chronological order, and generates links. Furthermore, it presents the latest "September Market Report.pdf" as a hot topic related to "market analysis," and provides the user with an explanation of "digital transformation" and related materials.

[0416] Thus, the present invention provides a system that efficiently searches for and provides relevant materials based on meeting content, and contributes to improving employees' knowledge.

[0417] The following describes the processing flow.

[0418] Step 1:

[0419] The user uploads the meeting audio data (e.g., a .wav file) to their device. The audio data is in a format such as "Meetings_aug30.wav".

[0420] Step 2:

[0421] The device sends the uploaded audio data to a speech recognition API, which converts the audio data into text data. This is where text such as "Strategic Meeting August 30, 2023" is generated.

[0422] Step 3:

[0423] The terminal receives the converted text data and formats it by removing unnecessary information (such as noise or irrelevant conversation). For example, unnecessary words like "um" or "well" are removed.

[0424] Step 4:

[0425] The device uses natural language processing tools to extract frequently occurring words from the formatted text. For example, keywords such as "strategy," "market," and "analysis" might be extracted.

[0426] Step 5:

[0427] The terminal sends the extracted list of frequently occurring words to the server. This list includes not only the frequently occurring words but also their frequency of occurrence.

[0428] Step 6:

[0429] Based on the frequently occurring word list received by the server, all documents in the company database are searched. The degree of agreement between the document content and the frequently occurring words, i.e., the synchronization rate, is calculated.

[0430] Step 7:

[0431] The server sorts the documents with the highest synchronization rate by newest first. For example, "Strategic Plan 2023.pdf" and "Latest Market Analysis Report 2023.pptx" are output.

[0432] Step 8:

[0433] The server generates links to a sorted list of materials and provides them to the user via the terminal. This allows the user to easily access relevant materials.

[0434] Step 9:

[0435] The server organizes documents within the company's internal database by category. For example, they might be divided into categories such as "Strategy," "Market," and "Analysis."

[0436] Step 10:

[0437] The server searches for and presents the latest hot topics related to frequently occurring keywords. For example, "September Market Report.pdf" might be presented as a hot topic within the "Market Analysis" category.

[0438] Step 11:

[0439] The device identifies terms that the user may not understand from among frequently used words. For example, "digital transformation" might be identified in this way.

[0440] Step 12:

[0441] The device provides explanations of terms it has identified and links to related materials. For example, it might display an explanation such as, "Digital transformation is the process by which companies use digital technologies to transform their business models," along with a link to "DX Introduction Guide.pdf".

[0442] (Example 1)

[0443] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0444] Traditional methods of recording and analyzing meetings made it difficult to efficiently utilize audio data, requiring significant time and effort for manual text conversion and document retrieval. Furthermore, the accuracy and speed of searching for and providing relevant materials were limited, and explanations of unfamiliar terminology were insufficient. Therefore, a system was needed that could simultaneously achieve effective utilization of meeting data and improve knowledge.

[0445] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0446] In this invention, the server includes means for converting audio data into text data, means for extracting frequently occurring words from the converted text data, means for searching a database for related information based on the extracted frequently occurring words and arranging it in order of decreasing synchronization rate, means for generating links to the arranged information, means for using an external API for recognizing audio data, and means for using a natural language processing tool for extracting frequently occurring words. This makes it possible to efficiently transcribe meeting audio data into text, provide related materials quickly and accurately, and simultaneously improve employees' knowledge.

[0447] "Audio data" refers to digital data files containing recordings of audio from meetings, conversations, and other similar events.

[0448] "Text data" refers to a digital data file created by converting audio data into text.

[0449] "Frequently occurring words" refer to words or expressions that appear with a specific frequency within text data.

[0450] An "external API" refers to an application programming interface provided by an external service or platform.

[0451] "Natural language processing tools" refer to software and libraries used to analyze text data, understand its meaning and grammar, and process it.

[0452] A "database" refers to a collection of information that allows for the efficient management and retrieval of large amounts of data.

[0453] A "link" is a hypertext element that allows a user to access a specific webpage or file by clicking on it.

[0454] "Synchronization rate" refers to an indicator that shows the degree of agreement between frequently occurring words and the content of the materials in the database.

[0455] "Explanation" refers to a text or explanation that clearly describes unfamiliar or unknown terms.

[0456] This invention is a system that converts audio data, such as from meetings, into text data, extracts frequently occurring words from that text data, and searches and provides related materials from a database based on those words. Furthermore, it also provides the latest topics related to the frequently occurring words and explanations of terms that the user may not be familiar with. Embodiments of this system are described below.

[0457] First, the user uploads the meeting audio data (e.g., a .wav file) to their device. The device then sends this audio data to a speech recognition API (e.g., Google Speech-to-Text or Amazon Transcribe) to convert it into text data. The converted text data is then formatted into a specific format (e.g., date and time, speaker, content of speech) and processed.

[0458] Next, the terminal inputs the converted text data into a natural language processing tool (e.g., NLTK or spaCy) to extract frequently occurring words. For example, if the meeting topic is "discussing future market strategies," frequently occurring words such as "market" and "strategy" will be extracted. The extraction results are compiled into a list of frequently occurring words.

[0459] The terminal sends the extracted list of frequently occurring words to the server. The server searches the database using the ElasticSearch engine or similar based on the list. The search results are sorted by calculating the synchronization rate between the content of each document and the frequently occurring words, and then sorted in descending order of synchronization rate. This results in a list of documents such as "Strategic Plan 2023.pdf" and "Market Analysis Report 2023.pptx," and links to the sorted documents are generated.

[0460] Furthermore, the server organizes all documents in the company database by theme (e.g., "Market Analysis" or "Strategy Formulation"), selects the latest information related to frequently used keywords, and presents it to the user. For example, within the "Market Analysis" category, the latest "September Updated Market Report.pdf" might be presented as a hot topic.

[0461] Furthermore, the device identifies terms from a list of frequently used words that the user may not understand, based on past learning data. For example, if the frequently used word "digital transformation" is identified as unfamiliar, it generates an easy-to-understand explanation and provides a link to related materials (e.g., "DX Introduction Guide.pdf").

[0462] Specific example:

[0463] An audio file, "Meetings_aug30.wav," from a meeting held on August 30th is uploaded to the user's device. This audio file is converted into text data via a speech recognition API on the device. The content is translated to something like "Discussion on future market strategy." Next, a natural language processing tool is used to extract frequently occurring words such as "market" and "strategy," and this is sent to the server. The server searches its database for related documents such as "Strategic Plan 2023.pdf" and "Market Analysis Report 2023.pptx," sorts them in reverse chronological order, and generates links. Furthermore, the latest "September Market Report.pdf" is presented as a hot topic related to "market analysis," and the user is provided with an explanation of "digital transformation" and related materials.

[0464] Example of a prompt

[0465] The following are specific examples of prompt statements that will be used as input to the generative AI model.

[0466] "Please describe a program to search for and provide materials related to the topics of 'market' and 'strategy' discussed at the August 30th meeting. Additionally, please explain the term 'digital transformation' and add a function to provide related materials."

[0467] This system enables efficient transcription of meeting content, prompt and accurate provision of related materials, and contributes to improving employees' knowledge.

[0468] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0469] Step 1: Acquire audio data and convert it to text.

[0470] The user uploads meeting audio data (e.g., a .wav file) to their device. The input is an audio data file. The device sends this audio data to the Google Speech-to-Text API and retrieves text data. The output is the converted text data. Specifically, the audio data is passed to the API, and the text data is received as the API response.

[0471] Step 2: Formatting the text data

[0472] The terminal formats the acquired text data into a specific format (e.g., date and time, speaker, content of the message). The input is text data obtained from the API. The output is the formatted text data. Specifically, the terminal organizes the text data using methods such as JSON or CSV format to make it suitable for subsequent processing.

[0473] Step 3: Extraction of frequently occurring words

[0474] The terminal inputs formatted text data into a natural language processing tool (e.g., NLTK or spaCy) and extracts frequently occurring words. The input is formatted text data. The output is a list of frequently occurring words. Specifically, the text data is tokenized, the frequency of each word is counted, and words exceeding a certain frequency are extracted.

[0475] Step 4: Send the list of frequently used words

[0476] The terminal sends the extracted list of frequently occurring words to the server. The input is the list of frequently occurring words. The output is the data sent to the server. Specifically, the process involves encoding the list of frequently occurring words in JSON format and sending it to the server using an HTTP request.

[0477] Step 5: Search for related materials

[0478] The server uses the Elasticsearch engine to search for materials in the database based on the received list of frequently occurring words. The input is the list of frequently occurring words. The output is the search results. Specifically, it generates an Elasticsearch query and executes the query against the database.

[0479] Step 6: Calculate and sort the data synchronization rate.

[0480] The server calculates the synchronization rate between the content of each document and its frequently occurring words, and sorts them in descending order of synchronization rate. The input is the search results. The output is the sorted list of documents. Specifically, it scores how well the frequently occurring words match the content of each document and sorts the documents based on that score.

[0481] Step 7: Generate and provide links

[0482] The server generates and provides links to sorted documents to the user. The input is a sorted list of documents. The output is links to the documents. Specifically, the server generates the links in HTML format, making them accessible to the user.

[0483] Step 8: Categorize materials and provide hot topics

[0484] The server organizes all documents in the company database by theme, selects the latest information related to frequently used keywords, and presents it to the user. Inputs are a list of documents and frequently used keywords. Outputs are documents organized by category and the latest hot topics. Specifically, it uses a classification algorithm to group documents by theme and selects the most recent documents.

[0485] Step 9: Terminology identification and explanation generation

[0486] The device identifies terms that the user may not understand from a list of frequently used words, generates easy-to-understand explanations for the unfamiliar terms, and displays links to related materials. Input consists of the list of frequently used words and the user's past learning data. Output is the term explanation and links. Specifically, it executes a discrimination algorithm based on the learning data, generates explanatory text, and attaches links to related materials.

[0487] (Application Example 1)

[0488] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0489] While it is crucial to respond quickly and accurately to customer questions and requests during customer service in stores, delays can occur when staff are busy or when there is inconsistency in product knowledge. Furthermore, insufficient information provided to customers can lead to decreased customer satisfaction. Additionally, it is difficult for new or part-time staff to fully grasp specialized terminology and detailed product information, which can result in a decline in the quality of customer service. A system is needed to address these challenges and improve customer satisfaction.

[0490] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0491] This invention includes a server that transcribes audio data of customer conversations into text in real time and searches for and provides related product information based on frequently occurring words; a server that searches for related materials from a database based on the extracted frequently occurring words and arranges them in order of synchronization rate; and a server that generates explanations for the identified terms and presents links to related materials. This makes it possible to convert conversations with customers into text data and quickly provide related product information and detailed explanations based on frequently occurring questions and requests.

[0492] "Audio data" refers to sound information, such as conversations and music, recorded and stored in digital format.

[0493] "Text data" refers to a digital representation of character information.

[0494] "Frequently occurring words" refer to words or phrases that appear particularly often within the text data.

[0495] A "database" is a system for efficiently managing and retrieving data, and is a collection of structured information.

[0496] "Synchronization rate" is an indicator that shows the degree of agreement between frequently occurring words and the data in the database.

[0497] A "link" is reference information that allows direct access to specific data or materials.

[0498] "Customer service" refers to the activities of interacting with and providing services to customers in stores and service industries.

[0499] "Real-time text conversion" is the process of instantly converting audio data into text data.

[0500] "Related product information" refers to detailed data and descriptions of products found through searches based on frequently occurring keywords.

[0501] "Category organization" is the process of classifying and managing data and materials based on common attributes or themes.

[0502] "Hot topics" refer to the latest information or topics that have recently become popular.

[0503] An "explanation" is a detailed explanation of a specific term or concept.

[0504] System program

[0505] The apparatus for carrying out this invention consists of a system in which a server, a terminal, and a user work together.

[0506] The terminal has the functionality to acquire voice data during customer service and convert it into text data. To achieve this, the Google Cloud Speech-to-Text API and Python's speech_recognition library are used. In addition, frequently occurring words are extracted from the converted text data using natural language processing tools (e.g., spaCy, NLTK).

[0507] The server has the functionality to search for related materials and product information from existing databases (e.g., Firebase) based on a list of frequently used words sent from the terminal, sort them in order of synchronicity, and generate links. Furthermore, it also has the functionality to present hot topics related to frequently used words.

[0508] Program Processing Description

[0509] 1. Acquisition and transcription of audio data

[0510] The system records conversations during customer service using a terminal and sends the recordings to the Google Cloud Speech-to-Text API, converting the audio data into text data. This makes the audio information easily searchable text data.

[0511] 2. Extraction of frequently occurring words

[0512] The converted text data is analyzed using natural language processing tools (NLTK, spaCy) to extract frequently occurring words. This allows for a quick understanding of customer interests and question trends.

[0513] 3. Searching for and providing related information

[0514] The device sends the extracted list of frequently occurring words to the server. The server searches its database (Firebase) for relevant product information and materials, sorts them in the optimal order based on the synchronization rate, and generates links.

[0515] 4. Explanation of hot topics and terminology

[0516] The server generates and presents detailed explanations, along with links, of the latest hot topics related to frequently used words and terms that customers may not understand.

[0517] Examples of specific cases and prompt statements

[0518] Specific example:

[0519] For example, if frequently occurring keywords such as "new product" and "promotion" are extracted from an audio file recorded during customer service on August 30th, the server will search for detailed information on products related to these keywords and the latest campaign information, and present them in a list sorted by the degree of synchronization. Furthermore, if it is determined that the term "promotion" is not well understood by new staff, an explanation of the term and links to related materials will also be provided.

[0520] Example of a prompt:

[0521] "We want to develop an application that transcribes audio data of customers being served on August 30th into text, extracts frequently occurring words, and then searches for and provides related product information and the latest promotional information."

[0522] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0523] Step 1:

[0524] The terminal acquires audio data during customer service. The user (store clerk) uses a smartphone or dedicated device to record the audio data of the conversation. This audio data becomes the input data, and in this step, an audio file is specifically generated.

[0525] Step 2:

[0526] The device sends the acquired audio data to the Google Cloud Speech-to-Text API, where it is converted into text data. The input here is the audio file obtained in step 1, and the API is used to convert that audio into text format and output it. Specifically, the audio data is sent to the API, and the converted text is returned from the API.

[0527] Step 3:

[0528] The terminal analyzes the converted text data using a natural language processing tool (e.g., spaCy or NLTK) and extracts frequently occurring words. The input is the text data obtained in step 2, and data processing such as word segmentation and part-of-speech analysis is performed using NLTK or spaCy to extract and output frequently occurring words. Specifically, the text is broken down, and the words that appear most frequently are listed.

[0529] Step 4:

[0530] The terminal sends the extracted list of frequently occurring words to the server. The input is the list of frequently occurring words obtained in step 3, and the terminal sends data to the server based on this. Specifically, the operation to send the list of frequently occurring words to the API endpoint is performed.

[0531] Step 5:

[0532] The server searches for relevant documents and product information from a database (such as Firebase) based on the list of frequently occurring words. The input for this step is the list of frequently occurring words submitted in step 4. The server searches the database to retrieve relevant documents and product information and outputs it as a list sorted by synchronization rate. Specifically, it forms a search query using frequently occurring words as keywords and extracts the corresponding data from the database.

[0533] Step 6:

[0534] The server generates links to relevant documents and product information from the search results and sends them to the terminal. The input is the search results obtained in step 5, and it generates and outputs links to the detail pages for each document and product. Specifically, it uses a link generation algorithm to form URLs, lists them, and sends them.

[0535] Step 7:

[0536] The server identifies hot topics and terms that customers may not understand, and generates explanations and related materials. The input is the search results obtained in step 5 and the user profile, and it generates and outputs explanatory text and links to related materials. Specifically, it identifies unfamiliar terms from frequently occurring words and historical data, and provides explanatory text and links to related materials based on those terms.

[0537] Step 8:

[0538] The terminal displays the received link and explanation to the user (store clerk) and provides it to the customer. The input consists of the link and explanation generated in steps 6 and 7, and is output by being displayed on a screen such as a smartphone. Specifically, the link is displayed in a clickable format, and the explanation is displayed in an information box.

[0539] Through these steps, users can respond quickly and accurately to customer questions and requests, thereby improving customer satisfaction.

[0540] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0541] This invention relates to a system that converts audio data, such as from meetings, into text data, extracts frequently occurring words from that text data, and searches for and provides related materials from an internal company database based on those words. Furthermore, by combining this with an emotion engine that recognizes the user's emotions, the system can also provide materials and change their priority according to the user's emotional state.

[0542] Acquisition of audio data and text conversion

[0543] The user uploads meeting audio data (e.g., a .wav file) to the device. The device sends this audio data to a speech recognition API, which converts the audio data into text data. The converted text data is then formatted into a specific format within the device.

[0544] Extraction of frequently occurring words

[0545] The terminal uses natural language processing tools to extract frequently occurring words from the converted text data. For example, if the meeting content was "discussing future market strategies," keywords such as "market" and "strategy" would be extracted.

[0546] Utilizing the Emotion Engine

[0547] The device utilizes an emotion engine to recognize the user's emotions in real time. For example, the emotion engine analyzes the user's facial expressions and voice tone while they are viewing documents and generates emotion data.

[0548] Searching for and providing related materials

[0549] Next, the terminal sends the extracted list of frequently occurring words and sentiment data to the server. The server searches for documents in the company's database based on the received list of frequently occurring words and sentiment data. The search results are sorted using a new criterion that integrates synchronization rate and sentiment influence, taking into account the degree of match between the content of each document and the frequently occurring words, as well as importance based on the user's sentiment data.

[0550] For example, if a user expresses positive feelings towards "marketing strategy," materials related to that topic will be displayed at the top of the search results. On the other hand, if negative feelings are detected, materials with a higher priority for mitigating those feelings will be presented.

[0551] Categorizing materials and providing hot topics

[0552] The server also has the function of organizing all documents in the company's internal database into categories based on themes and attributes. This allows for efficient management of documents by category, such as "Market Analysis" and "Strategy Formulation." Furthermore, it selects and presents the latest hot topics based on sentiment data. For example, within the "Market Analysis" category, the latest "September Updated Market Report.pdf" is presented as a hot topic.

[0553] Explanation of terminology and improvement of employee skills

[0554] The device identifies terms that the user may not understand from a list of frequently used words. For example, if the frequently used word "digital transformation" is included, the device will determine, based on past data and the user's skill level, that this term is unfamiliar. The device will generate an easy-to-understand explanation of this term and also display a link to related materials (e.g., "DX Introduction Guide.pdf").

[0555] Explanation with specific examples

[0556] The following is a concrete example. An audio file, "Meetings_aug30.wav," from a meeting held on August 30th is uploaded to the terminal. The terminal uses a speech recognition API to transcribe this audio file into text, and the content is translated into something like "Discussion on future market strategy." A natural language processing tool is used to extract frequently occurring words such as "market" and "strategy," and an emotion engine analyzes the user's emotional state (e.g., excitement, interest, negative emotion) in real time. The extracted list of frequently occurring words and emotion data are sent to the server.

[0557] The server searches the internal database for relevant documents such as "Strategic Plan 2023.pdf" and "Market Analysis Report 2023.pptx," and re-evaluates their priority based on synchronization rate and sentiment data. The documents are then sorted in the new order, and links are generated. Furthermore, the latest "September Market Report.pdf" is presented as a hot topic, and the user is provided with an explanation of "Digital Transformation" and related materials (e.g., "DX Introduction Guide.pdf").

[0558] Thus, the present invention provides a system that efficiently searches for and provides relevant materials based on meeting content, and realizes information provision that also takes into account the user's emotional state.

[0559] The following describes the processing flow.

[0560] Step 1:

[0561] The user uploads the meeting audio data (e.g., a .wav file) to their device. The audio data is, for example, named "Meetings_aug30.wav".

[0562] Step 2:

[0563] The device sends the uploaded audio data to a speech recognition API, which converts the audio data into text data. For example, this might result in "Meeting content: Market strategy and cost reduction."

[0564] Step 3:

[0565] The terminal receives the converted text data and performs preprocessing to remove unnecessary characters and noise, and then formats it. For example, unnecessary parts such as "um" or "well" are removed.

[0566] Step 4:

[0567] The device uses natural language processing tools to extract frequently occurring words from pre-processed text data. For example, "market," "strategy," and "cost reduction" might be extracted.

[0568] Step 5:

[0569] The terminal sends the extracted list of frequently occurring words to the server. This list includes each frequently occurring word and its frequency of occurrence.

[0570] Step 6:

[0571] The server searches for documents within the company database based on the list of frequently occurring words it receives. It then calculates the synchronization rate between the content of each document and the list of frequently occurring words.

[0572] Step 7:

[0573] The server sorts the documents with the highest synchronization rate by newest first. For example, "Strategic Plan 2023.pdf" and "Latest Market Analysis Report 2023.pptx" are output.

[0574] Step 8:

[0575] The server generates links to a sorted list of materials and provides them to the user via the terminal. This allows the user to easily access relevant materials.

[0576] Step 9:

[0577] The device uses an emotion engine to analyze the user's emotions in real time. For example, it performs facial recognition and voice tone analysis to collect data on the emotional state the user exhibits while viewing materials.

[0578] Step 10:

[0579] The device sends emotional data to the server. For example, if a user expresses positive emotions towards "marketing strategy," that data is sent to the server.

[0580] Step 11:

[0581] The server re-evaluates the priority of previously searched materials based on sentiment data. Materials related to topics showing positive sentiment will appear higher in the search results.

[0582] Step 12:

[0583] If a server is showing negative emotions, resources to alleviate those emotions will be recommended. For example, resources that help reduce stress or solve problems will be selected and presented preferentially.

[0584] Step 13:

[0585] The server organizes all documents in the company's internal database by category. For example, they might be classified into categories such as "Strategy," "Market," and "Analysis."

[0586] Step 14:

[0587] The server searches for and presents the latest hot topics related to frequently occurring keywords. For example, "September Market Report.pdf" might be presented as a hot topic.

[0588] Step 15:

[0589] The device identifies terms that the user may not understand from among frequently used words. For example, "digital transformation" might be identified in this way.

[0590] Step 16:

[0591] The device provides explanations of terms it has identified and displays links to related materials. For example, it might provide an explanation such as, "Digital transformation is the process by which companies use digital technologies to transform their business models," along with a link to "DX Introduction Guide.pdf."

[0592] (Example 2)

[0593] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0594] Conventional systems that convert meeting audio into text data and provide related materials only utilize speech recognition and extraction of frequently occurring words, failing to provide information that takes user emotions into consideration. Therefore, there were issues such as users not receiving the information they needed appropriately and being unable to respond to changes in their emotions during use. Furthermore, features to deepen understanding of meeting content, such as category organization and glossary, were insufficient.

[0595] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0596] In this invention, the server includes means for converting audio data such as meetings into text data, means for extracting frequently occurring words from the converted text data, and means for searching for relevant materials from the company's internal database based on the extracted frequently occurring words and user sentiment data, and sorting them by synchronization rate and priority based on sentiment data. This makes it possible to search for and provide relevant materials that take into account the user's emotional state. Furthermore, by adding means for presenting the latest hot topics related to frequently occurring words and sentiment data, and means for identifying terms that the user may not understand and generating and providing explanations, more efficient and user-friendly information provision can be achieved.

[0597] 1. "Audio data" refers to data in digital file format that records audio, including oral communication such as meetings and interviews.

[0598] 2. "Text data" refers to character information converted from audio data, and is data expressed in a readable format.

[0599] 3. "Frequently occurring words" refer to words or phrases that appear particularly often in the text data and are extracted as important keywords.

[0600] 4. "Emotional data" refers to digital data generated as a result of analyzing a user's emotional state (e.g., positive, negative, excited, etc.).

[0601] 5. An "internal database" is a digital storage system used to accumulate and manage various documents and data within a company.

[0602] 6. "Related materials" refer to highly relevant documents and digital content searched from the company's internal database based on the extracted frequently occurring words and sentiment data.

[0603] 7. "Synchronization rate" is an indicator that shows how well the extracted frequently occurring words match the content of related materials.

[0604] 8. "Priority" is an indicator that influences the display order of search results and is determined based on the match rate of frequently occurring words and sentiment data.

[0605] 9. A "link" is a hypertext-based reference that allows a user to navigate to a specific document or webpage by clicking on it.

[0606] 10. A "category" is a classification of related materials based on subject matter or attributes, and is a concept designed to enable users to efficiently search for and utilize information.

[0607] 11. "Hot topics" refer to the latest information and topics considered to be of high interest, and are determined based on user interest and sentiment data.

[0608] 12. A "glossary" is a document that clearly explains the meaning and background of specific terms that users may not understand.

[0609] This invention is a system that converts audio data from meetings and other events into text data, extracts frequently occurring words from that text data, and searches for and provides related materials from the company's internal database based on those words. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it can also provide materials and change their priority according to the user's emotional state.

[0610] Hardware and software to be used

[0611] Terminal: A device on which users upload audio data and perform tasks such as displaying materials and emotion recognition. Examples include PCs and smartphones.

[0612] Server: A central server that performs tasks such as converting audio data to text, extracting frequently occurring words, searching for related materials, and prioritizing the order of materials.

[0613] software:

[0614] Speech-to-Text API: Google Cloud Speech-to-Text

[0615] Natural language processing tools: NLTK, spaCy

[0616] Emotion engine: Microsoft Azure Emotion API

[0617] Database search engine: Elasticsearch

[0618] Acquisition of audio data and text conversion

[0619] The user uploads meeting audio data (e.g., a .wav file) to their device. The device sends this audio data to a speech recognition API, which converts the audio data into text data. The converted text data is then formatted within the device into a specific format (e.g., JSON format).

[0620] Specific example:

[0621] The user clicks the "Select audio file" button, selects "Meetings_aug30.wav" from their local file system, and uploads it.

[0622] The device sends this audio file to the Google Cloud Speech-to-Text API, where it is converted into text data. For example, the converted text data might read, "Discussing future market strategy guidelines."

[0623] Extraction of frequently occurring words

[0624] The device uses natural language processing tools to extract frequently occurring words from the converted text data. For example, keywords such as "market" and "strategy" are extracted. The extracted keywords are then compiled into a list.

[0625] Specific example:

[0626] The terminal analyzes text data using NLTK and spaCy to extract frequently occurring words such as "market" and "strategy."

[0627] The extracted keywords are listed. For example, the list might look like this: ["market", "strategy"].

[0628] Utilizing the Emotion Engine

[0629] The device utilizes an emotion engine to recognize the user's emotions in real time. For example, it analyzes the user's facial expressions and voice tone while they are viewing documents and generates emotion data.

[0630] Specific example:

[0631] The device captures the user's facial expressions with its camera and acquires their voice tone with its microphone.

[0632] This data is sent to the Microsoft Azure Emotion API, and the generated emotion data is received. For example, the data might look like this: {"emotion": "positive", "confidence": 0.92}.

[0633] Searching for and providing related materials

[0634] Next, the terminal sends the extracted list of frequently occurring words and sentiment data to the server. The server searches for documents in the company's database based on the received list of frequently occurring words and sentiment data. The search results are sorted considering the degree of match between the content of each document and the frequently occurring words, as well as their importance based on the user's sentiment data.

[0635] Specific example:

[0636] The server uses Elasticsearch to search for materials that match "market" and "strategy".

[0637] For example, "Strategic Plan 2023.pdf" and "Market Analysis Report 2023.pptx" are searched, and their priority is re-evaluated based on relevance and sentiment data.

[0638] Categorizing materials and providing hot topics

[0639] The server also has the function of organizing all documents in the company's internal database into categories based on themes and attributes. This allows for efficient management of documents by category, such as "market analysis" and "strategy formulation." Furthermore, it selects and presents the latest hot topics to users based on sentiment data.

[0640] Specific example:

[0641] Based on sentiment data, the latest "September Updated Market Report.pdf" is presented as a hot topic.

[0642] Explanation of terminology and improvement of employee skills

[0643] The terminal identifies terms that the user may not understand from a list of frequently used words. For example, if the frequently used word "digital transformation" is included, the terminal will determine, based on past data and the user's skill level, that this term is unfamiliar. The terminal will generate an easy-to-understand explanation of this term and also display a link to related materials (e.g., "DX Introduction Guide.pdf").

[0644] Specific example:

[0645] The device recognizes the keyword "digital transformation" and displays links to related explanatory texts and documents (e.g., "DX Introduction Guide.pdf").

[0646] Example of a prompt:

[0647] I uploaded the meeting audio data. Frequently occurring words such as "market strategy" and "future plans" were extracted. The user's sentiment is positive, and they are interested in materials related to market strategy. Please search and sort the related materials. Also, please provide an explanation and related materials on "digital transformation."

[0648] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0649] Step 1: Upload audio data

[0650] The user uploads meeting audio data (e.g., a .wav file) to the device. The input is the audio file, and the output is a notification that the upload to the device is complete. The device saves the input audio data to temporary storage and prepares to send it to the speech recognition API in the next step.

[0651] Specific actions:

[0652] The user clicks the "Select audio file" button and selects "Meetings_aug30.wav" from the local file system.

[0653] The device accepts the file and displays a notification that the upload is complete.

[0654] Step 2: Convert audio to text

[0655] The device sends audio data to a speech recognition API (e.g., Google Cloud Speech-to-Text) and converts it into text data. The input is audio data, and the output is converted text data (e.g., in JSON format). The device receives the text data and formats it into a specific format.

[0656] Specific actions:

[0657] The device sends "Meetings_aug30.wav" to the Google Cloud Speech-to-Text API.

[0658] The device receives the text data returned from the API. For example, the data is formatted in the format {"text": "Discussion on future market strategy"}.

[0659] Step 3: Extraction of frequently occurring words

[0660] The terminal extracts frequently occurring words from the converted text data using natural language processing tools (e.g., NLTK, spaCy). The input is text data, and the output is a list of frequently occurring words. The terminal analyzes the text data to extract the main keywords of the content discussed by the user.

[0661] Specific actions:

[0662] The terminal analyzes text data using NLTK and spaCy to extract frequently occurring words such as "market" and "strategy."

[0663] The terminal lists the extracted keywords. For example, it generates a list such as ["market", "strategy"].

[0664] Step 4: Analysis using the emotion engine

[0665] The device uses an emotion engine (e.g., Microsoft Azure Emotion API) to recognize the user's emotions in real time. The input is the user's facial expressions and voice tone, and the output is emotion data. The device analyzes the collected data to confirm the user's emotional state.

[0666] Specific actions:

[0667] The device captures the user's facial expressions with its camera and records their voice tone with its microphone while they are viewing the document.

[0668] The device sends the captured data to the Microsoft Azure Emotion API and receives emotion data. For example, it might receive data such as {"emotion": "positive", "confidence": 0.92}.

[0669] Step 5: Search and sort related materials

[0670] The terminal sends the extracted list of frequently occurring words and sentiment data to the server. The input is the list of frequently occurring words and sentiment data, and the output is a sorted list of related documents. The server accesses the internal database, searches for the relevant documents, and sorts them based on priority.

[0671] Specific actions:

[0672] The device sends a list of frequently occurring words and sentiment data to the server.

[0673] The server uses Elasticsearch to search for materials that match the terms "market" and "strategy".

[0674] The server sorts search results based on the degree of relevance of frequently occurring words and sentiment data. For example, "Strategic Plan 2023.pdf" and "Market Analysis Report 2023.pptx" will appear at the top of the search results.

[0675] Step 6: Display and provide materials

[0676] The server sends a sorted list of materials to the terminal. The input is the sorted list of materials, and the output is the material links displayed to the user. The terminal displays the material links to the user based on the search results.

[0677] Specific actions:

[0678] The server generates a sorted list of documents and sends it to the terminal.

[0679] The device displays links to documents on the user's screen. For example, "Strategic Plan 2023.pdf", "Market Analysis Report 2023.pptx", and "September Updated Market Report.pdf" will be displayed.

[0680] Step 7: Explanation of terminology and provision of related materials

[0681] The device identifies terms that the user may not understand from among frequently used words and generates explanations. The input is frequently used words, and the output is a term definition and links to related materials. It generates explanatory text for the identified terms and displays links to related materials to the user.

[0682] Specific actions:

[0683] The device recognizes keywords such as "digital transformation."

[0684] The terminal generates an explanation of "digital transformation" and links to related materials (e.g., "DX Introduction Guide.pdf").

[0685] The user's screen displays explanatory text and links to related resources.

[0686] (Application Example 2)

[0687] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0688] Current autonomous vehicles offer limited information tailored to the interests and emotions of passengers and users, and lack systems that provide relevant information in real time based on conversation. This makes it difficult to improve passenger convenience and satisfaction. Furthermore, there is a lack of appropriate means to provide information about terminology that passengers do not understand.

[0689] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for converting audio data such as meetings into text data, means for extracting frequently occurring words from the converted text data, means for searching for related materials from a database based on the extracted frequently occurring words and arranging them in order of synchronization rate, means for generating links to the arranged materials, means for converting conversations of passengers or users into text data, extracting frequently occurring words from the text data and recognizing the emotional state of passengers in real time, and means for searching for related content based on the extracted frequently occurring words and emotional data, and displaying it with priority. As a result, relevant information can be obtained in real time from conversations that passengers have in an autonomous vehicle, and information can be provided while also taking into account the emotions of passengers. In addition, by appropriately providing explanations of terms that passengers do not understand, passenger satisfaction and convenience can be greatly improved.

[0690] "Audio data from meetings, etc." refers to audio information recorded during meetings or other similar gatherings.

[0691] "Means of converting to text data" refers to methods or techniques for converting audio data into text information.

[0692] "Frequently occurring words" refer to words or phrases that appear frequently in text data.

[0693] "Emotional state" refers to data that indicates the user's emotions, such as positive or negative, excited or relieved.

[0694] A "database" is an electronic storage facility for systematically saving and managing related materials and information.

[0695] "Related materials" refer to information and documents that are related to the frequently extracted words.

[0696] "Synchronization rate" is an indicator that shows the degree of agreement or relevance between extracted frequently occurring words and related materials.

[0697] An "emotion engine" is a technology or software that analyzes a user's emotions in real time and generates data based on those emotions.

[0698] "Means of prioritizing display" refers to methods and techniques for sorting and displaying related information or content according to specific criteria.

[0699] "Conversation" is the process by which multiple people communicate with each other through sound.

[0700] This invention relates to a system primarily applied in autonomous vehicles that converts passenger and user conversations into text data in real time, extracts frequently occurring words, recognizes emotional states, and provides relevant information based on the results. The system is implemented based on the following procedure and configuration.

[0701] System Configuration

[0702] 1. Acquisition of audio data and conversion to text:

[0703] The server uses microphones inside the autonomous vehicle to acquire audio data of conversations between passengers and users. The acquired audio data is then converted into text data using the Google Speech-to-Text API.

[0704] 2. Extraction of frequently occurring words:

[0705] The server extracts frequently occurring words from text data using the NLTK library (Python), a natural language processing (NLP) tool.

[0706] 3. Recognition of emotional states:

[0707] The server uses Microsoft Azure's Emotion API to recognize the emotional state of passengers and users in real time and generate emotional data.

[0708] 4. Searching for and providing related information:

[0709] Based on the extracted frequent words and sentiment data, the server searches the Elasticsearch database to find relevant content (e.g., music, news, tourist information, advertisements, etc.), prioritizes it, and displays it.

[0710] Program Processing Overview

[0711] 1. Processing of audio data:

[0712] The audio data is sent from the microphone to the server and converted into text data using the Google Speech-to-Text API. The following is an example of a specific prompt.

[0713] Example prompt:

[0714] Based on the audio data, transcribe the content into text, extract frequently occurring words, obtain sentiment data using a sentiment engine, search for related content using Elasticsearch, and display it according to priority.

[0715] 2. Processing text data:

[0716] The NLTK library is used to extract frequently occurring words from the converted text data. The extracted keywords are important words that reflect the conversation content and are used in the search query.

[0717] 3. Processing of emotional data:

[0718] Text data is sent to the emotion engine, and emotion data is generated using Azure's Emotion API. This emotion data will play a crucial role in the subsequent information provision process.

[0719] 4. Searching for and displaying related content:

[0720] Based on the extracted frequent words and sentiment data, the server searches the Elasticsearch database to find relevant content. The content is sorted according to priority based on sentiment state and displayed on the smart display. For example, if it is recognized that a passenger is "talking about restaurants nearby," restaurant information will be displayed preferentially.

[0721] Specific example

[0722] Let's say a passenger is talking on the train and says, "I want to know about tourist attractions in this area." The server captures this conversation audio and uses the Google Speech-to-Text API to convert "tourist attractions in this area" into text. Using the NLTK library, "tourist attractions" is extracted as a key keyword, and the passenger's level of interest is further analyzed by an emotion engine. The Elasticsearch database is searched, and information based on the keyword "tourist attractions" is displayed preferentially. For example, "popular tourist attractions in this area" or "recommended tourist destinations list" might be displayed on the smart display.

[0723] Thus, this invention realizes a system that can improve passenger convenience and satisfaction in autonomous vehicles by providing information in real time that corresponds to the content of passengers' or users' conversations and emotional states.

[0724] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0725] Step 1:

[0726] The server acquires audio data using microphones inside the autonomous vehicle. The acquired audio data is saved as is. Specifically, it is recorded in an audio file format (e.g., a .wav file).

[0727] Input: Audio of conversations between passengers and users inside the vehicle.

[0728] Output: Audio data file (.wav)

[0729] Step 2:

[0730] The server sends the audio data to the Google Speech-to-Text API, where it is converted into text data. The converted text data is then stored for subsequent processing and formatted into a concise form.

[0731] Input: Audio data file (.wav)

[0732] Output: Text data

[0733] Step 3:

[0734] The server uses the NLTK library to extract frequently occurring words from text data. It tokenizes (segments) the text data, calculates the frequency of each word, and picks out the most frequent words.

[0735] Input: Text data

[0736] Output: List of frequently occurring words

[0737] Step 4:

[0738] The server sends the extracted text data to the emotion engine, which uses Microsoft Azure's Emotion API to generate emotion data. This emotion data provides a detailed description of the user's emotional state, such as positive, negative, or excited.

[0739] Input: Text data

[0740] Output: Sentiment data

[0741] Step 5:

[0742] The server searches the Elasticsearch database based on the extracted list of frequent words and sentiment data. The extracted frequent words are used in the search queries, and the search results are prioritized based on the sentiment data.

[0743] Input: List of frequently occurring words, sentiment data

[0744] Output: Related Content List

[0745] Step 6:

[0746] The server selects relevant content from the search results in order of priority and displays it on the smart display. For example, if a passenger is talking about "restaurants," information about nearby restaurants will be displayed.

[0747] Input: Related Content List

[0748] Output: Content displayed on the smart display

[0749] Step 7:

[0750] The system uses a terminal to identify terms that passengers may not understand and generates and displays explanations of those terms along with links to related resources. For example, if the term "digital transformation" is included, an explanation and a link to more detailed resources will be displayed.

[0751] Input: List of frequently occurring words

[0752] Output: Glossary of terms, links to related resources

[0753] The above outlines the specific processing steps for implementing this invention. This enables the real-time provision of information tailored to the conversation content and emotional state of passengers and users.

[0754] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0755] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0756] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0757] [Third Embodiment]

[0758] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0759] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0760] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0761] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0762] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0763] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0764] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0765] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0766] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0767] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0768] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0769] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0770] This invention relates to a system that converts audio data, such as from meetings, into text data, extracts frequently occurring words from that text data, and searches for and provides related materials from an internal company database based on those words. It also provides explanations of the latest hot topics related to the frequently occurring words, as well as terms that employees may not be familiar with.

[0771] Acquisition of audio data and text conversion

[0772] The user uploads meeting audio data (e.g., a .wav file) to the device. The device sends this audio data to a speech recognition API, which converts the audio data into text data. The converted text data is then formatted into a specific format within the device.

[0773] Extraction of frequently occurring words

[0774] The terminal uses natural language processing tools to extract frequently occurring words from the converted text data. For example, if the meeting content was "discussing future market strategies," keywords such as "market" and "strategy" would be extracted.

[0775] Searching for and providing related materials

[0776] Next, the terminal sends the extracted list of frequently occurring words to the server. The server searches for documents in the company's database based on the received list of frequently occurring words. The search results are sorted by calculating the synchronization rate between the content of each document and the frequently occurring words, and then sorting them in descending order of synchronization rate. As a result, documents such as "Strategic Plan 2023.pdf" and "Market Analysis Report 2023.pptx" are listed in reverse chronological order. The server generates links to these sorted documents and provides them to the user.

[0777] Categorizing materials and providing hot topics

[0778] The server also has the function of organizing all documents in the company's internal database into categories based on themes and attributes. This allows for efficient management of documents by category, such as "Market Analysis" and "Strategy Formulation." Furthermore, it selects and presents the latest documents related to frequently used keywords as hot topics. For example, within the "Market Analysis" category, the latest "September Updated Market Report.pdf" is presented as a hot topic.

[0779] Explanation of terminology and improvement of employee skills

[0780] The device identifies terms that the user may not understand from a list of frequently used words. For example, if the frequently used word "digital transformation" is included, the device will determine, based on past data and the user's skill level, that this term is unfamiliar. The device will generate an easy-to-understand explanation of this term and also display a link to related materials (e.g., "DX Introduction Guide.pdf").

[0781] Explanation with specific examples

[0782] The following is a concrete example. An audio file, "Meetings_aug30.wav," from a meeting held on August 30th is uploaded to the terminal. The terminal uses a speech recognition API to convert this audio file into text, and the content is rephrased as "Discussion on future market strategy policies." A natural language processing tool is used to extract frequently occurring words such as "market" and "strategy," and this is sent to the server. The server searches the company's internal database for related documents such as "Strategic Plan 2023.pdf" and "Market Analysis Report 2023.pptx," sorts them in reverse chronological order, and generates links. Furthermore, it presents the latest "September Market Report.pdf" as a hot topic related to "market analysis," and provides the user with an explanation of "digital transformation" and related materials.

[0783] Thus, the present invention provides a system that efficiently searches for and provides relevant materials based on meeting content, and contributes to improving employees' knowledge.

[0784] The following describes the processing flow.

[0785] Step 1:

[0786] The user uploads the meeting audio data (e.g., a .wav file) to their device. The audio data is in a format such as "Meetings_aug30.wav".

[0787] Step 2:

[0788] The device sends the uploaded audio data to a speech recognition API, which converts the audio data into text data. This is where text such as "Strategic Meeting August 30, 2023" is generated.

[0789] Step 3:

[0790] The terminal receives the converted text data and formats it by removing unnecessary information (such as noise or irrelevant conversation). For example, unnecessary words like "um" or "well" are removed.

[0791] Step 4:

[0792] The device uses natural language processing tools to extract frequently occurring words from the formatted text. For example, keywords such as "strategy," "market," and "analysis" might be extracted.

[0793] Step 5:

[0794] The terminal sends the extracted list of frequently occurring words to the server. This list includes not only the frequently occurring words but also their frequency of occurrence.

[0795] Step 6:

[0796] Based on the frequently occurring word list received by the server, all documents in the company database are searched. The degree of agreement between the document content and the frequently occurring words, i.e., the synchronization rate, is calculated.

[0797] Step 7:

[0798] The server sorts the documents with the highest synchronization rate by newest first. For example, "Strategic Plan 2023.pdf" and "Latest Market Analysis Report 2023.pptx" are output.

[0799] Step 8:

[0800] The server generates links to a sorted list of materials and provides them to the user via the terminal. This allows the user to easily access relevant materials.

[0801] Step 9:

[0802] The server organizes documents within the company's internal database by category. For example, they might be divided into categories such as "Strategy," "Market," and "Analysis."

[0803] Step 10:

[0804] The server searches for and presents the latest hot topics related to frequently occurring keywords. For example, "September Market Report.pdf" might be presented as a hot topic within the "Market Analysis" category.

[0805] Step 11:

[0806] The device identifies terms that the user may not understand from among frequently used words. For example, "digital transformation" might be identified in this way.

[0807] Step 12:

[0808] The device provides explanations of terms it has identified and links to related materials. For example, it might display an explanation such as, "Digital transformation is the process by which companies use digital technologies to transform their business models," along with a link to "DX Introduction Guide.pdf".

[0809] (Example 1)

[0810] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0811] Traditional methods of recording and analyzing meetings made it difficult to efficiently utilize audio data, requiring significant time and effort for manual text conversion and document retrieval. Furthermore, the accuracy and speed of searching for and providing relevant materials were limited, and explanations of unfamiliar terminology were insufficient. Therefore, a system was needed that could simultaneously achieve effective utilization of meeting data and improve knowledge.

[0812] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0813] In this invention, the server includes means for converting audio data into text data, means for extracting frequently occurring words from the converted text data, means for searching a database for related information based on the extracted frequently occurring words and arranging it in order of decreasing synchronization rate, means for generating links to the arranged information, means for using an external API for recognizing audio data, and means for using a natural language processing tool for extracting frequently occurring words. This makes it possible to efficiently transcribe meeting audio data into text, provide related materials quickly and accurately, and simultaneously improve employees' knowledge.

[0814] "Audio data" refers to digital data files containing recordings of audio from meetings, conversations, and other similar events.

[0815] "Text data" refers to a digital data file created by converting audio data into text.

[0816] "Frequently occurring words" refer to words or expressions that appear with a specific frequency within text data.

[0817] An "external API" refers to an application programming interface provided by an external service or platform.

[0818] "Natural language processing tools" refer to software and libraries used to analyze text data, understand its meaning and grammar, and process it.

[0819] A "database" refers to a collection of information that allows for the efficient management and retrieval of large amounts of data.

[0820] A "link" is a hypertext element that allows a user to access a specific webpage or file by clicking on it.

[0821] "Synchronization rate" refers to an indicator that shows the degree of agreement between frequently occurring words and the content of the materials in the database.

[0822] "Explanation" refers to a text or explanation that clearly describes unfamiliar or unknown terms.

[0823] This invention is a system that converts audio data, such as from meetings, into text data, extracts frequently occurring words from that text data, and searches and provides related materials from a database based on those words. Furthermore, it also provides the latest topics related to the frequently occurring words and explanations of terms that the user may not be familiar with. Embodiments of this system are described below.

[0824] First, the user uploads the meeting audio data (e.g., a .wav file) to their device. The device then sends this audio data to a speech recognition API (e.g., Google Speech-to-Text or Amazon Transcribe) to convert it into text data. The converted text data is then formatted into a specific format (e.g., date and time, speaker, content of speech) and processed.

[0825] Next, the terminal inputs the converted text data into a natural language processing tool (e.g., NLTK or spaCy) to extract frequently occurring words. For example, if the meeting topic is "discussing future market strategies," frequently occurring words such as "market" and "strategy" will be extracted. The extraction results are compiled into a list of frequently occurring words.

[0826] The terminal sends the extracted list of frequently occurring words to the server. The server searches the database using the ElasticSearch engine or similar based on the list. The search results are sorted by calculating the synchronization rate between the content of each document and the frequently occurring words, and then sorted in descending order of synchronization rate. This results in a list of documents such as "Strategic Plan 2023.pdf" and "Market Analysis Report 2023.pptx," and links to the sorted documents are generated.

[0827] Furthermore, the server organizes all documents in the company database by theme (e.g., "Market Analysis" or "Strategy Formulation"), selects the latest information related to frequently used keywords, and presents it to the user. For example, within the "Market Analysis" category, the latest "September Updated Market Report.pdf" might be presented as a hot topic.

[0828] Furthermore, the device identifies terms from a list of frequently used words that the user may not understand, based on past learning data. For example, if the frequently used word "digital transformation" is identified as unfamiliar, it generates an easy-to-understand explanation and provides a link to related materials (e.g., "DX Introduction Guide.pdf").

[0829] Specific example:

[0830] An audio file, "Meetings_aug30.wav," from a meeting held on August 30th is uploaded to the user's device. This audio file is converted into text data via a speech recognition API on the device. The content is translated to something like "Discussion on future market strategy." Next, a natural language processing tool is used to extract frequently occurring words such as "market" and "strategy," and this is sent to the server. The server searches its database for related documents such as "Strategic Plan 2023.pdf" and "Market Analysis Report 2023.pptx," sorts them in reverse chronological order, and generates links. Furthermore, the latest "September Market Report.pdf" is presented as a hot topic related to "market analysis," and the user is provided with an explanation of "digital transformation" and related materials.

[0831] Example of a prompt

[0832] The following are specific examples of prompt statements that will be used as input to the generative AI model.

[0833] "Please describe a program to search for and provide materials related to the topics of 'market' and 'strategy' discussed at the August 30th meeting. Additionally, please explain the term 'digital transformation' and add a function to provide related materials."

[0834] This system enables efficient transcription of meeting content, prompt and accurate provision of related materials, and contributes to improving employees' knowledge.

[0835] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0836] Step 1: Acquire audio data and convert it to text.

[0837] The user uploads meeting audio data (e.g., a .wav file) to their device. The input is an audio data file. The device sends this audio data to the Google Speech-to-Text API and retrieves text data. The output is the converted text data. Specifically, the audio data is passed to the API, and the text data is received as the API response.

[0838] Step 2: Formatting the text data

[0839] The terminal formats the acquired text data into a specific format (e.g., date and time, speaker, content of the message). The input is text data obtained from the API. The output is the formatted text data. Specifically, the terminal organizes the text data using methods such as JSON or CSV format to make it suitable for subsequent processing.

[0840] Step 3: Extraction of frequently occurring words

[0841] The terminal inputs formatted text data into a natural language processing tool (e.g., NLTK or spaCy) and extracts frequently occurring words. The input is formatted text data. The output is a list of frequently occurring words. Specifically, the text data is tokenized, the frequency of each word is counted, and words exceeding a certain frequency are extracted.

[0842] Step 4: Send the list of frequently used words

[0843] The terminal sends the extracted list of frequently occurring words to the server. The input is the list of frequently occurring words. The output is the data sent to the server. Specifically, the process involves encoding the list of frequently occurring words in JSON format and sending it to the server using an HTTP request.

[0844] Step 5: Search for related materials

[0845] The server uses the Elasticsearch engine to search for materials in the database based on the received list of frequently occurring words. The input is the list of frequently occurring words. The output is the search results. Specifically, it generates an Elasticsearch query and executes the query against the database.

[0846] Step 6: Calculate and sort the data synchronization rate.

[0847] The server calculates the synchronization rate between the content of each document and its frequently occurring words, and sorts them in descending order of synchronization rate. The input is the search results. The output is the sorted list of documents. Specifically, it scores how well the frequently occurring words match the content of each document and sorts the documents based on that score.

[0848] Step 7: Generate and provide links

[0849] The server generates and provides links to sorted documents to the user. The input is a sorted list of documents. The output is links to the documents. Specifically, the server generates the links in HTML format, making them accessible to the user.

[0850] Step 8: Categorize materials and provide hot topics

[0851] The server organizes all documents in the company database by theme, selects the latest information related to frequently used keywords, and presents it to the user. Inputs are a list of documents and frequently used keywords. Outputs are documents organized by category and the latest hot topics. Specifically, it uses a classification algorithm to group documents by theme and selects the most recent documents.

[0852] Step 9: Terminology identification and explanation generation

[0853] The device identifies terms that the user may not understand from a list of frequently used words, generates easy-to-understand explanations for the unfamiliar terms, and displays links to related materials. Input consists of the list of frequently used words and the user's past learning data. Output is the term explanation and links. Specifically, it executes a discrimination algorithm based on the learning data, generates explanatory text, and attaches links to related materials.

[0854] (Application Example 1)

[0855] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0856] While it is crucial to respond quickly and accurately to customer questions and requests during customer service in stores, delays can occur when staff are busy or when there is inconsistency in product knowledge. Furthermore, insufficient information provided to customers can lead to decreased customer satisfaction. Additionally, it is difficult for new or part-time staff to fully grasp specialized terminology and detailed product information, which can result in a decline in the quality of customer service. A system is needed to address these challenges and improve customer satisfaction.

[0857] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0858] This invention includes a server that transcribes audio data of customer conversations into text in real time and searches for and provides related product information based on frequently occurring words; a server that searches for related materials from a database based on the extracted frequently occurring words and arranges them in order of synchronization rate; and a server that generates explanations for the identified terms and presents links to related materials. This makes it possible to convert conversations with customers into text data and quickly provide related product information and detailed explanations based on frequently occurring questions and requests.

[0859] "Audio data" refers to sound information, such as conversations and music, recorded and stored in digital format.

[0860] "Text data" refers to a digital representation of character information.

[0861] "Frequently occurring words" refer to words or phrases that appear particularly often within the text data.

[0862] A "database" is a system for efficiently managing and retrieving data, and is a collection of structured information.

[0863] "Synchronization rate" is an indicator that shows the degree of agreement between frequently occurring words and the data in the database.

[0864] A "link" is reference information that allows direct access to specific data or materials.

[0865] "Customer service" refers to the activities of interacting with and providing services to customers in stores and service industries.

[0866] "Real-time text conversion" is the process of instantly converting audio data into text data.

[0867] "Related product information" refers to detailed data and descriptions of products found through searches based on frequently occurring keywords.

[0868] "Category organization" is the process of classifying and managing data and materials based on common attributes or themes.

[0869] "Hot topics" refer to the latest information or topics that have recently become popular.

[0870] An "explanation" is a detailed explanation of a specific term or concept.

[0871] System program

[0872] The apparatus for carrying out this invention consists of a system in which a server, a terminal, and a user work together.

[0873] The terminal has the functionality to acquire voice data during customer service and convert it into text data. To achieve this, the Google Cloud Speech-to-Text API and Python's speech_recognition library are used. In addition, frequently occurring words are extracted from the converted text data using natural language processing tools (e.g., spaCy, NLTK).

[0874] The server has the functionality to search for related materials and product information from existing databases (e.g., Firebase) based on a list of frequently used words sent from the terminal, sort them in order of synchronicity, and generate links. Furthermore, it also has the functionality to present hot topics related to frequently used words.

[0875] Program Processing Description

[0876] 1. Acquisition and transcription of audio data

[0877] The system records conversations during customer service using a terminal and sends the recordings to the Google Cloud Speech-to-Text API, converting the audio data into text data. This makes the audio information easily searchable text data.

[0878] 2. Extraction of frequently occurring words

[0879] The converted text data is analyzed using natural language processing tools (NLTK, spaCy) to extract frequently occurring words. This allows for a quick understanding of customer interests and question trends.

[0880] 3. Searching for and providing related information

[0881] The device sends the extracted list of frequently occurring words to the server. The server searches its database (Firebase) for relevant product information and materials, sorts them in the optimal order based on the synchronization rate, and generates links.

[0882] 4. Explanation of hot topics and terminology

[0883] The server generates and presents detailed explanations, along with links, of the latest hot topics related to frequently used words and terms that customers may not understand.

[0884] Examples of specific cases and prompt statements

[0885] Specific example:

[0886] For example, if frequently occurring keywords such as "new product" and "promotion" are extracted from an audio file recorded during customer service on August 30th, the server will search for detailed information on products related to these keywords and the latest campaign information, and present them in a list sorted by the degree of synchronization. Furthermore, if it is determined that the term "promotion" is not well understood by new staff, an explanation of the term and links to related materials will also be provided.

[0887] Example of a prompt:

[0888] "We want to develop an application that transcribes audio data of customers being served on August 30th into text, extracts frequently occurring words, and then searches for and provides related product information and the latest promotional information."

[0889] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0890] Step 1:

[0891] The terminal acquires audio data during customer service. The user (store clerk) uses a smartphone or dedicated device to record the audio data of the conversation. This audio data becomes the input data, and in this step, an audio file is specifically generated.

[0892] Step 2:

[0893] The device sends the acquired audio data to the Google Cloud Speech-to-Text API, where it is converted into text data. The input here is the audio file obtained in step 1, and the API is used to convert that audio into text format and output it. Specifically, the audio data is sent to the API, and the converted text is returned from the API.

[0894] Step 3:

[0895] The terminal analyzes the converted text data using a natural language processing tool (e.g., spaCy or NLTK) and extracts frequently occurring words. The input is the text data obtained in step 2, and data processing such as word segmentation and part-of-speech analysis is performed using NLTK or spaCy to extract and output frequently occurring words. Specifically, the text is broken down, and the words that appear most frequently are listed.

[0896] Step 4:

[0897] The terminal sends the extracted list of frequently occurring words to the server. The input is the list of frequently occurring words obtained in step 3, and the terminal sends data to the server based on this. Specifically, the operation to send the list of frequently occurring words to the API endpoint is performed.

[0898] Step 5:

[0899] The server searches for relevant documents and product information from a database (such as Firebase) based on the list of frequently occurring words. The input for this step is the list of frequently occurring words submitted in step 4. The server searches the database to retrieve relevant documents and product information and outputs it as a list sorted by synchronization rate. Specifically, it forms a search query using frequently occurring words as keywords and extracts the corresponding data from the database.

[0900] Step 6:

[0901] The server generates links to relevant documents and product information from the search results and sends them to the terminal. The input is the search results obtained in step 5, and it generates and outputs links to the detail pages for each document and product. Specifically, it uses a link generation algorithm to form URLs, lists them, and sends them.

[0902] Step 7:

[0903] The server identifies hot topics and terms that customers may not understand, and generates explanations and related materials. The input is the search results obtained in step 5 and the user profile, and it generates and outputs explanatory text and links to related materials. Specifically, it identifies unfamiliar terms from frequently occurring words and historical data, and provides explanatory text and links to related materials based on those terms.

[0904] Step 8:

[0905] The terminal displays the received link and explanation to the user (store clerk) and provides it to the customer. The input consists of the link and explanation generated in steps 6 and 7, and is output by being displayed on a screen such as a smartphone. Specifically, the link is displayed in a clickable format, and the explanation is displayed in an information box.

[0906] Through these steps, users can respond quickly and accurately to customer questions and requests, thereby improving customer satisfaction.

[0907] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0908] This invention relates to a system that converts audio data, such as from meetings, into text data, extracts frequently occurring words from that text data, and searches for and provides related materials from an internal company database based on those words. Furthermore, by combining this with an emotion engine that recognizes the user's emotions, the system can also provide materials and change their priority according to the user's emotional state.

[0909] Acquisition of audio data and text conversion

[0910] The user uploads meeting audio data (e.g., a .wav file) to the device. The device sends this audio data to a speech recognition API, which converts the audio data into text data. The converted text data is then formatted into a specific format within the device.

[0911] Extraction of frequently occurring words

[0912] The terminal uses natural language processing tools to extract frequently occurring words from the converted text data. For example, if the meeting content was "discussing future market strategies," keywords such as "market" and "strategy" would be extracted.

[0913] Utilizing the Emotion Engine

[0914] The device utilizes an emotion engine to recognize the user's emotions in real time. For example, the emotion engine analyzes the user's facial expressions and voice tone while they are viewing documents and generates emotion data.

[0915] Searching for and providing related materials

[0916] Next, the terminal sends the extracted list of frequently occurring words and sentiment data to the server. The server searches for documents in the company's database based on the received list of frequently occurring words and sentiment data. The search results are sorted using a new criterion that integrates synchronization rate and sentiment influence, taking into account the degree of match between the content of each document and the frequently occurring words, as well as importance based on the user's sentiment data.

[0917] For example, if a user expresses positive feelings towards "marketing strategy," materials related to that topic will be displayed at the top of the search results. On the other hand, if negative feelings are detected, materials with a higher priority for mitigating those feelings will be presented.

[0918] Categorizing materials and providing hot topics

[0919] The server also has the function of organizing all documents in the company's internal database into categories based on themes and attributes. This allows for efficient management of documents by category, such as "Market Analysis" and "Strategy Formulation." Furthermore, it selects and presents the latest hot topics based on sentiment data. For example, within the "Market Analysis" category, the latest "September Updated Market Report.pdf" is presented as a hot topic.

[0920] Explanation of terminology and improvement of employee skills

[0921] The device identifies terms that the user may not understand from a list of frequently used words. For example, if the frequently used word "digital transformation" is included, the device will determine, based on past data and the user's skill level, that this term is unfamiliar. The device will generate an easy-to-understand explanation of this term and also display a link to related materials (e.g., "DX Introduction Guide.pdf").

[0922] Explanation with specific examples

[0923] The following is a concrete example. An audio file, "Meetings_aug30.wav," from a meeting held on August 30th is uploaded to the terminal. The terminal uses a speech recognition API to transcribe this audio file into text, and the content is translated into something like "Discussion on future market strategy." A natural language processing tool is used to extract frequently occurring words such as "market" and "strategy," and an emotion engine analyzes the user's emotional state (e.g., excitement, interest, negative emotion) in real time. The extracted list of frequently occurring words and emotion data are sent to the server.

[0924] The server searches the internal database for relevant documents such as "Strategic Plan 2023.pdf" and "Market Analysis Report 2023.pptx," and re-evaluates their priority based on synchronization rate and sentiment data. The documents are then sorted in the new order, and links are generated. Furthermore, the latest "September Market Report.pdf" is presented as a hot topic, and the user is provided with an explanation of "Digital Transformation" and related materials (e.g., "DX Introduction Guide.pdf").

[0925] Thus, the present invention provides a system that efficiently searches for and provides relevant materials based on meeting content, and realizes information provision that also takes into account the user's emotional state.

[0926] The following describes the processing flow.

[0927] Step 1:

[0928] The user uploads the meeting audio data (e.g., a .wav file) to their device. The audio data is, for example, named "Meetings_aug30.wav".

[0929] Step 2:

[0930] The device sends the uploaded audio data to a speech recognition API, which converts the audio data into text data. For example, this might result in "Meeting content: Market strategy and cost reduction."

[0931] Step 3:

[0932] The terminal receives the converted text data and performs preprocessing to remove unnecessary characters and noise, and then formats it. For example, unnecessary parts such as "um" or "well" are removed.

[0933] Step 4:

[0934] The device uses natural language processing tools to extract frequently occurring words from pre-processed text data. For example, "market," "strategy," and "cost reduction" might be extracted.

[0935] Step 5:

[0936] The terminal sends the extracted list of frequently occurring words to the server. This list includes each frequently occurring word and its frequency of occurrence.

[0937] Step 6:

[0938] The server searches for documents within the company database based on the list of frequently occurring words it receives. It then calculates the synchronization rate between the content of each document and the list of frequently occurring words.

[0939] Step 7:

[0940] The server sorts the documents with the highest synchronization rate by newest first. For example, "Strategic Plan 2023.pdf" and "Latest Market Analysis Report 2023.pptx" are output.

[0941] Step 8:

[0942] The server generates links to a sorted list of materials and provides them to the user via the terminal. This allows the user to easily access relevant materials.

[0943] Step 9:

[0944] The device uses an emotion engine to analyze the user's emotions in real time. For example, it performs facial recognition and voice tone analysis to collect data on the emotional state the user exhibits while viewing materials.

[0945] Step 10:

[0946] The device sends emotional data to the server. For example, if a user expresses positive emotions towards "marketing strategy," that data is sent to the server.

[0947] Step 11:

[0948] The server re-evaluates the priority of previously searched materials based on sentiment data. Materials related to topics showing positive sentiment will appear higher in the search results.

[0949] Step 12:

[0950] If a server is showing negative emotions, resources to alleviate those emotions will be recommended. For example, resources that help reduce stress or solve problems will be selected and presented preferentially.

[0951] Step 13:

[0952] The server organizes all documents in the company's internal database by category. For example, they might be classified into categories such as "Strategy," "Market," and "Analysis."

[0953] Step 14:

[0954] The server searches for and presents the latest hot topics related to frequently occurring keywords. For example, "September Market Report.pdf" might be presented as a hot topic.

[0955] Step 15:

[0956] The device identifies terms that the user may not understand from among frequently used words. For example, "digital transformation" might be identified in this way.

[0957] Step 16:

[0958] The device provides explanations of terms it has identified and displays links to related materials. For example, it might provide an explanation such as, "Digital transformation is the process by which companies use digital technologies to transform their business models," along with a link to "DX Introduction Guide.pdf."

[0959] (Example 2)

[0960] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0961] Conventional systems that convert meeting audio into text data and provide related materials only utilize speech recognition and extraction of frequently occurring words, failing to provide information that takes user emotions into consideration. Therefore, there were issues such as users not receiving the information they needed appropriately and being unable to respond to changes in their emotions during use. Furthermore, features to deepen understanding of meeting content, such as category organization and glossary, were insufficient.

[0962] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0963] In this invention, the server includes means for converting audio data such as meetings into text data, means for extracting frequently occurring words from the converted text data, and means for searching for relevant materials from the company's internal database based on the extracted frequently occurring words and user sentiment data, and sorting them by synchronization rate and priority based on sentiment data. This makes it possible to search for and provide relevant materials that take into account the user's emotional state. Furthermore, by adding means for presenting the latest hot topics related to frequently occurring words and sentiment data, and means for identifying terms that the user may not understand and generating and providing explanations, more efficient and user-friendly information provision can be achieved.

[0964] 1. "Audio data" refers to data in digital file format that records audio, including oral communication such as meetings and interviews.

[0965] 2. "Text data" refers to character information converted from audio data, and is data expressed in a readable format.

[0966] 3. "Frequently occurring words" refer to words or phrases that appear particularly often in the text data and are extracted as important keywords.

[0967] 4. "Emotional data" refers to digital data generated as a result of analyzing a user's emotional state (e.g., positive, negative, excited, etc.).

[0968] 5. An "internal database" is a digital storage system used to accumulate and manage various documents and data within a company.

[0969] 6. "Related materials" refer to highly relevant documents and digital content searched from the company's internal database based on the extracted frequently occurring words and sentiment data.

[0970] 7. "Synchronization rate" is an indicator that shows how well the extracted frequently occurring words match the content of related materials.

[0971] 8. "Priority" is an indicator that influences the display order of search results and is determined based on the match rate of frequently occurring words and sentiment data.

[0972] 9. A "link" is a hypertext-based reference that allows a user to navigate to a specific document or webpage by clicking on it.

[0973] 10. A "category" is a classification of related materials based on subject matter or attributes, and is a concept designed to enable users to efficiently search for and utilize information.

[0974] 11. "Hot topics" refer to the latest information and topics considered to be of high interest, and are determined based on user interest and sentiment data.

[0975] 12. A "glossary" is a document that clearly explains the meaning and background of specific terms that users may not understand.

[0976] This invention is a system that converts audio data from meetings and other events into text data, extracts frequently occurring words from that text data, and searches for and provides related materials from the company's internal database based on those words. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it can also provide materials and change their priority according to the user's emotional state.

[0977] Hardware and software to be used

[0978] Terminal: A device on which users upload audio data and perform tasks such as displaying materials and emotion recognition. Examples include PCs and smartphones.

[0979] Server: A central server that performs tasks such as converting audio data to text, extracting frequently occurring words, searching for related materials, and prioritizing the order of materials.

[0980] software:

[0981] Speech-to-Text API: Google Cloud Speech-to-Text

[0982] Natural language processing tools: NLTK, spaCy

[0983] Emotion engine: Microsoft Azure Emotion API

[0984] Database search engine: Elasticsearch

[0985] Acquisition of audio data and text conversion

[0986] The user uploads meeting audio data (e.g., a .wav file) to their device. The device sends this audio data to a speech recognition API, which converts the audio data into text data. The converted text data is then formatted within the device into a specific format (e.g., JSON format).

[0987] Specific example:

[0988] The user clicks the "Select audio file" button, selects "Meetings_aug30.wav" from their local file system, and uploads it.

[0989] The device sends this audio file to the Google Cloud Speech-to-Text API, where it is converted into text data. For example, the converted text data might read, "Discussing future market strategy guidelines."

[0990] Extraction of frequently occurring words

[0991] The device uses natural language processing tools to extract frequently occurring words from the converted text data. For example, keywords such as "market" and "strategy" are extracted. The extracted keywords are then compiled into a list.

[0992] Specific example:

[0993] The terminal analyzes text data using NLTK and spaCy to extract frequently occurring words such as "market" and "strategy."

[0994] The extracted keywords are listed. For example, the list might look like this: ["market", "strategy"].

[0995] Utilizing the Emotion Engine

[0996] The device utilizes an emotion engine to recognize the user's emotions in real time. For example, it analyzes the user's facial expressions and voice tone while they are viewing documents and generates emotion data.

[0997] Specific example:

[0998] The device captures the user's facial expressions with its camera and acquires their voice tone with its microphone.

[0999] This data is sent to the Microsoft Azure Emotion API, and the generated emotion data is received. For example, the data might look like this: {"emotion": "positive", "confidence": 0.92}.

[1000] Searching for and providing related materials

[1001] Next, the terminal sends the extracted list of frequently occurring words and sentiment data to the server. The server searches for documents in the company's database based on the received list of frequently occurring words and sentiment data. The search results are sorted considering the degree of match between the content of each document and the frequently occurring words, as well as their importance based on the user's sentiment data.

[1002] Specific example:

[1003] The server uses Elasticsearch to search for materials that match "market" and "strategy".

[1004] For example, "Strategic Plan 2023.pdf" and "Market Analysis Report 2023.pptx" are searched, and their priority is re-evaluated based on relevance and sentiment data.

[1005] Categorizing materials and providing hot topics

[1006] The server also has the function of organizing all documents in the company's internal database into categories based on themes and attributes. This allows for efficient management of documents by category, such as "market analysis" and "strategy formulation." Furthermore, it selects and presents the latest hot topics to users based on sentiment data.

[1007] Specific example:

[1008] Based on sentiment data, the latest "September Updated Market Report.pdf" is presented as a hot topic.

[1009] Explanation of terminology and improvement of employee skills

[1010] The terminal identifies terms that the user may not understand from a list of frequently used words. For example, if the frequently used word "digital transformation" is included, the terminal will determine, based on past data and the user's skill level, that this term is unfamiliar. The terminal will generate an easy-to-understand explanation of this term and also display a link to related materials (e.g., "DX Introduction Guide.pdf").

[1011] Specific example:

[1012] The device recognizes the keyword "digital transformation" and displays links to related explanatory texts and documents (e.g., "DX Introduction Guide.pdf").

[1013] Example of a prompt:

[1014] I uploaded the meeting audio data. Frequently occurring words such as "market strategy" and "future plans" were extracted. The user's sentiment is positive, and they are interested in materials related to market strategy. Please search and sort the related materials. Also, please provide an explanation and related materials on "digital transformation."

[1015] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1016] Step 1: Upload audio data

[1017] The user uploads meeting audio data (e.g., a .wav file) to the device. The input is the audio file, and the output is a notification that the upload to the device is complete. The device saves the input audio data to temporary storage and prepares to send it to the speech recognition API in the next step.

[1018] Specific actions:

[1019] The user clicks the "Select audio file" button and selects "Meetings_aug30.wav" from the local file system.

[1020] The device accepts the file and displays a notification that the upload is complete.

[1021] Step 2: Convert audio to text

[1022] The device sends audio data to a speech recognition API (e.g., Google Cloud Speech-to-Text) and converts it into text data. The input is audio data, and the output is converted text data (e.g., in JSON format). The device receives the text data and formats it into a specific format.

[1023] Specific actions:

[1024] The device sends "Meetings_aug30.wav" to the Google Cloud Speech-to-Text API.

[1025] The device receives the text data returned from the API. For example, the data is formatted in the format {"text": "Discussion on future market strategy"}.

[1026] Step 3: Extraction of frequently occurring words

[1027] The terminal extracts frequently occurring words from the converted text data using natural language processing tools (e.g., NLTK, spaCy). The input is text data, and the output is a list of frequently occurring words. The terminal analyzes the text data to extract the main keywords of the content discussed by the user.

[1028] Specific actions:

[1029] The terminal analyzes text data using NLTK and spaCy to extract frequently occurring words such as "market" and "strategy."

[1030] The terminal lists the extracted keywords. For example, it generates a list such as ["market", "strategy"].

[1031] Step 4: Analysis using the emotion engine

[1032] The device uses an emotion engine (e.g., Microsoft Azure Emotion API) to recognize the user's emotions in real time. The input is the user's facial expressions and voice tone, and the output is emotion data. The device analyzes the collected data to confirm the user's emotional state.

[1033] Specific actions:

[1034] The device captures the user's facial expressions with its camera and records their voice tone with its microphone while they are viewing the document.

[1035] The device sends the captured data to the Microsoft Azure Emotion API and receives emotion data. For example, it might receive data such as {"emotion": "positive", "confidence": 0.92}.

[1036] Step 5: Search and sort related materials

[1037] The terminal sends the extracted list of frequently occurring words and sentiment data to the server. The input is the list of frequently occurring words and sentiment data, and the output is a sorted list of related documents. The server accesses the internal database, searches for the relevant documents, and sorts them based on priority.

[1038] Specific actions:

[1039] The device sends a list of frequently occurring words and sentiment data to the server.

[1040] The server uses Elasticsearch to search for materials that match the terms "market" and "strategy".

[1041] The server sorts search results based on the degree of relevance of frequently occurring words and sentiment data. For example, "Strategic Plan 2023.pdf" and "Market Analysis Report 2023.pptx" will appear at the top of the search results.

[1042] Step 6: Display and provide materials

[1043] The server sends a sorted list of materials to the terminal. The input is the sorted list of materials, and the output is the material links displayed to the user. The terminal displays the material links to the user based on the search results.

[1044] Specific actions:

[1045] The server generates a sorted list of documents and sends it to the terminal.

[1046] The device displays links to documents on the user's screen. For example, "Strategic Plan 2023.pdf", "Market Analysis Report 2023.pptx", and "September Updated Market Report.pdf" will be displayed.

[1047] Step 7: Explanation of terminology and provision of related materials

[1048] The device identifies terms that the user may not understand from among frequently used words and generates explanations. The input is frequently used words, and the output is a term definition and links to related materials. It generates explanatory text for the identified terms and displays links to related materials to the user.

[1049] Specific actions:

[1050] The device recognizes keywords such as "digital transformation."

[1051] The terminal generates an explanation of "digital transformation" and links to related materials (e.g., "DX Introduction Guide.pdf").

[1052] The user's screen displays explanatory text and links to related resources.

[1053] (Application Example 2)

[1054] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1055] Current autonomous vehicles offer limited information tailored to the interests and emotions of passengers and users, and lack systems that provide relevant information in real time based on conversation. This makes it difficult to improve passenger convenience and satisfaction. Furthermore, there is a lack of appropriate means to provide information about terminology that passengers do not understand.

[1056] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for converting audio data such as meetings into text data, means for extracting frequently occurring words from the converted text data, means for searching for related materials from a database based on the extracted frequently occurring words and arranging them in order of synchronization rate, means for generating links to the arranged materials, means for converting conversations of passengers or users into text data, extracting frequently occurring words from the text data and recognizing the emotional state of passengers in real time, and means for searching for related content based on the extracted frequently occurring words and emotional data, and displaying it with priority. As a result, relevant information can be obtained in real time from conversations that passengers have in an autonomous vehicle, and information can be provided while also taking into account the emotions of passengers. In addition, by appropriately providing explanations of terms that passengers do not understand, passenger satisfaction and convenience can be greatly improved.

[1057] "Audio data from meetings, etc." refers to audio information recorded during meetings or other similar gatherings.

[1058] "Means of converting to text data" refers to methods or techniques for converting audio data into text information.

[1059] "Frequently occurring words" refer to words or phrases that appear frequently in text data.

[1060] "Emotional state" refers to data that indicates the user's emotions, such as positive or negative, excited or relieved.

[1061] A "database" is an electronic storage facility for systematically saving and managing related materials and information.

[1062] "Related materials" refer to information and documents that are related to the frequently extracted words.

[1063] "Synchronization rate" is an indicator that shows the degree of agreement or relevance between extracted frequently occurring words and related materials.

[1064] An "emotion engine" is a technology or software that analyzes a user's emotions in real time and generates data based on those emotions.

[1065] "Means of prioritizing display" refers to methods and techniques for sorting and displaying related information or content according to specific criteria.

[1066] "Conversation" is the process by which multiple people communicate with each other through sound.

[1067] This invention relates to a system primarily applied in autonomous vehicles that converts passenger and user conversations into text data in real time, extracts frequently occurring words, recognizes emotional states, and provides relevant information based on the results. The system is implemented based on the following procedure and configuration.

[1068] System Configuration

[1069] 1. Acquisition of audio data and conversion to text:

[1070] The server uses microphones inside the autonomous vehicle to acquire audio data of conversations between passengers and users. The acquired audio data is then converted into text data using the Google Speech-to-Text API.

[1071] 2. Extraction of frequently occurring words:

[1072] The server extracts frequently occurring words from text data using the NLTK library (Python), a natural language processing (NLP) tool.

[1073] 3. Recognition of emotional states:

[1074] The server uses Microsoft Azure's Emotion API to recognize the emotional state of passengers and users in real time and generate emotional data.

[1075] 4. Searching for and providing related information:

[1076] Based on the extracted frequent words and sentiment data, the server searches the Elasticsearch database to find relevant content (e.g., music, news, tourist information, advertisements, etc.), prioritizes it, and displays it.

[1077] Program Processing Overview

[1078] 1. Processing of audio data:

[1079] The audio data is sent from the microphone to the server and converted into text data using the Google Speech-to-Text API. The following is an example of a specific prompt.

[1080] Example prompt:

[1081] Based on the audio data, transcribe the content into text, extract frequently occurring words, obtain sentiment data using a sentiment engine, search for related content using Elasticsearch, and display it according to priority.

[1082] 2. Processing text data:

[1083] The NLTK library is used to extract frequently occurring words from the converted text data. The extracted keywords are important words that reflect the conversation content and are used in the search query.

[1084] 3. Processing of emotional data:

[1085] Text data is sent to the emotion engine, and emotion data is generated using Azure's Emotion API. This emotion data will play a crucial role in the subsequent information provision process.

[1086] 4. Searching for and displaying related content:

[1087] Based on the extracted frequent words and sentiment data, the server searches the Elasticsearch database to find relevant content. The content is sorted according to priority based on sentiment state and displayed on the smart display. For example, if it is recognized that a passenger is "talking about restaurants nearby," restaurant information will be displayed preferentially.

[1088] Specific example

[1089] Let's say a passenger is talking on the train and says, "I want to know about tourist attractions in this area." The server captures this conversation audio and uses the Google Speech-to-Text API to convert "tourist attractions in this area" into text. Using the NLTK library, "tourist attractions" is extracted as a key keyword, and the passenger's level of interest is further analyzed by an emotion engine. The Elasticsearch database is searched, and information based on the keyword "tourist attractions" is displayed preferentially. For example, "popular tourist attractions in this area" or "recommended tourist destinations list" might be displayed on the smart display.

[1090] Thus, this invention realizes a system that can improve passenger convenience and satisfaction in autonomous vehicles by providing information in real time that corresponds to the content of passengers' or users' conversations and emotional states.

[1091] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1092] Step 1:

[1093] The server acquires audio data using microphones inside the autonomous vehicle. The acquired audio data is saved as is. Specifically, it is recorded in an audio file format (e.g., a .wav file).

[1094] Input: Audio of conversations between passengers and users inside the vehicle.

[1095] Output: Audio data file (.wav)

[1096] Step 2:

[1097] The server sends the audio data to the Google Speech-to-Text API, where it is converted into text data. The converted text data is then stored for subsequent processing and formatted into a concise form.

[1098] Input: Audio data file (.wav)

[1099] Output: Text data

[1100] Step 3:

[1101] The server uses the NLTK library to extract frequently occurring words from text data. It tokenizes (segments) the text data, calculates the frequency of each word, and picks out the most frequent words.

[1102] Input: Text data

[1103] Output: List of frequently occurring words

[1104] Step 4:

[1105] The server sends the extracted text data to the emotion engine, which uses Microsoft Azure's Emotion API to generate emotion data. This emotion data provides a detailed description of the user's emotional state, such as positive, negative, or excited.

[1106] Input: Text data

[1107] Output: Sentiment data

[1108] Step 5:

[1109] The server searches the Elasticsearch database based on the extracted list of frequent words and sentiment data. The extracted frequent words are used in the search queries, and the search results are prioritized based on the sentiment data.

[1110] Input: List of frequently occurring words, sentiment data

[1111] Output: Related Content List

[1112] Step 6:

[1113] The server selects relevant content from the search results in order of priority and displays it on the smart display. For example, if a passenger is talking about "restaurants," information about nearby restaurants will be displayed.

[1114] Input: Related Content List

[1115] Output: Content displayed on the smart display

[1116] Step 7:

[1117] The system uses a terminal to identify terms that passengers may not understand and generates and displays explanations of those terms along with links to related resources. For example, if the term "digital transformation" is included, an explanation and a link to more detailed resources will be displayed.

[1118] Input: List of frequently occurring words

[1119] Output: Glossary of terms, links to related resources

[1120] The above outlines the specific processing steps for implementing this invention. This enables the real-time provision of information tailored to the conversation content and emotional state of passengers and users.

[1121] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1122] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1123] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1124] [Fourth Embodiment]

[1125] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1126] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1127] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1128] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1129] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1130] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1131] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1132] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1133] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1134] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1135] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1136] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1137] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1138] This invention relates to a system that converts audio data, such as from meetings, into text data, extracts frequently occurring words from that text data, and searches for and provides related materials from an internal company database based on those words. It also provides explanations of the latest hot topics related to the frequently occurring words, as well as terms that employees may not be familiar with.

[1139] Acquisition of audio data and text conversion

[1140] The user uploads meeting audio data (e.g., a .wav file) to the device. The device sends this audio data to a speech recognition API, which converts the audio data into text data. The converted text data is then formatted into a specific format within the device.

[1141] Extraction of frequently occurring words

[1142] The terminal uses natural language processing tools to extract frequently occurring words from the converted text data. For example, if the meeting content was "discussing future market strategies," keywords such as "market" and "strategy" would be extracted.

[1143] Searching for and providing related materials

[1144] Next, the terminal sends the extracted list of frequently occurring words to the server. The server searches for documents in the company's database based on the received list of frequently occurring words. The search results are sorted by calculating the synchronization rate between the content of each document and the frequently occurring words, and then sorting them in descending order of synchronization rate. As a result, documents such as "Strategic Plan 2023.pdf" and "Market Analysis Report 2023.pptx" are listed in reverse chronological order. The server generates links to these sorted documents and provides them to the user.

[1145] Categorizing materials and providing hot topics

[1146] The server also has the function of organizing all documents in the company's internal database into categories based on themes and attributes. This allows for efficient management of documents by category, such as "Market Analysis" and "Strategy Formulation." Furthermore, it selects and presents the latest documents related to frequently used keywords as hot topics. For example, within the "Market Analysis" category, the latest "September Updated Market Report.pdf" is presented as a hot topic.

[1147] Explanation of terminology and improvement of employee skills

[1148] The device identifies terms that the user may not understand from a list of frequently used words. For example, if the frequently used word "digital transformation" is included, the device will determine, based on past data and the user's skill level, that this term is unfamiliar. The device will generate an easy-to-understand explanation of this term and also display a link to related materials (e.g., "DX Introduction Guide.pdf").

[1149] Explanation with specific examples

[1150] The following is a concrete example. An audio file, "Meetings_aug30.wav," from a meeting held on August 30th is uploaded to the terminal. The terminal uses a speech recognition API to convert this audio file into text, and the content is rephrased as "Discussion on future market strategy policies." A natural language processing tool is used to extract frequently occurring words such as "market" and "strategy," and this is sent to the server. The server searches the company's internal database for related documents such as "Strategic Plan 2023.pdf" and "Market Analysis Report 2023.pptx," sorts them in reverse chronological order, and generates links. Furthermore, it presents the latest "September Market Report.pdf" as a hot topic related to "market analysis," and provides the user with an explanation of "digital transformation" and related materials.

[1151] Thus, the present invention provides a system that efficiently searches for and provides relevant materials based on meeting content, and contributes to improving employees' knowledge.

[1152] The following describes the processing flow.

[1153] Step 1:

[1154] The user uploads the meeting audio data (e.g., a .wav file) to their device. The audio data is in a format such as "Meetings_aug30.wav".

[1155] Step 2:

[1156] The device sends the uploaded audio data to a speech recognition API, which converts the audio data into text data. This is where text such as "Strategic Meeting August 30, 2023" is generated.

[1157] Step 3:

[1158] The terminal receives the converted text data and formats it by removing unnecessary information (such as noise or irrelevant conversation). For example, unnecessary words like "um" or "well" are removed.

[1159] Step 4:

[1160] The device uses natural language processing tools to extract frequently occurring words from the formatted text. For example, keywords such as "strategy," "market," and "analysis" might be extracted.

[1161] Step 5:

[1162] The terminal sends the extracted list of frequently occurring words to the server. This list includes not only the frequently occurring words but also their frequency of occurrence.

[1163] Step 6:

[1164] Based on the frequently occurring word list received by the server, all documents in the company database are searched. The degree of agreement between the document content and the frequently occurring words, i.e., the synchronization rate, is calculated.

[1165] Step 7:

[1166] The server sorts the documents with the highest synchronization rate by newest first. For example, "Strategic Plan 2023.pdf" and "Latest Market Analysis Report 2023.pptx" are output.

[1167] Step 8:

[1168] The server generates links to a sorted list of materials and provides them to the user via the terminal. This allows the user to easily access relevant materials.

[1169] Step 9:

[1170] The server organizes documents within the company's internal database by category. For example, they might be divided into categories such as "Strategy," "Market," and "Analysis."

[1171] Step 10:

[1172] The server searches for and presents the latest hot topics related to frequently occurring keywords. For example, "September Market Report.pdf" might be presented as a hot topic within the "Market Analysis" category.

[1173] Step 11:

[1174] The device identifies terms that the user may not understand from among frequently used words. For example, "digital transformation" might be identified in this way.

[1175] Step 12:

[1176] The device provides explanations of terms it has identified and links to related materials. For example, it might display an explanation such as, "Digital transformation is the process by which companies use digital technologies to transform their business models," along with a link to "DX Introduction Guide.pdf".

[1177] (Example 1)

[1178] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1179] Traditional methods of recording and analyzing meetings made it difficult to efficiently utilize audio data, requiring significant time and effort for manual text conversion and document retrieval. Furthermore, the accuracy and speed of searching for and providing relevant materials were limited, and explanations of unfamiliar terminology were insufficient. Therefore, a system was needed that could simultaneously achieve effective utilization of meeting data and improve knowledge.

[1180] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1181] In this invention, the server includes means for converting audio data into text data, means for extracting frequently occurring words from the converted text data, means for searching a database for related information based on the extracted frequently occurring words and arranging it in order of decreasing synchronization rate, means for generating links to the arranged information, means for using an external API for recognizing audio data, and means for using a natural language processing tool for extracting frequently occurring words. This makes it possible to efficiently transcribe meeting audio data into text, provide related materials quickly and accurately, and simultaneously improve employees' knowledge.

[1182] "Audio data" refers to digital data files containing recordings of audio from meetings, conversations, and other similar events.

[1183] "Text data" refers to a digital data file created by converting audio data into text.

[1184] "Frequently occurring words" refer to words or expressions that appear with a specific frequency within text data.

[1185] An "external API" refers to an application programming interface provided by an external service or platform.

[1186] "Natural language processing tools" refer to software and libraries used to analyze text data, understand its meaning and grammar, and process it.

[1187] A "database" refers to a collection of information that allows for the efficient management and retrieval of large amounts of data.

[1188] A "link" is a hypertext element that allows a user to access a specific webpage or file by clicking on it.

[1189] "Synchronization rate" refers to an indicator that shows the degree of agreement between frequently occurring words and the content of the materials in the database.

[1190] "Explanation" refers to a text or explanation that clearly describes unfamiliar or unknown terms.

[1191] This invention is a system that converts audio data, such as from meetings, into text data, extracts frequently occurring words from that text data, and searches and provides related materials from a database based on those words. Furthermore, it also provides the latest topics related to the frequently occurring words and explanations of terms that the user may not be familiar with. Embodiments of this system are described below.

[1192] First, the user uploads the meeting audio data (e.g., a .wav file) to their device. The device then sends this audio data to a speech recognition API (e.g., Google Speech-to-Text or Amazon Transcribe) to convert it into text data. The converted text data is then formatted into a specific format (e.g., date and time, speaker, content of speech) and processed.

[1193] Next, the terminal inputs the converted text data into a natural language processing tool (e.g., NLTK or spaCy) to extract frequently occurring words. For example, if the meeting topic is "discussing future market strategies," frequently occurring words such as "market" and "strategy" will be extracted. The extraction results are compiled into a list of frequently occurring words.

[1194] The terminal sends the extracted list of frequently occurring words to the server. The server searches the database using the ElasticSearch engine or similar based on the list. The search results are sorted by calculating the synchronization rate between the content of each document and the frequently occurring words, and then sorted in descending order of synchronization rate. This results in a list of documents such as "Strategic Plan 2023.pdf" and "Market Analysis Report 2023.pptx," and links to the sorted documents are generated.

[1195] Furthermore, the server organizes all documents in the company database by theme (e.g., "Market Analysis" or "Strategy Formulation"), selects the latest information related to frequently used keywords, and presents it to the user. For example, within the "Market Analysis" category, the latest "September Updated Market Report.pdf" might be presented as a hot topic.

[1196] Furthermore, the device identifies terms from a list of frequently used words that the user may not understand, based on past learning data. For example, if the frequently used word "digital transformation" is identified as unfamiliar, it generates an easy-to-understand explanation and provides a link to related materials (e.g., "DX Introduction Guide.pdf").

[1197] Specific example:

[1198] An audio file, "Meetings_aug30.wav," from a meeting held on August 30th is uploaded to the user's device. This audio file is converted into text data via a speech recognition API on the device. The content is translated to something like "Discussion on future market strategy." Next, a natural language processing tool is used to extract frequently occurring words such as "market" and "strategy," and this is sent to the server. The server searches its database for related documents such as "Strategic Plan 2023.pdf" and "Market Analysis Report 2023.pptx," sorts them in reverse chronological order, and generates links. Furthermore, the latest "September Market Report.pdf" is presented as a hot topic related to "market analysis," and the user is provided with an explanation of "digital transformation" and related materials.

[1199] Example of a prompt

[1200] The following are specific examples of prompt statements that will be used as input to the generative AI model.

[1201] "Please describe a program to search for and provide materials related to the topics of 'market' and 'strategy' discussed at the August 30th meeting. Additionally, please explain the term 'digital transformation' and add a function to provide related materials."

[1202] This system enables efficient transcription of meeting content, prompt and accurate provision of related materials, and contributes to improving employees' knowledge.

[1203] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1204] Step 1: Acquire audio data and convert it to text.

[1205] The user uploads meeting audio data (e.g., a .wav file) to their device. The input is an audio data file. The device sends this audio data to the Google Speech-to-Text API and retrieves text data. The output is the converted text data. Specifically, the audio data is passed to the API, and the text data is received as the API response.

[1206] Step 2: Formatting the text data

[1207] The terminal formats the acquired text data into a specific format (e.g., date and time, speaker, content of the message). The input is text data obtained from the API. The output is the formatted text data. Specifically, the terminal organizes the text data using methods such as JSON or CSV format to make it suitable for subsequent processing.

[1208] Step 3: Extraction of frequently occurring words

[1209] The terminal inputs formatted text data into a natural language processing tool (e.g., NLTK or spaCy) and extracts frequently occurring words. The input is formatted text data. The output is a list of frequently occurring words. Specifically, the text data is tokenized, the frequency of each word is counted, and words exceeding a certain frequency are extracted.

[1210] Step 4: Send the list of frequently used words

[1211] The terminal sends the extracted list of frequently occurring words to the server. The input is the list of frequently occurring words. The output is the data sent to the server. Specifically, the process involves encoding the list of frequently occurring words in JSON format and sending it to the server using an HTTP request.

[1212] Step 5: Search for related materials

[1213] The server uses the Elasticsearch engine to search for materials in the database based on the received list of frequently occurring words. The input is the list of frequently occurring words. The output is the search results. Specifically, it generates an Elasticsearch query and executes the query against the database.

[1214] Step 6: Calculate and sort the data synchronization rate.

[1215] The server calculates the synchronization rate between the content of each document and its frequently occurring words, and sorts them in descending order of synchronization rate. The input is the search results. The output is the sorted list of documents. Specifically, it scores how well the frequently occurring words match the content of each document and sorts the documents based on that score.

[1216] Step 7: Generate and provide links

[1217] The server generates and provides links to sorted documents to the user. The input is a sorted list of documents. The output is links to the documents. Specifically, the server generates the links in HTML format, making them accessible to the user.

[1218] Step 8: Categorize materials and provide hot topics

[1219] The server organizes all documents in the company database by theme, selects the latest information related to frequently used keywords, and presents it to the user. Inputs are a list of documents and frequently used keywords. Outputs are documents organized by category and the latest hot topics. Specifically, it uses a classification algorithm to group documents by theme and selects the most recent documents.

[1220] Step 9: Terminology identification and explanation generation

[1221] The device identifies terms that the user may not understand from a list of frequently used words, generates easy-to-understand explanations for the unfamiliar terms, and displays links to related materials. Input consists of the list of frequently used words and the user's past learning data. Output is the term explanation and links. Specifically, it executes a discrimination algorithm based on the learning data, generates explanatory text, and attaches links to related materials.

[1222] (Application Example 1)

[1223] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1224] While it is crucial to respond quickly and accurately to customer questions and requests during customer service in stores, delays can occur when staff are busy or when there is inconsistency in product knowledge. Furthermore, insufficient information provided to customers can lead to decreased customer satisfaction. Additionally, it is difficult for new or part-time staff to fully grasp specialized terminology and detailed product information, which can result in a decline in the quality of customer service. A system is needed to address these challenges and improve customer satisfaction.

[1225] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1226] This invention includes a server that transcribes audio data of customer conversations into text in real time and searches for and provides related product information based on frequently occurring words; a server that searches for related materials from a database based on the extracted frequently occurring words and arranges them in order of synchronization rate; and a server that generates explanations for the identified terms and presents links to related materials. This makes it possible to convert conversations with customers into text data and quickly provide related product information and detailed explanations based on frequently occurring questions and requests.

[1227] "Audio data" refers to sound information, such as conversations and music, recorded and stored in digital format.

[1228] "Text data" refers to a digital representation of character information.

[1229] "Frequently occurring words" refer to words or phrases that appear particularly often within the text data.

[1230] A "database" is a system for efficiently managing and retrieving data, and is a collection of structured information.

[1231] "Synchronization rate" is an indicator that shows the degree of agreement between frequently occurring words and the data in the database.

[1232] A "link" is reference information that allows direct access to specific data or materials.

[1233] "Customer service" refers to the activities of interacting with and providing services to customers in stores and service industries.

[1234] "Real-time text conversion" is the process of instantly converting audio data into text data.

[1235] "Related product information" refers to detailed data and descriptions of products found through searches based on frequently occurring keywords.

[1236] "Category organization" is the process of classifying and managing data and materials based on common attributes or themes.

[1237] "Hot topics" refer to the latest information or topics that have recently become popular.

[1238] An "explanation" is a detailed explanation of a specific term or concept.

[1239] System program

[1240] The apparatus for carrying out this invention consists of a system in which a server, a terminal, and a user work together.

[1241] The terminal has the functionality to acquire voice data during customer service and convert it into text data. To achieve this, the Google Cloud Speech-to-Text API and Python's speech_recognition library are used. In addition, frequently occurring words are extracted from the converted text data using natural language processing tools (e.g., spaCy, NLTK).

[1242] The server has the functionality to search for related materials and product information from existing databases (e.g., Firebase) based on a list of frequently used words sent from the terminal, sort them in order of synchronicity, and generate links. Furthermore, it also has the functionality to present hot topics related to frequently used words.

[1243] Program Processing Description

[1244] 1. Acquisition and transcription of audio data

[1245] The system records conversations during customer service using a terminal and sends the recordings to the Google Cloud Speech-to-Text API, converting the audio data into text data. This makes the audio information easily searchable text data.

[1246] 2. Extraction of frequently occurring words

[1247] The converted text data is analyzed using natural language processing tools (NLTK, spaCy) to extract frequently occurring words. This allows for a quick understanding of customer interests and question trends.

[1248] 3. Searching for and providing related information

[1249] The device sends the extracted list of frequently occurring words to the server. The server searches its database (Firebase) for relevant product information and materials, sorts them in the optimal order based on the synchronization rate, and generates links.

[1250] 4. Explanation of hot topics and terminology

[1251] The server generates and presents detailed explanations, along with links, of the latest hot topics related to frequently used words and terms that customers may not understand.

[1252] Examples of specific cases and prompt statements

[1253] Specific example:

[1254] For example, if frequently occurring keywords such as "new product" and "promotion" are extracted from an audio file recorded during customer service on August 30th, the server will search for detailed information on products related to these keywords and the latest campaign information, and present them in a list sorted by the degree of synchronization. Furthermore, if it is determined that the term "promotion" is not well understood by new staff, an explanation of the term and links to related materials will also be provided.

[1255] Example of a prompt:

[1256] "We want to develop an application that transcribes audio data of customers being served on August 30th into text, extracts frequently occurring words, and then searches for and provides related product information and the latest promotional information."

[1257] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1258] Step 1:

[1259] The terminal acquires audio data during customer service. The user (store clerk) uses a smartphone or dedicated device to record the audio data of the conversation. This audio data becomes the input data, and in this step, an audio file is specifically generated.

[1260] Step 2:

[1261] The device sends the acquired audio data to the Google Cloud Speech-to-Text API, where it is converted into text data. The input here is the audio file obtained in step 1, and the API is used to convert that audio into text format and output it. Specifically, the audio data is sent to the API, and the converted text is returned from the API.

[1262] Step 3:

[1263] The terminal analyzes the converted text data using a natural language processing tool (e.g., spaCy or NLTK) and extracts frequently occurring words. The input is the text data obtained in step 2, and data processing such as word segmentation and part-of-speech analysis is performed using NLTK or spaCy to extract and output frequently occurring words. Specifically, the text is broken down, and the words that appear most frequently are listed.

[1264] Step 4:

[1265] The terminal sends the extracted list of frequently occurring words to the server. The input is the list of frequently occurring words obtained in step 3, and the terminal sends data to the server based on this. Specifically, the operation to send the list of frequently occurring words to the API endpoint is performed.

[1266] Step 5:

[1267] The server searches for relevant documents and product information from a database (such as Firebase) based on the list of frequently occurring words. The input for this step is the list of frequently occurring words submitted in step 4. The server searches the database to retrieve relevant documents and product information and outputs it as a list sorted by synchronization rate. Specifically, it forms a search query using frequently occurring words as keywords and extracts the corresponding data from the database.

[1268] Step 6:

[1269] The server generates links to relevant documents and product information from the search results and sends them to the terminal. The input is the search results obtained in step 5, and it generates and outputs links to the detail pages for each document and product. Specifically, it uses a link generation algorithm to form URLs, lists them, and sends them.

[1270] Step 7:

[1271] The server identifies hot topics and terms that customers may not understand, and generates explanations and related materials. The input is the search results obtained in step 5 and the user profile, and it generates and outputs explanatory text and links to related materials. Specifically, it identifies unfamiliar terms from frequently occurring words and historical data, and provides explanatory text and links to related materials based on those terms.

[1272] Step 8:

[1273] The terminal displays the received link and explanation to the user (store clerk) and provides it to the customer. The input consists of the link and explanation generated in steps 6 and 7, and is output by being displayed on a screen such as a smartphone. Specifically, the link is displayed in a clickable format, and the explanation is displayed in an information box.

[1274] Through these steps, users can respond quickly and accurately to customer questions and requests, thereby improving customer satisfaction.

[1275] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1276] This invention relates to a system that converts audio data, such as from meetings, into text data, extracts frequently occurring words from that text data, and searches for and provides related materials from an internal company database based on those words. Furthermore, by combining this with an emotion engine that recognizes the user's emotions, the system can also provide materials and change their priority according to the user's emotional state.

[1277] Acquisition of audio data and text conversion

[1278] The user uploads meeting audio data (e.g., a .wav file) to the device. The device sends this audio data to a speech recognition API, which converts the audio data into text data. The converted text data is then formatted into a specific format within the device.

[1279] Extraction of frequently occurring words

[1280] The terminal uses natural language processing tools to extract frequently occurring words from the converted text data. For example, if the meeting content was "discussing future market strategies," keywords such as "market" and "strategy" would be extracted.

[1281] Utilizing the Emotion Engine

[1282] The device utilizes an emotion engine to recognize the user's emotions in real time. For example, the emotion engine analyzes the user's facial expressions and voice tone while they are viewing documents and generates emotion data.

[1283] Searching for and providing related materials

[1284] Next, the terminal sends the extracted list of frequently occurring words and sentiment data to the server. The server searches for documents in the company's database based on the received list of frequently occurring words and sentiment data. The search results are sorted using a new criterion that integrates synchronization rate and sentiment influence, taking into account the degree of match between the content of each document and the frequently occurring words, as well as importance based on the user's sentiment data.

[1285] For example, if a user expresses positive feelings towards "marketing strategy," materials related to that topic will be displayed at the top of the search results. On the other hand, if negative feelings are detected, materials with a higher priority for mitigating those feelings will be presented.

[1286] Categorizing materials and providing hot topics

[1287] The server also has the function of organizing all documents in the company's internal database into categories based on themes and attributes. This allows for efficient management of documents by category, such as "Market Analysis" and "Strategy Formulation." Furthermore, it selects and presents the latest hot topics based on sentiment data. For example, within the "Market Analysis" category, the latest "September Updated Market Report.pdf" is presented as a hot topic.

[1288] Explanation of terminology and improvement of employee skills

[1289] The device identifies terms that the user may not understand from a list of frequently used words. For example, if the frequently used word "digital transformation" is included, the device will determine, based on past data and the user's skill level, that this term is unfamiliar. The device will generate an easy-to-understand explanation of this term and also display a link to related materials (e.g., "DX Introduction Guide.pdf").

[1290] Explanation with specific examples

[1291] The following is a concrete example. An audio file, "Meetings_aug30.wav," from a meeting held on August 30th is uploaded to the terminal. The terminal uses a speech recognition API to transcribe this audio file into text, and the content is translated into something like "Discussion on future market strategy." A natural language processing tool is used to extract frequently occurring words such as "market" and "strategy," and an emotion engine analyzes the user's emotional state (e.g., excitement, interest, negative emotion) in real time. The extracted list of frequently occurring words and emotion data are sent to the server.

[1292] The server searches the internal database for relevant documents such as "Strategic Plan 2023.pdf" and "Market Analysis Report 2023.pptx," and re-evaluates their priority based on synchronization rate and sentiment data. The documents are then sorted in the new order, and links are generated. Furthermore, the latest "September Market Report.pdf" is presented as a hot topic, and the user is provided with an explanation of "Digital Transformation" and related materials (e.g., "DX Introduction Guide.pdf").

[1293] Thus, the present invention provides a system that efficiently searches for and provides relevant materials based on meeting content, and realizes information provision that also takes into account the user's emotional state.

[1294] The following describes the processing flow.

[1295] Step 1:

[1296] The user uploads the meeting audio data (e.g., a .wav file) to their device. The audio data is, for example, named "Meetings_aug30.wav".

[1297] Step 2:

[1298] The device sends the uploaded audio data to a speech recognition API, which converts the audio data into text data. For example, this might result in "Meeting content: Market strategy and cost reduction."

[1299] Step 3:

[1300] The terminal receives the converted text data and performs preprocessing to remove unnecessary characters and noise, and then formats it. For example, unnecessary parts such as "um" or "well" are removed.

[1301] Step 4:

[1302] The device uses natural language processing tools to extract frequently occurring words from pre-processed text data. For example, "market," "strategy," and "cost reduction" might be extracted.

[1303] Step 5:

[1304] The terminal sends the extracted list of frequently occurring words to the server. This list includes each frequently occurring word and its frequency of occurrence.

[1305] Step 6:

[1306] The server searches for documents within the company database based on the list of frequently occurring words it receives. It then calculates the synchronization rate between the content of each document and the list of frequently occurring words.

[1307] Step 7:

[1308] The server sorts the documents with the highest synchronization rate by newest first. For example, "Strategic Plan 2023.pdf" and "Latest Market Analysis Report 2023.pptx" are output.

[1309] Step 8:

[1310] The server generates links to a sorted list of materials and provides them to the user via the terminal. This allows the user to easily access relevant materials.

[1311] Step 9:

[1312] The device uses an emotion engine to analyze the user's emotions in real time. For example, it performs facial recognition and voice tone analysis to collect data on the emotional state the user exhibits while viewing materials.

[1313] Step 10:

[1314] The device sends emotional data to the server. For example, if a user expresses positive emotions towards "marketing strategy," that data is sent to the server.

[1315] Step 11:

[1316] The server re-evaluates the priority of previously searched materials based on sentiment data. Materials related to topics showing positive sentiment will appear higher in the search results.

[1317] Step 12:

[1318] If a server is showing negative emotions, resources to alleviate those emotions will be recommended. For example, resources that help reduce stress or solve problems will be selected and presented preferentially.

[1319] Step 13:

[1320] The server organizes all documents in the company's internal database by category. For example, they might be classified into categories such as "Strategy," "Market," and "Analysis."

[1321] Step 14:

[1322] The server searches for and presents the latest hot topics related to frequently occurring keywords. For example, "September Market Report.pdf" might be presented as a hot topic.

[1323] Step 15:

[1324] The device identifies terms that the user may not understand from among frequently used words. For example, "digital transformation" might be identified in this way.

[1325] Step 16:

[1326] The device provides explanations of terms it has identified and displays links to related materials. For example, it might provide an explanation such as, "Digital transformation is the process by which companies use digital technologies to transform their business models," along with a link to "DX Introduction Guide.pdf."

[1327] (Example 2)

[1328] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1329] Conventional systems that convert meeting audio into text data and provide related materials only utilize speech recognition and extraction of frequently occurring words, failing to provide information that takes user emotions into consideration. Therefore, there were issues such as users not receiving the information they needed appropriately and being unable to respond to changes in their emotions during use. Furthermore, features to deepen understanding of meeting content, such as category organization and glossary, were insufficient.

[1330] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1331] In this invention, the server includes means for converting audio data such as meetings into text data, means for extracting frequently occurring words from the converted text data, and means for searching for relevant materials from the company's internal database based on the extracted frequently occurring words and user sentiment data, and sorting them by synchronization rate and priority based on sentiment data. This makes it possible to search for and provide relevant materials that take into account the user's emotional state. Furthermore, by adding means for presenting the latest hot topics related to frequently occurring words and sentiment data, and means for identifying terms that the user may not understand and generating and providing explanations, more efficient and user-friendly information provision can be achieved.

[1332] 1. "Audio data" refers to data in digital file format that records audio, including oral communication such as meetings and interviews.

[1333] 2. "Text data" refers to character information converted from audio data, and is data expressed in a readable format.

[1334] 3. "Frequently occurring words" refer to words or phrases that appear particularly often in the text data and are extracted as important keywords.

[1335] 4. "Emotional data" refers to digital data generated as a result of analyzing a user's emotional state (e.g., positive, negative, excited, etc.).

[1336] 5. An "internal database" is a digital storage system used to accumulate and manage various documents and data within a company.

[1337] 6. "Related materials" refer to highly relevant documents and digital content searched from the company's internal database based on the extracted frequently occurring words and sentiment data.

[1338] 7. "Synchronization rate" is an indicator that shows how well the extracted frequently occurring words match the content of related materials.

[1339] 8. "Priority" is an indicator that influences the display order of search results and is determined based on the match rate of frequently occurring words and sentiment data.

[1340] 9. A "link" is a hypertext-based reference that allows a user to navigate to a specific document or webpage by clicking on it.

[1341] 10. A "category" is a classification of related materials based on subject matter or attributes, and is a concept designed to enable users to efficiently search for and utilize information.

[1342] 11. "Hot topics" refer to the latest information and topics considered to be of high interest, and are determined based on user interest and sentiment data.

[1343] 12. A "glossary" is a document that clearly explains the meaning and background of specific terms that users may not understand.

[1344] This invention is a system that converts audio data from meetings and other events into text data, extracts frequently occurring words from that text data, and searches for and provides related materials from the company's internal database based on those words. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it can also provide materials and change their priority according to the user's emotional state.

[1345] Hardware and software to be used

[1346] Terminal: A device on which users upload audio data and perform tasks such as displaying materials and emotion recognition. Examples include PCs and smartphones.

[1347] Server: A central server that performs tasks such as converting audio data to text, extracting frequently occurring words, searching for related materials, and prioritizing the order of materials.

[1348] software:

[1349] Speech-to-Text API: Google Cloud Speech-to-Text

[1350] Natural language processing tools: NLTK, spaCy

[1351] Emotion engine: Microsoft Azure Emotion API

[1352] Database search engine: Elasticsearch

[1353] Acquisition of audio data and text conversion

[1354] The user uploads meeting audio data (e.g., a .wav file) to their device. The device sends this audio data to a speech recognition API, which converts the audio data into text data. The converted text data is then formatted within the device into a specific format (e.g., JSON format).

[1355] Specific example:

[1356] The user clicks the "Select audio file" button, selects "Meetings_aug30.wav" from their local file system, and uploads it.

[1357] The device sends this audio file to the Google Cloud Speech-to-Text API, where it is converted into text data. For example, the converted text data might read, "Discussing future market strategy guidelines."

[1358] Extraction of frequently occurring words

[1359] The device uses natural language processing tools to extract frequently occurring words from the converted text data. For example, keywords such as "market" and "strategy" are extracted. The extracted keywords are then compiled into a list.

[1360] Specific example:

[1361] The terminal analyzes text data using NLTK and spaCy to extract frequently occurring words such as "market" and "strategy."

[1362] The extracted keywords are listed. For example, the list might look like this: ["market", "strategy"].

[1363] Utilizing the Emotion Engine

[1364] The device utilizes an emotion engine to recognize the user's emotions in real time. For example, it analyzes the user's facial expressions and voice tone while they are viewing documents and generates emotion data.

[1365] Specific example:

[1366] The device captures the user's facial expressions with its camera and acquires their voice tone with its microphone.

[1367] This data is sent to the Microsoft Azure Emotion API, and the generated emotion data is received. For example, the data might look like this: {"emotion": "positive", "confidence": 0.92}.

[1368] Searching for and providing related materials

[1369] Next, the terminal sends the extracted list of frequently occurring words and sentiment data to the server. The server searches for documents in the company's database based on the received list of frequently occurring words and sentiment data. The search results are sorted considering the degree of match between the content of each document and the frequently occurring words, as well as their importance based on the user's sentiment data.

[1370] Specific example:

[1371] The server uses Elasticsearch to search for materials that match "market" and "strategy".

[1372] For example, "Strategic Plan 2023.pdf" and "Market Analysis Report 2023.pptx" are searched, and their priority is re-evaluated based on relevance and sentiment data.

[1373] Categorizing materials and providing hot topics

[1374] The server also has the function of organizing all documents in the company's internal database into categories based on themes and attributes. This allows for efficient management of documents by category, such as "market analysis" and "strategy formulation." Furthermore, it selects and presents the latest hot topics to users based on sentiment data.

[1375] Specific example:

[1376] Based on sentiment data, the latest "September Updated Market Report.pdf" is presented as a hot topic.

[1377] Explanation of terminology and improvement of employee skills

[1378] The terminal identifies terms that the user may not understand from a list of frequently used words. For example, if the frequently used word "digital transformation" is included, the terminal will determine, based on past data and the user's skill level, that this term is unfamiliar. The terminal will generate an easy-to-understand explanation of this term and also display a link to related materials (e.g., "DX Introduction Guide.pdf").

[1379] Specific example:

[1380] The device recognizes the keyword "digital transformation" and displays links to related explanatory texts and documents (e.g., "DX Introduction Guide.pdf").

[1381] Example of a prompt:

[1382] I uploaded the meeting audio data. Frequently occurring words such as "market strategy" and "future plans" were extracted. The user's sentiment is positive, and they are interested in materials related to market strategy. Please search and sort the related materials. Also, please provide an explanation and related materials on "digital transformation."

[1383] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1384] Step 1: Upload audio data

[1385] The user uploads meeting audio data (e.g., a .wav file) to the device. The input is the audio file, and the output is a notification that the upload to the device is complete. The device saves the input audio data to temporary storage and prepares to send it to the speech recognition API in the next step.

[1386] Specific actions:

[1387] The user clicks the "Select audio file" button and selects "Meetings_aug30.wav" from the local file system.

[1388] The device accepts the file and displays a notification that the upload is complete.

[1389] Step 2: Convert audio to text

[1390] The device sends audio data to a speech recognition API (e.g., Google Cloud Speech-to-Text) and converts it into text data. The input is audio data, and the output is converted text data (e.g., in JSON format). The device receives the text data and formats it into a specific format.

[1391] Specific actions:

[1392] The device sends "Meetings_aug30.wav" to the Google Cloud Speech-to-Text API.

[1393] The device receives the text data returned from the API. For example, the data is formatted in the format {"text": "Discussion on future market strategy"}.

[1394] Step 3: Extraction of frequently occurring words

[1395] The terminal extracts frequently occurring words from the converted text data using natural language processing tools (e.g., NLTK, spaCy). The input is text data, and the output is a list of frequently occurring words. The terminal analyzes the text data to extract the main keywords of the content discussed by the user.

[1396] Specific actions:

[1397] The terminal analyzes text data using NLTK and spaCy to extract frequently occurring words such as "market" and "strategy."

[1398] The terminal lists the extracted keywords. For example, it generates a list such as ["market", "strategy"].

[1399] Step 4: Analysis using the emotion engine

[1400] The device uses an emotion engine (e.g., Microsoft Azure Emotion API) to recognize the user's emotions in real time. The input is the user's facial expressions and voice tone, and the output is emotion data. The device analyzes the collected data to confirm the user's emotional state.

[1401] Specific actions:

[1402] The device captures the user's facial expressions with its camera and records their voice tone with its microphone while they are viewing the document.

[1403] The device sends the captured data to the Microsoft Azure Emotion API and receives emotion data. For example, it might receive data such as {"emotion": "positive", "confidence": 0.92}.

[1404] Step 5: Search and sort related materials

[1405] The terminal sends the extracted list of frequently occurring words and sentiment data to the server. The input is the list of frequently occurring words and sentiment data, and the output is a sorted list of related documents. The server accesses the internal database, searches for the relevant documents, and sorts them based on priority.

[1406] Specific actions:

[1407] The device sends a list of frequently occurring words and sentiment data to the server.

[1408] The server uses Elasticsearch to search for materials that match the terms "market" and "strategy".

[1409] The server sorts search results based on the degree of relevance of frequently occurring words and sentiment data. For example, "Strategic Plan 2023.pdf" and "Market Analysis Report 2023.pptx" will appear at the top of the search results.

[1410] Step 6: Display and provide materials

[1411] The server sends a sorted list of materials to the terminal. The input is the sorted list of materials, and the output is the material links displayed to the user. The terminal displays the material links to the user based on the search results.

[1412] Specific actions:

[1413] The server generates a sorted list of documents and sends it to the terminal.

[1414] The device displays links to documents on the user's screen. For example, "Strategic Plan 2023.pdf", "Market Analysis Report 2023.pptx", and "September Updated Market Report.pdf" will be displayed.

[1415] Step 7: Explanation of terminology and provision of related materials

[1416] The device identifies terms that the user may not understand from among frequently used words and generates explanations. The input is frequently used words, and the output is a term definition and links to related materials. It generates explanatory text for the identified terms and displays links to related materials to the user.

[1417] Specific actions:

[1418] The device recognizes keywords such as "digital transformation."

[1419] The terminal generates an explanation of "digital transformation" and links to related materials (e.g., "DX Introduction Guide.pdf").

[1420] The user's screen displays explanatory text and links to related resources.

[1421] (Application Example 2)

[1422] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1423] Current autonomous vehicles offer limited information tailored to the interests and emotions of passengers and users, and lack systems that provide relevant information in real time based on conversation. This makes it difficult to improve passenger convenience and satisfaction. Furthermore, there is a lack of appropriate means to provide information about terminology that passengers do not understand.

[1424] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for converting audio data such as meetings into text data, means for extracting frequently occurring words from the converted text data, means for searching for related materials from a database based on the extracted frequently occurring words and arranging them in order of synchronization rate, means for generating links to the arranged materials, means for converting conversations of passengers or users into text data, extracting frequently occurring words from the text data and recognizing the emotional state of passengers in real time, and means for searching for related content based on the extracted frequently occurring words and emotional data, and displaying it with priority. As a result, relevant information can be obtained in real time from conversations that passengers have in an autonomous vehicle, and information can be provided while also taking into account the emotions of passengers. In addition, by appropriately providing explanations of terms that passengers do not understand, passenger satisfaction and convenience can be greatly improved.

[1425] "Audio data from meetings, etc." refers to audio information recorded during meetings or other similar gatherings.

[1426] "Means of converting to text data" refers to methods or techniques for converting audio data into text information.

[1427] "Frequently occurring words" refer to words or phrases that appear frequently in text data.

[1428] "Emotional state" refers to data that indicates the user's emotions, such as positive or negative, excited or relieved.

[1429] A "database" is an electronic storage facility for systematically saving and managing related materials and information.

[1430] "Related materials" refer to information and documents that are related to the frequently extracted words.

[1431] "Synchronization rate" is an indicator that shows the degree of agreement or relevance between extracted frequently occurring words and related materials.

[1432] An "emotion engine" is a technology or software that analyzes a user's emotions in real time and generates data based on those emotions.

[1433] "Means of prioritizing display" refers to methods and techniques for sorting and displaying related information or content according to specific criteria.

[1434] "Conversation" is the process by which multiple people communicate with each other through sound.

[1435] This invention relates to a system primarily applied in autonomous vehicles that converts passenger and user conversations into text data in real time, extracts frequently occurring words, recognizes emotional states, and provides relevant information based on the results. The system is implemented based on the following procedure and configuration.

[1436] System Configuration

[1437] 1. Acquisition of audio data and conversion to text:

[1438] The server uses microphones inside the autonomous vehicle to acquire audio data of conversations between passengers and users. The acquired audio data is then converted into text data using the Google Speech-to-Text API.

[1439] 2. Extraction of frequently occurring words:

[1440] The server extracts frequently occurring words from text data using the NLTK library (Python), a natural language processing (NLP) tool.

[1441] 3. Recognition of emotional states:

[1442] The server uses Microsoft Azure's Emotion API to recognize the emotional state of passengers and users in real time and generate emotional data.

[1443] 4. Searching for and providing related information:

[1444] Based on the extracted frequent words and sentiment data, the server searches the Elasticsearch database to find relevant content (e.g., music, news, tourist information, advertisements, etc.), prioritizes it, and displays it.

[1445] Program Processing Overview

[1446] 1. Processing of audio data:

[1447] The audio data is sent from the microphone to the server and converted into text data using the Google Speech-to-Text API. The following is an example of a specific prompt.

[1448] Example prompt:

[1449] Based on the audio data, transcribe the content into text, extract frequently occurring words, obtain sentiment data using a sentiment engine, search for related content using Elasticsearch, and display it according to priority.

[1450] 2. Processing text data:

[1451] The NLTK library is used to extract frequently occurring words from the converted text data. The extracted keywords are important words that reflect the conversation content and are used in the search query.

[1452] 3. Processing of emotional data:

[1453] Text data is sent to the emotion engine, and emotion data is generated using Azure's Emotion API. This emotion data will play a crucial role in the subsequent information provision process.

[1454] 4. Searching for and displaying related content:

[1455] Based on the extracted frequent words and sentiment data, the server searches the Elasticsearch database to find relevant content. The content is sorted according to priority based on sentiment state and displayed on the smart display. For example, if it is recognized that a passenger is "talking about restaurants nearby," restaurant information will be displayed preferentially.

[1456] Specific example

[1457] Let's say a passenger is talking on the train and says, "I want to know about tourist attractions in this area." The server captures this conversation audio and uses the Google Speech-to-Text API to convert "tourist attractions in this area" into text. Using the NLTK library, "tourist attractions" is extracted as a key keyword, and the passenger's level of interest is further analyzed by an emotion engine. The Elasticsearch database is searched, and information based on the keyword "tourist attractions" is displayed preferentially. For example, "popular tourist attractions in this area" or "recommended tourist destinations list" might be displayed on the smart display.

[1458] Thus, this invention realizes a system that can improve passenger convenience and satisfaction in autonomous vehicles by providing information in real time that corresponds to the content of passengers' or users' conversations and emotional states.

[1459] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1460] Step 1:

[1461] The server acquires audio data using microphones inside the autonomous vehicle. The acquired audio data is saved as is. Specifically, it is recorded in an audio file format (e.g., a .wav file).

[1462] Input: Audio of conversations between passengers and users inside the vehicle.

[1463] Output: Audio data file (.wav)

[1464] Step 2:

[1465] The server sends the audio data to the Google Speech-to-Text API, where it is converted into text data. The converted text data is then stored for subsequent processing and formatted into a concise form.

[1466] Input: Audio data file (.wav)

[1467] Output: Text data

[1468] Step 3:

[1469] The server uses the NLTK library to extract frequently occurring words from text data. It tokenizes (segments) the text data, calculates the frequency of each word, and picks out the most frequent words.

[1470] Input: Text data

[1471] Output: List of frequently occurring words

[1472] Step 4:

[1473] The server sends the extracted text data to the emotion engine, which uses Microsoft Azure's Emotion API to generate emotion data. This emotion data provides a detailed description of the user's emotional state, such as positive, negative, or excited.

[1474] Input: Text data

[1475] Output: Sentiment data

[1476] Step 5:

[1477] The server searches the Elasticsearch database based on the extracted list of frequent words and sentiment data. The extracted frequent words are used in the search queries, and the search results are prioritized based on the sentiment data.

[1478] Input: List of frequently occurring words, sentiment data

[1479] Output: Related Content List

[1480] Step 6:

[1481] The server selects relevant content from the search results in order of priority and displays it on the smart display. For example, if a passenger is talking about "restaurants," information about nearby restaurants will be displayed.

[1482] Input: Related Content List

[1483] Output: Content displayed on the smart display

[1484] Step 7:

[1485] The system uses a terminal to identify terms that passengers may not understand and generates and displays explanations of those terms along with links to related resources. For example, if the term "digital transformation" is included, an explanation and a link to more detailed resources will be displayed.

[1486] Input: List of frequently occurring words

[1487] Output: Glossary of terms, links to related resources

[1488] The above outlines the specific processing steps for implementing this invention. This enables the real-time provision of information tailored to the conversation content and emotional state of passengers and users.

[1489] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1490] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1491] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[1492] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1493] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[1494] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[1495] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[1496] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[1497] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[1498] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[1499] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[1500] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[1501] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[1502] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1503] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[1504] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[1505] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[1506] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[1507] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[1508] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[1509] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted as being incorporated by reference.

[1510] The following is further disclosed regarding the embodiments described above.

[1511] (Claim 1)

[1512] A means of converting audio data from meetings and other events into text data,

[1513] A method for extracting frequently occurring words from converted text data,

[1514] A method to search for related documents from the company's database based on the extracted frequently occurring words and sort them in order of synchronicity,

[1515] A means of generating links to the arranged materials,

[1516] A system that includes this.

[1517] (Claim 2)

[1518] Based on the extracted frequently occurring words and related materials, a method for organizing materials within the company database by category,

[1519] A means of presenting the latest hot topics related to frequently used words,

[1520] The system according to claim 1, further comprising:

[1521] (Claim 3)

[1522] A means to identify terms that the user may not understand from among the extracted frequently occurring words,

[1523] A means of generating explanations for identified terms and providing links to related materials,

[1524] The system according to claim 1, further comprising:

[1525] "Example 1"

[1526] (Claim 1)

[1527] A means of converting audio data into text data,

[1528] A method for extracting frequently occurring words from converted text data,

[1529] A method for searching for related information from a database based on the extracted frequently occurring words and sorting them in order of synchronicity,

[1530] A means of generating links to the arranged information,

[1531] A means of using an external API to recognize audio data,

[1532] Methods for extracting frequently occurring words using natural language processing tools,

[1533] A system that includes this.

[1534] (Claim 2)

[1535] A means of organizing the information in the database into categories based on the extracted frequently occurring words and related information,

[1536] A means of presenting the latest topics related to frequently used words,

[1537] The system according to claim 1, further comprising:

[1538] (Claim 3)

[1539] A means to identify terms that the user may not understand from among the extracted frequently occurring words,

[1540] A means for generating explanations for identified terms and presenting links to related information,

[1541] The system according to claim 1, further comprising:

[1542] "Application Example 1"

[1543] (Claim 1)

[1544] A means of converting audio data from meetings and other events into text data,

[1545] A method for extracting frequently occurring words from converted text data,

[1546] A method for searching related materials from a database based on frequently extracted words and sorting them in order of synchronicity,

[1547] A means of generating links to the arranged materials,

[1548] A method for transcribing audio data of customer conversations in real time, and for searching and providing related product information based on frequently occurring words,

[1549] A system that includes this.

[1550] (Claim 2)

[1551] A method for organizing the materials in the database by category based on the extracted frequently occurring words and related materials,

[1552] A means of presenting the latest hot topics related to frequently used words,

[1553] The system according to claim 1, further comprising:

[1554] (Claim 3)

[1555] A means to identify terms that the user may not understand from among the extracted frequently occurring words,

[1556] A means of generating explanations for identified terms and providing links to related materials,

[1557] The system according to claim 1, further comprising:

[1558] "Example 2 of combining an emotion engine"

[1559] (Claim 1)

[1560] A means of converting audio data from meetings and other events into text data,

[1561] A method for extracting frequently occurring words from converted text data,

[1562] A method for searching for relevant materials from the company's internal database based on extracted frequently occurring words and user sentiment data, and sorting them by synchronization rate and priority based on sentiment data,

[1563] A means of generating links to the arranged materials,

[1564] A system that includes this.

[1565] (Claim 2)

[1566] Based on the extracted frequently occurring words and related materials, a method for organizing materials within the company database by category,

[1567] A means of presenting the latest hot topics related to frequently occurring words and sentiment data,

[1568] The system according to claim 1, further comprising:

[1569] (Claim 3)

[1570] A means to identify terms that the user may not understand from among the extracted frequently occurring words,

[1571] A means of generating explanations for identified terms and providing links to related materials,

[1572] The system according to claim 1, further comprising:

[1573] "Application example 2 when combining with an emotional engine"

[1574] (Claim 1)

[1575] A means of converting audio data from meetings and other events into text data,

[1576] A method for extracting frequently occurring words from converted text data,

[1577] A method for searching related materials from a database based on frequently extracted words and sorting them in order of synchronicity,

[1578] A means of generating links to the arranged materials,

[1579] A method for converting passenger and user conversations into text data, extracting frequently occurring words from that text data, and recognizing the passengers' emotional state in real time,

[1580] A method for searching for and prioritizing relevant content based on extracted frequently occurring words and sentiment data,

[1581] A system that includes this.

[1582] (Claim 2)

[1583] A method for organizing the materials in the database by category based on the extracted frequently occurring words and related materials,

[1584] A means of presenting the latest hot topics related to frequently used words,

[1585] The system according to claim 1.

[1586] (Claim 3)

[1587] A method for identifying terms that users may not understand from among the frequently extracted words,

[1588] A means of generating explanations for identified terms and providing links to related materials,

[1589] The system according to claim 1. [Explanation of Symbols]

[1590] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of converting audio data from meetings and other events into text data, A method for extracting frequently occurring words from converted text data, A method to search for related documents from the company's database based on the extracted frequently occurring words and sort them in order of synchronicity, A means of generating links to the arranged materials, A system that includes this.

2. Based on the extracted frequently occurring words and related materials, a method for organizing materials within the company database by category, A means of presenting the latest hot topics related to frequently used words, The system according to claim 1, further comprising:

3. A means to identify terms that the user may not understand from among the extracted frequently occurring words, A means of generating explanations for identified terms and providing links to related materials, The system according to claim 1, further comprising:

Images