System

Abstract

A system is provided.SOLUTION: This system is provided with a means for integrating and managing an officer attendance history, a conference record, company information, staff information, an organization chart, patent information, brand information, a service history and event information, a means including a retrieval interface for retrieving the information, a means for summarizing and displaying retrieval results and a means for displaying the detailed information of the pertinent retrieval results.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In modern companies, the scattering of information and the complexity of searching and managing it are major problems. In particular, checking old information, specific events, executive histories, employee entry and exit records, patent and trademark registration dates, etc., requires a lot of time and effort. This reduces business efficiency and makes it difficult to make quick decisions. The purpose of this invention is to solve these problems by providing a system that centrally manages this information and allows quick access to information regardless of changes in personnel or the passage of time. [Means for solving the problem]

[0005] The present invention provides a system that includes a means for integrating and managing executive appointment histories, meeting records, company information, employee information, organizational charts, patent information, trademark information, service history, and event information, a search interface for searching the information, a means for summarizing and displaying search results, and a means for displaying detailed information about relevant search results. The system also includes a means for converting audio data from meeting records into text, and the search interface has a means for enabling searches in natural language. This allows for efficient management, search, and display of various information, realizing an environment where information can be accessed quickly and accurately regardless of changes in personnel or the passage of time.

[0006] An "official history" is a record that includes the dates and periods in which a company's officers held specific positions, as well as details of those positions.

[0007] "Meeting records" refer to documents and audio data that include the content discussed, decisions made, participants, dates and times, etc., at meetings such as board meetings held within a company.

[0008] "Company information" refers to basic information about a company, including, for example, the company's founding date, location, corporate number, capital, and registration information.

[0009] "Employee information" refers to detailed information about each employee of a company, such as their name, start date, departure date, last department, and immediate supervisor.

[0010] An "organizational chart" is a diagram that shows the organizational structure within a company, visually illustrating the positions, departmental arrangements, and hierarchical relationships.

[0011] "Patent information" refers to information about patents acquired by a company, including the patent acquisition date, patent number, and invention details.

[0012] "Trademark information" refers to information about trademarks registered by a company, including the trademark registration date, registration number, trademark design and name, etc.

[0013] "Service history" refers to records that include the start and end dates of various services provided by a company, service details, and change history.

[0014] "Event information" is detailed information about an event held within a company, including the event date, summary, participants, and organizing department.

[0015] A "search interface" is a screen or form that a user uses to search for specific information, and is a means of retrieving information from a database by entering search keywords.

[0016] "Means for converting voice data into text" refers to technology that analyzes recorded voice information and converts it into text information.

[0017] "Natural language search" is a method in which a user inputs natural sentences or keywords, and the system understands their intent and searches for relevant information.

[0018] "Summarization" refers to the ability to display search results in a concise, easy-to-understand format.

[0019] The "means for displaying detailed information" refers to a function that provides the user with detailed information about a specific item selected from the search results. [Brief explanation of the drawings]

[0020] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4]FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0021] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0022] First, the terms used in the following description will be explained.

[0023] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0024] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0025] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0026] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0027] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0028] [First embodiment]

[0029] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0030] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0031] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0032] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0033] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0034] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0035] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0036] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0037] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0038] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0039] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0040] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0041] This invention provides a system that centrally manages in-house information and enables quick information retrieval regardless of changes in personnel or the passage of time. This system is composed of a server, terminals, and users, each playing their own role.

[0042] 1. Information intake

[0043] First, various information (executive appointment history, meeting records, company information, employee information, organizational chart, patent information, trademark information, service history, event information) is collected on the server. The server reads this information from CSV files, Excel files, etc. and stores it in a centrally managed database. Any missing data is also processed to ensure data integrity.

[0044] Example: Data on the history of executive appointments is read from a CSV file, and after necessary preprocessing, it is stored in the database. This creates a centrally managed database.

[0045] 2. Information Search

[0046] Users access the search interface using a terminal (PC or mobile device). The search interface supports searches in natural language, allowing users to intuitively enter search keywords. The entered search keywords are sent from the terminal to the server.

[0047] Example: When a user types "Yamada Taro executive history" into a search box, the search query is submitted and the server searches the database for relevant information.

[0048] 3. Summary display of search results

[0049] The server searches the database based on the search query and extracts relevant results. The extracted results are processed in summary form and sent to the terminal, where the user can view the summary of the search results on the terminal screen.

[0050] Example: The server searches the executive history database and extracts information related to "Yamada Taro." As a result, the name, title, and date of appointment of the relevant executive are displayed in summary form.

[0051] 4. Viewing detailed information

[0052] When a user selects a specific item from the summarized search results, the terminal sends a request to the server to obtain detailed information about that item. The server then retrieves the detailed information about the selected item from the database and sends it to the terminal.

[0053] Example: When a user clicks on the details of "Yamada Taro," detailed career information for that executive (such as date of appointment, position, department, etc.) is displayed.

[0054] 5. Converting Audio Data to Text

[0055] If there is audio data from the conference recording, the server converts the audio data into text, which is then stored in the database for search and display.

[0056] Example: Converting recordings of board meetings into text and storing the content in a database so that it can be searched in natural language.

[0057] 6. Natural Language Search

[0058] The search interface provides users with the ability to enter queries in natural language, allowing for analysis of specific keywords as well as detailed sentences to identify relevant information.

[0059] Example: If a user enters "meeting notes about new service announcements in 2019," the server will parse the query and search for new services announced in 2019 and related meeting notes.

[0060] This system allows for the fast and efficient management and search of various internal company information. Even if the person in charge changes, the centralized management of information ensures business continuity, and past information and future action items can be easily accessed. This significantly improves business efficiency and enables quick decision-making.

[0061] The processing flow will be explained below.

[0062] Step 1:

[0063] Information capture

[0064] The server loads information such as executive appointments and meeting records from external data sources such as CSV and Excel files, including the automatic loading of regularly updated data.

[0065] Example: The server reads "Executive Appointment History.csv" and converts information such as executive names, appointment dates, and positions into a data frame (e.g., pandas).

[0066] Step 2:

[0067] Data Preprocessing

[0068] The server preprocesses the data it loads, which includes imputing missing values, converting data types, and removing unnecessary data.

[0069] For example, the server will impute missing values ​​with 'N / A' and convert date fields to the appropriate data type (e.g. datetime).

[0070] Step 3:

[0071] Storage in the database

[0072] The server stores the preprocessed data in a centralized database, thereby realizing integrated management of information.

[0073] Example: The server inserts executive history data into an SQL database and creates the appropriate indexes.

[0074] Step 4:

[0075] Entering a query through the search interface

[0076] The user accesses the search interface on the terminal and inputs the information they want to search for in natural language.

[0077] Example: User enters "2021 Board Meeting Minutes."

[0078] Step 5:

[0079] Submitting a search query

[0080] The terminal transmits the input search query to the server, which then receives the query and begins analyzing it.

[0081] Example: A device sends a query for "2021 board meeting minutes" as an HTTP request to a server.

[0082] Step 6:

[0083] Database search

[0084] The server analyzes the incoming search query and searches a centralized database, sorting the results by relevance.

[0085] Example: The server uses an SQL query to extract data corresponding to "2021 Board of Directors Meeting Minutes."

[0086] Step 7:

[0087] Search result summary

[0088] The server generates data to summarize and display search results, allowing users to see concise information.

[0089] Example: The server extracts the most important fields from the search results (date, time, topic, participants, etc.) and presents them in a summarized format.

[0090] Step 8:

[0091] Displaying search results

[0092] The server sends the summarized search results to the terminal, which then displays them to the user, who can then check the list of search results.

[0093] Example: Display a list of summarized board meeting minutes in a device browser.

[0094] Step 9:

[0095] Request more information

[0096] When the user selects an item for which they wish to check details, the terminal sends a request to the server to obtain detailed information about that item.

[0097] Example: When a user clicks on a particular board meeting record, a request is sent to the server to get more information about it.

[0098] Step 10:

[0099] Get more information

[0100] The server retrieves detailed information about the selected item from the database and sends it to the terminal, allowing the user to view the detailed information.

[0101] Example: The server retrieves detailed information (e.g., text of speeches) of the selected board meeting record from the database and sends it to the terminal.

[0102] Step 11:

[0103] Viewing detailed information

[0104] The terminal displays the received detailed information to the user, allowing the user to check the details of the specific information.

[0105] Example: A detailed record of the selected board meeting is displayed on the terminal screen.

[0106] Step 12:

[0107] Converting audio data to text

[0108] If the meeting recording is provided as audio data, the server converts the audio data into text, which is then stored in a database and made searchable.

[0109] Example: The server uses speech recognition technology to convert the audio data of a meeting into text and stores the text in a database.

[0110] Step 13:

[0111] Providing natural language search

[0112] The server analyzes search queries entered in natural language and provides appropriate search results, enabling users to perform searches easily and intuitively.

[0113] Example: When a user enters a query for "all company events held in 2020," the server parses the natural language query and searches the database for relevant event information.

[0114] Example 1

[0115] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0116] Modern business activities require centralized management and search of a wide range of information, including executive history, meeting records, company information, employee information, organizational charts, intellectual property information, trademark information, service history information, and event information. However, when this information is stored in multiple different formats, it is difficult to automatically detect and correct missing information or format inconsistencies. Furthermore, there is a need for systems that can convert voice data into text and efficiently search using natural language, but current systems lack these capabilities.

[0117] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0118] In this invention, the server includes means for integrating and managing executive history information, meeting records, company information, employee information, organizational structure charts, intellectual property information, trademark information, service history information, and event information, a search interface for searching the information, means for summarizing and displaying search results, means for displaying detailed information of relevant search results, and means for automatically detecting and processing missing information and format inconsistencies in the information. This makes it possible to provide a system that efficiently manages a variety of information within a company, automatically ensures data consistency, and enables intuitive information searches using natural language.

[0119] "Officer history information" is detailed information about the positions held by company officers in the past and the periods of their service.

[0120] "Meeting records" are documents or data that record the content, minutes, participants, and decisions of meetings held within a company.

[0121] "Company information" is a set of data including basic information about a company, its management policies, performance, location, and contact information.

[0122] "Employee information" refers to data such as personal information, job title, role, salary, and career history for all employees belonging to a company.

[0123] An "organizational structure chart" is a diagram that visually shows the relationships between departments and positions within a company.

[0124] "Intellectual property information" refers to information relating to intellectual property rights owned by a company, such as patents, trademarks, copyrights, and trade secrets.

[0125] "Trademark Information" is detailed information about a company's trademarked brands, logos, slogans, etc.

[0126] "Service history information" is data that records history information and provision status regarding various services provided by a company.

[0127] "Event information" refers to information such as detailed information, dates, and participant lists of events hosted or participated in by a company.

[0128] The "search interface means" is a means for providing an interface that is used when a user searches for required information.

[0129] "Means for automatically detecting and handling missing data and format inconsistencies" refers to the system's functionality for automatically detecting missing data and format inconsistencies and handling them appropriately when they occur.

[0130] "Natural language search" is a function that understands search queries and provides appropriate data when users search for information using everyday language.

[0131] This invention provides a system that can centrally manage information within a company and search for it quickly and efficiently. The main components of the system include a server, terminals, and users.

[0132] Server Roles

[0133] The server operates a database for centrally managing various information within the company. The database management system used may be, for example, MySQL (registered trademark). The server reads executive history information, meeting records, company information, employee information, organizational structure charts, intellectual property information, trademark information, service history information, and event information from CSV files and Excel files, and stores this information in the database. It also has the ability to automatically detect missing information and format inconsistencies and handle them appropriately.

[0134] In addition, the server uses speech recognition software (such as Google® Cloud Speech-to-Text) to convert the audio data of the meeting recording into text, and also stores the converted text data in a database.

[0135] Device Role

[0136] A terminal is a device that users use to operate the system, and can include PCs, mobile devices, etc. A search interface is provided on the terminal, allowing users to intuitively enter search keywords using natural language. These search keywords are sent from the terminal to the server.

[0137] User operations

[0138] A user uses a terminal to search for the information they need. For example, they type "Yamada Taro executive appointment history" into the search box. When this search query is sent to the server, the server searches the database for relevant information and sends the results in summary form to the terminal. The user can check the summary of the search results and click to view more detailed information.

[0139] Specific examples

[0140] For example, to search for the executive history of director Yamada Taro, a user would enter "Yamada Taro executive appointment history" into the search box on their device. The server receives this query, searches the database for relevant information, and sends Yamada Taro's position, appointment date, and other information in a summary format to the device. If the user wants to check more detailed information from the summary results, they can send a request from their device again, and the server will provide the detailed information.

[0141] As a concrete example of converting audio data from meeting records into text, the recording data of a board meeting is converted into text using Google Cloud Speech-to-Text and the content is stored in a database. This text data can later be searched as "Meeting records regarding the announcement of new services in 2019."

[0142] Prompt Sentence Examples

[0143] For example, enter the following prompt into the system:

[0144] "Find the conference notes for the 2019 New Service Launch."

[0145] In this way, the system can efficiently manage a wide variety of information according to user needs and support natural language searches, greatly improving business efficiency.

[0146] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0147] Step 1: Capture information

[0148] Input: CSV and Excel files containing executive history information, meeting notes, company information, employee information, organizational charts, intellectual property information, trademark information, service history information, and event information.

[0149] How it works: The server reads these files and detects missing data and format inconsistencies. It automatically handles any missing data and formats that are mismatched.

[0150] Data processing / calculation: Standardize data formats and fill in missing data to ensure consistency.

[0151] Output: Store the organized data in a database.

[0152] Specific operation: Read executive history information from a CSV file, fill in missing values, and store it in a MySQL database.

[0153] Step 2: Enter search keywords and submit

[0154] Input: Keywords entered by the user into the device's search interface (e.g., "Yamada Taro executive appointment history").

[0155] How it works: A user enters keywords into the search box on their device and clicks the search button. This input is sent from the device to the server.

[0156] Data processing / calculation: None in particular (sending keywords).

[0157] Output: Sending search keywords from the terminal to the server.

[0158] Specific operation: The user enters "Yamada Taro executive appointment history" and clicks the search button.

[0159] Step 3: Processing the search query and extracting search results

[0160] Input: Search keywords sent from the device (e.g., "Yamada Taro executive appointment history").

[0161] Operation: The server searches the database based on the received search keywords and extracts relevant information.

[0162] Data processing / calculation: Generate database queries and extract relevant data.

[0163] Output: Extraction of relevant search results.

[0164] Specific operation: The server searches and extracts executive history information related to "Yamada Taro" from the database.

[0165] Step 4: Summary of search results

[0166] Input: Search results extracted by the server (e.g., executive history information for "Yamada Taro").

[0167] Operation: The server sends the extracted results in summary form to the terminal.

[0168] Data processing / computation: summarizing and formatting information.

[0169] Output: Sends summarized search results to the terminal.

[0170] Specific operation: The server summarizes the job title and start date for Yamada Taro and sends it to the terminal.

[0171] Step 5: Request and view more information

[0172] Input: The item the user selects from the summary results (e.g., detailed information about "Yamada Taro").

[0173] How it works: When a user clicks on a specific item, the device sends a request to the server to get more information. The server retrieves the details from the database and sends them to the device.

[0174] Data processing / calculation: Obtain detailed information from the database.

[0175] Output: Send detailed information to the terminal for display.

[0176] Specific operation: The user clicks on the detailed information link for "Yamada Taro," and the server retrieves and displays the department he holds and his specific job history.

[0177] Step 6: Convert audio data to text

[0178] Input: Audio data from a meeting recording.

[0179] How it works: The server converts the audio data into text using Google Cloud Speech-to-Text.

[0180] Data processing / calculation: Converts voice data into text data.

[0181] Output: The converted text data.

[0182] Specific operation: The recording data of the board meeting is converted into text using Google Cloud Speech-to-Text and saved in the database as "Minutes of the board meeting held on January 1, 2019."

[0183] Step 7: Perform a natural language search

[0184] Input: A natural language query entered by a user into a search interface (e.g., "Conference notes on new service announcements in 2019").

[0185] How it works: The server uses natural language processing tools (such as NLTK) to parse the search query and search a database for relevant information.

[0186] Data processing / calculation: Search query analysis and query generation.

[0187] Output: Send relevant search results to your device.

[0188] Specific operation: The server analyzes the "Meeting records regarding the announcement of new services in 2019" using a natural language processing tool, searches for the relevant meeting records, and sends them to the terminal.

[0189] (Application example 1)

[0190] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0191] In conventional information management systems, important information such as factory manufacturing information, machine maintenance history, and work instructions is stored in a distributed manner, making it difficult to quickly search and centrally manage information. Furthermore, when managing the contents of meetings and discussions as voice data, there was a lack of systems that could convert the voice data into text and make it easily searchable. Furthermore, there was an insufficient system that supported searches in natural language, making intuitive searches difficult.

[0192] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0193] In this invention, the server includes means for integrating and managing executive appointment histories, meeting records, company information, employee information, organizational charts, patent information, trademark information, service history, and event information, means including a search interface for searching the information, means for summarizing and displaying search results, means for displaying detailed information of relevant search results, means for converting audio data of meeting records and meetings into text, and means for indexing and storing the text data in a database. This enables unified management of information, rapid search, and intuitive search in natural language.

[0194] "Officer appointment history" is data that records and manages information such as the appointment date, position, and term of office of an officer.

[0195] A "meeting record" is a document that records the contents of a meeting, speakers, remarks, agenda, conclusions, etc., and can be referenced later.

[0196] "Company information" refers to data including basic company information, date of establishment, location, business activities, capital, and board of directors composition.

[0197] "Employee information" refers to data that records and manages personal and work-related information such as an employee's name, department, position, date of joining, and contact information.

[0198] An "organizational chart" is a visual representation of the structure of a company or organization, showing each department and their relationships.

[0199] "Patent information" is data that records detailed information about patents, such as patent number, patent name, inventor, application date, registration date, and patent owner.

[0200] "Trademark information" is data that records detailed information such as the trademark name, trademark registration number, registration date, and trademark owner.

[0201] "Service history" is data that records information such as the type of service provided by a company, the start date of service, the person in charge, the service content, and the customer.

[0202] "Event information" is data that records information such as the name, date, time, location, participants, and content of events that a company has hosted or participated in.

[0203] A "search interface" is a portion of software that provides input functionality that a user uses to quickly search for specific information.

[0204] A "means for converting voice data to text" is a technique or device for taking voice input and converting it into textual data.

[0205] A "means for indexing and storing data in a database" is a technique or method for organizing and storing data in a database in an easily searchable format.

[0206] "Natural language search" is a search function that analyzes search queries entered by users in natural language and identifies and presents relevant information.

[0207] The present invention provides a smart manufacturing information management system that is installed on robots used in factories. This system centrally manages important information within the factory and supports searches using natural language. A specific embodiment of the system will be described below.

[0208] 1. System configuration

[0209] The system is composed of the roles of a server, a terminal, and a user.

[0210] server

[0211] The server maintains a database that centrally manages information collected within the factory, including executive appointment histories, meeting records, company information, employee information, organizational charts, patent information, trademark information, service history, and event information. The server also reads this information from CSV files and Excel files and stores it in the database. The server processes the voice data using software to convert the voice data to text (for example, the speech_recognition library).

[0212] Terminal

[0213] Users access the search interface using a terminal such as a PC or mobile device. The terminal accepts search query input and sends it to the server. The terminal also has the function of displaying search results in summary form and in detailed information.

[0214] 2. Information intake

[0215] The server reads various information used within the factory from CSV and Excel files, including information on newly appointed executives and the latest meeting records. The read data is stored in a database, and any missing or inconsistent data is dealt with appropriately.

[0216] 3. Search function

[0217] Users perform natural language searches through their devices. For example, they can enter "contents of the quality control meeting in September 2023," and the query is sent to the server. The server analyzes the query and searches the database for relevant information. Search results are displayed on the device in summary form, and detailed information is displayed when the user selects a specific item.

[0218] 4. Converting Audio Data to Text

[0219] The server converts recordings of meetings and discussions into text. For example, if you upload a recording of a quality control meeting, it will be converted into text using the speech_recognition library and the content will be stored in a database.

[0220] 5. Specific Examples

[0221] For example, if a quality control meeting is held at a factory and the audio data is recorded, the recording can be uploaded to the smart manufacturing information management system and the meeting content converted into text, and then by searching for "quality control meeting content in September 2023," the relevant meeting records can be easily retrieved.

[0222] Prompt Sentence Examples

[0223] "Upload the audio file, convert the meeting record to text, and save it in the database. Then, search for 'Contents of the Quality Control Meeting in September 2023' and display it."

[0224] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0225] Step 1: Capture information

[0226] The server reads executive appointment histories, meeting records, company information, employee information, organizational charts, patent information, trademark information, service history, and event information collected within the factory from CSV and Excel files. The server stores the read data in a database and processes any missing or inconsistent data appropriately. The input is CSV and Excel files, and the output is a centrally managed database.

[0227] Step 2: Enter a search query

[0228] A user accesses the search interface using a terminal (PC or mobile device) and inputs a search query in natural language. For example, they might input "Contents of the quality control meeting in September 2023." The input is the user's search query, and the output is a search request to the server.

[0229] Step 3: Parsing the search query and searching the data

[0230] The server analyzes the search query received from the user and searches for relevant information from the database. For example, if the query "Contents of the quality control meeting in September 2023" is entered, the server analyzes the query using a natural language processing algorithm and extracts the relevant meeting records from the database. The input is the search query, and the output is the search result data.

[0231] Step 4: Summarizing and displaying search results

[0232] The server summarizes the extracted search results and sends them to the terminal. The search results are displayed in summary form on the user's terminal screen. For example, the title and main topics of a particular meeting record are displayed as a summary. The input is the search result data, and the output is the summarized search results.

[0233] Step 5: Obtain and display detailed information

[0234] The user selects a specific item from the summarized search results and requests detailed information. The server retrieves the detailed information for the selected item from the database and sends it to the terminal. For example, if the user clicks on the detailed information for a meeting record, all statements and decisions made during the meeting are displayed as details. The input is the user's request for detailed information, and the output is the detailed information.

[0235] Step 6: Convert audio data to text

[0236] The server receives the audio data of the meeting or conference and converts it into text. It uses the speech_recognition library to analyze the audio data and convert it into text data. The input is the audio data and the output is the converted text data.

[0237] Step 7: Save the text data

[0238] The server indexes the converted text data in a database and stores it. For example, the audio recording of a quality control meeting is converted into text and the content is stored in a database. The input is text data, and the output is an indexed database.

[0239] Step 8: Perform a natural language search

[0240] The user then uses the device to perform a natural language search to quickly obtain the desired information. For example, if the user searches for "quality control meeting contents in September 2023," relevant meeting records are displayed as search results. The input is the user's search query, and the output is the relevant search results.

[0241] These processing steps enable centralized management of important information within the factory, rapid search, and intuitive search in natural language.

[0242] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0243] This invention is a system that centrally manages in-house information and can quickly search for information regardless of changes in personnel or the passage of time. By combining this system with an emotion engine, it becomes possible to provide information flexibly according to the user's emotional state.

[0244] 1. Information intake

[0245] First, various information (executive appointment history, meeting records, company information, employee information, organizational chart, patent information, trademark information, service history, event information) is collected on the server. The server reads this information from CSV files, Excel files, etc. and stores it in a centrally managed database. Any missing data is also processed to ensure data integrity.

[0246] Example: The server reads "Executive Appointment History.csv" and converts information such as executive names, appointment dates, and positions into a data frame (e.g., pandas).

[0247] 2. Information Search

[0248] Users access the search interface using a terminal (PC or mobile device) and enter the information they want to search for in natural language. The search interface is linked to an emotion engine that analyzes emotions from the user's input and usage status. The emotion engine analyzes emotions from the user's input and adjusts the search results.

[0249] Example: A user types in "2021 board meeting minutes," submits that search query, and the server searches the database for relevant information.

[0250] 3. Emotion Engine Analysis

[0251] The emotion engine analyzes user input and usage patterns to determine their emotions. Based on this analysis, it changes the display format and order of search results. It provides quick information when the user is in a hurry, and displays more details when more information is needed.

[0252] For example, if the emotion engine recognizes "impatience" from user input, it will prioritize and display the most relevant information, allowing the user to quickly obtain information.

[0253] 4. Summarizing and displaying search results

[0254] The server generates data to summarize and display the search results. Based on the analysis results of the emotion engine, it determines the summary format and display order of the search results. The server then sends the summarized search results to the terminal, which then displays them to the user.

[0255] Example: The server searches the executive appointment history database and extracts information related to "Yamada Taro." As a result, the name, title, appointment date, etc. of the relevant executive are displayed in summary form. If the emotion engine detects "doubt," it provides more detailed information.

[0256] 5. Viewing detailed information

[0257] When a user selects a specific item from the summarized search results, the device sends a request to the server to obtain detailed information about that item. The server retrieves the detailed information about the selected item from the database and sends it to the device. The emotion engine also monitors the user's emotions as they check the detailed information and provides additional information as needed.

[0258] Example: When a user clicks on the details of "Yamada Taro," detailed career information for that executive (such as date of appointment, position, department, etc.) is displayed.

[0259] 6. Audio to Text Conversion

[0260] If the conference record is provided as audio data, the server converts the audio data into text, which is then stored in the database for search and display.

[0261] Example: Converting recordings of board meetings into text and storing the content in a database so that it can be searched in natural language.

[0262] 7. Natural Language Search

[0263] The search interface allows users to input natural sentences and keywords, allowing the system to understand their intent and search for relevant information. This allows the system to analyze not only specific keywords but also detailed sentences to identify relevant information.

[0264] Example: When a user enters a query for "all company events held in 2020," the server parses the natural language query and searches the database for relevant event information.

[0265] This system can flexibly provide information according to the user's emotional state. This allows users to quickly obtain the information they need, improving work efficiency. In addition, by combining it with an emotion engine, the user experience is improved and the stress of information search is reduced.

[0266] The processing flow will be explained below.

[0267] Step 1:

[0268] Information capture

[0269] The server loads information such as executive appointments and meeting records from external data sources such as CSV and Excel files, including the automatic loading of regularly updated data.

[0270] Example: The server reads "Executive Appointment History.csv" and converts information such as executive names, appointment dates, and positions into a data frame (e.g., pandas).

[0271] Step 2:

[0272] Data Preprocessing

[0273] The server preprocesses the data it loads, which includes imputing missing values, converting data types, and removing unnecessary data.

[0274] For example, the server will impute missing values ​​with 'N / A' and convert date fields to the appropriate data type (e.g. datetime).

[0275] Step 3:

[0276] Storage in the database

[0277] The server stores the preprocessed data in a centralized database, thereby realizing integrated management of information.

[0278] Example: The server inserts executive history data into an SQL database and creates the appropriate indexes.

[0279] Step 4:

[0280] Entering a query through the search interface

[0281] The user accesses the search interface on the terminal and inputs the information they want to search for in natural language.

[0282] Example: User enters "2021 Board Meeting Minutes."

[0283] Step 5:

[0284] Submitting a search query

[0285] The terminal transmits the input search query to the server, which then receives the query and begins analyzing it.

[0286] Example: A device sends a query for "2021 board meeting minutes" as an HTTP request to a server.

[0287] Step 6:

[0288] Database search

[0289] The server analyzes the incoming search query and searches a centralized database, sorting the results by relevance.

[0290] Example: The server uses an SQL query to extract data corresponding to "2021 Board of Directors Meeting Minutes."

[0291] Step 7:

[0292] Emotion Engine Analysis

[0293] The server uses an emotion engine to analyze emotions from user input and usage, and adjusts search results based on the user's emotional state.

[0294] Example: If the server detects the emotion "anxiety," it will prioritize displaying the most relevant information.

[0295] Step 8:

[0296] Search result summary

[0297] The server generates data to summarize and display search results, allowing users to see concise information.

[0298] Example: The server extracts the most important fields from the search results (date, time, topic, participants, etc.) and presents them in a summarized format.

[0299] Step 9:

[0300] Displaying search results

[0301] The server sends the summarized search results to the terminal, which then displays them to the user, who can then check the list of search results.

[0302] Example: Display a list of summarized board meeting minutes in a device browser.

[0303] Step 10:

[0304] Request more information

[0305] When the user selects an item for which they wish to check details, the terminal sends a request to the server to obtain detailed information about that item.

[0306] Example: When a user clicks on a particular board meeting record, a request is sent to the server to get more information about it.

[0307] Step 11:

[0308] Get more information

[0309] The server retrieves detailed information about the selected item from the database and sends it to the terminal, allowing the user to view the detailed information.

[0310] Example: The server retrieves detailed information (e.g., text of speeches) of the selected board meeting record from the database and sends it to the terminal.

[0311] Step 12:

[0312] Viewing detailed information

[0313] The terminal displays the received detailed information to the user, allowing the user to check the details of the specific information.

[0314] Example: A detailed record of the selected board meeting is displayed on the terminal screen.

[0315] Step 13:

[0316] Converting audio data to text

[0317] If the conference recording is provided as audio data, the server converts the audio data into text, which is then stored in the database for search and display.

[0318] Example: The server uses speech recognition technology to convert the audio data of a meeting into text and stores the text in a database.

[0319] Step 14:

[0320] Providing natural language search

[0321] The server analyzes search queries entered in natural language and provides appropriate search results, enabling users to perform searches easily and intuitively.

[0322] Example: When a user enters a query for "all company events held in 2020," the server parses the natural language query and searches the database for relevant event information.

[0323] Step 15:

[0324] Emotional Engine Feedback

[0325] The server changes the display format and presentation order of search results depending on the user's emotional state. Furthermore, the emotion engine learns from user feedback and uses it to improve the accuracy of future search results.

[0326] For example, if the sentiment engine detects "doubt," it will prioritize more detailed search results.

[0327] Example 2

[0328] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0329] In conventional in-house information management systems, information is managed in a decentralized manner, making it difficult to quickly search for information when the person in charge changes or when time passes. Furthermore, the system is unable to provide information flexibly according to the user's emotional state, which can be stressful. Additionally, handling voice data is difficult, making it difficult to efficiently search and summarize voice data.

[0330] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for integrating and managing executive information, meeting information, company information, employee data, organizational structure, intellectual property information, trademark data, service history, and event records, a means including an interface for searching the information, a means for adjusting and displaying search results according to the user's emotional state, a means for generating and displaying summaries of the corresponding search results, and a means for displaying detailed information. This allows information to be centrally managed, enabling rapid information retrieval regardless of changes in personnel or the passage of time. Furthermore, appropriate information can be provided according to the user's emotional state, reducing user stress. Furthermore, by converting and managing audio data of meeting information into text, audio data can be efficiently searched and summarized.

[0331] "Officer information" is data relating to the appointment history and positions of officers in a company or organization.

[0332] "Meeting information" refers to data including minutes, participant lists, decisions, etc., of meetings held within a company or organization.

[0333] "Corporate information" refers to data that includes basic information such as a company overview, performance, business activities, and organizational structure.

[0334] "Employee Data" means data that includes personal information, job title, salary, work history, etc., of employees within a company or organization.

[0335] "Organizational structure" is information that indicates the arrangement of departments and divisions in a company or organization, the hierarchical structure of positions, and the like.

[0336] "Intellectual property information" refers to data relating to intellectual property such as patents, trademarks, and copyrights held by a company or organization.

[0337] "Trademark Data" means data relating to trademarks registered by a company or organization and their details.

[0338] "Service history" refers to data such as usage history of services and products provided by a company or organization, and customer feedback.

[0339] "Event records" are detailed information about events, seminars, exhibitions, etc. held by a company or organization.

[0340] "Interface" refers to the input screen or operation panel on a PC or mobile device that users use to search for information.

[0341] "Means for adjusting and displaying search results according to the user's emotional state" refers to a function that analyzes the user's emotions and adjusts the display format and order of search results based on the results.

[0342] "Means for generating and displaying summaries" refers to the function of concisely summarizing search results and displaying them in a format that is easy for the user to view.

[0343] "Means for displaying detailed information" refers to a function that displays detailed information when a user selects a specific search result.

[0344] "Means for converting voice data into text" refers to the function of recognizing voice data such as meeting records, converting it into text format, and saving it.

[0345] This invention is a system for centrally managing in-house information and for quickly and flexibly searching and displaying the information. This system is implemented as follows.

[0346] The server imports various information, such as executive information, meeting information, company information, employee data, organizational structure, intellectual property information, trademark data, service history, and event records, in the form of CSV files or Excel files, converts it into a data frame using the pandas library, and stores it in a centrally managed database. During this process, it detects and automatically supplements or corrects missing data and format inconsistencies. It also uses Google Speech-to-Text or similar speech recognition software to convert the audio data of the meeting information into text and store it in the database.

[0347] A user accesses a search interface using a PC or mobile device. The user enters a search query in natural language, which is then sent from the device to the server. For example, a query such as "2021 board meeting minutes" might be considered. The server receives the query and analyzes the user's emotional state using an emotion engine, such as the transformers library.

[0348] The emotion engine analyzes the user's emotions and adjusts the display format and order of search results based on the results. For example, if the user is feeling impatient, the most relevant information will be displayed first, providing information quickly. The emotion analysis results are also reflected in the generation of summaries of search results, which are sent to the device in an appropriate format depending on the user's emotional state.

[0349] The server searches the database, extracts the relevant information, formats it, and generates a summary. The generated summary is sent to the device in a format that is easy for the user to view, based on the analysis results of the emotion engine. When the user selects a specific item from the search results displayed in summary form, the device sends a request for detailed information to the server. The server retrieves the detailed information, sends it to the device in an appropriate format, and displays it to the user.

[0350] For example, if a user wants to find detailed information about an executive named "Yamada Taro," the user enters "details about Yamada Taro" into the search interface. This query is sent to the server, which retrieves the relevant executive information from the database and displays it as detailed information.

[0351] This system centralizes information management, making it possible to search for information quickly and effectively without being affected by changes in personnel or the passage of time.In addition, the introduction of an emotion engine enables flexible and prompt provision of information according to the user's emotions, reducing user stress.

[0352] Examples of prompts include "2021 board meeting minutes," "More information about Taro Yamada," and "All company events held in 2020." By entering these prompts, the system can quickly search for the relevant information and provide it to the user.

[0353] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0354] Step 1: Capture information

[0355] The server imports various information such as executive information, meeting information, company information, employee data, organizational structure, intellectual property information, trademark data, service history, and event records from CSV and Excel files. Specifically, it uses the pandas library to convert these files into data frames and stores them in a centrally managed database. The input is a CSV or Excel file, and the output is the data stored in the database.

[0356] Step 2: Data integrity check and completion

[0357] The server detects missing or inconsistent values ​​in the imported data and automatically completes or corrects them, for example by filling blank cells with a default value such as "unknown." The input is a data frame, and the output is the consistent data.

[0358] Step 3: Convert audio data to text

[0359] The server receives the audio data of the conference information and converts it into text using speech recognition software (e.g., Google Speech-to-Text). The converted text data is also stored in a database. The input is audio data, and the output is the converted text data.

[0360] Step 4: Finding information

[0361] A user uses a terminal to access a search interface and inputs the information they want to search for in natural language. For example, they input a query such as "2021 board meeting minutes." The terminal then sends this query to the server. The input is the user's search query, and the output is the transmission of the search query to the server.

[0362] Step 5: Analysis by Emotion Engine

[0363] The server analyzes the received search query and detects the user's emotions using an emotion engine. The emotion engine uses libraries such as the transformers library. Based on the analysis results, the server adjusts the display format and order of search results. The input is the search query, and the output is the emotion analysis results.

[0364] Step 6: Generating and formatting search results

[0365] The server retrieves search results from the database and generates a summary based on the analysis results of the emotion engine. For example, it summarizes information related to "Yamada Taro" and sends it to the terminal in the form of job title, date of appointment, etc. The input is the database search results and emotion analysis results, and the output is the summarized search results.

[0366] Step 7: Viewing search results

[0367] The terminal displays the search results in summary form received from the server to the user. The user checks the displayed information and selects detailed information as necessary. The input is the summarized search results, and the output is the display to the user.

[0368] Step 8: Request and view more information

[0369] When a user selects a specific item from the search results displayed in summary form, the terminal requests detailed information from the server. The server retrieves the detailed information from the database and sends it to the terminal. The terminal displays the detailed information to the user. The input is a request for detailed information, and the output is the display of the detailed information.

[0370] (Application example 2)

[0371] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0372] Conventional factory robot management systems have had problems such as difficulty in centrally managing robot operation history and maintenance records, and difficulty for operators in quickly obtaining the information they need. Furthermore, information search does not provide flexible information according to the user's emotional state, which means operators cannot obtain appropriate information in emergencies or during normal times, resulting in reduced work efficiency.

[0373] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for integrating and managing executive appointment histories, meeting records, company information, employee information, organizational charts, patent information, trademark information, service history, and event information; means including a search interface for searching for the information; means for summarizing and displaying search results; means for displaying detailed information of relevant search results; means for adjusting the search results according to the user's emotional state using an emotion engine; means for integrating and managing the operation history and maintenance records of factory robots in a database; and means for searching for related information based on a query in the user's natural language. This enables the user to flexibly and quickly obtain necessary information according to their emotional state, thereby achieving efficient operation of factory robots.

[0374] "Officer appointment history" refers to historical information on the appointment dates and positions of company officers.

[0375] "Meeting records" refers to recorded information such as minutes, audio data, and attendee information related to meetings within a company.

[0376] "Company information" refers to basic information about a company, such as the date of establishment, the name of the representative, and the main business activities.

[0377] "Employee information" refers to individual information about employees within a company, such as name, department, position, and start date.

[0378] An "organizational chart" is a diagram that shows the hierarchical internal structure of a company, visually representing the relationships between departments and positions.

[0379] "Patent information" refers to information about patents held by a company, including, for example, patent number, invention name, application date, registration date, etc.

[0380] "Trademark information" refers to information about trademarks held by a company, including, for example, trademark registration number, trademark name, registration date, etc.

[0381] "Service history" refers to the performance and history information of various services provided by a company, such as the date of service execution, customer information, and service content.

[0382] "Event information" refers to information about an event held by a company, such as the event date, participant list, and event content.

[0383] "Search interface" refers to the UI (user interface) that a user uses to search for specific information.

[0384] An "emotion engine" is a system that analyzes a user's emotional state (e.g., urgency, impatience, doubt, etc.) and adjusts the information provided based on that.

[0385] "Operation history" refers to the operating status and history information of a factory robot, such as work content, operating time, and error records.

[0386] "Maintenance records" refers to historical information about maintenance performed on a factory robot, such as inspection dates, repair details, and replacement parts.

[0387] A "natural language query" refers to a search request written in everyday language or sentence format, and refers to a technology that analyzes the entire sentence, not just specific keywords, to search for related information.

[0388] This invention builds a system that centrally manages the operation history and maintenance records of factory robots and is equipped with an emotion engine to provide flexible information according to the user's emotional state.

[0389] The server creates a database to integrate and manage executive appointment histories, meeting records, company information, employee information, organizational charts, patent information, trademark information, service history, and event information. This information is read from CSV files and Excel files, converted into a data frame (e.g., pandas), and stored. The server maintains data integrity and takes appropriate action if there is missing data.

[0390] Users access the search interface using a terminal (PC or mobile device) and enter search queries in natural language. To enable natural language searches, the server employs Natural Language Processing (NLP) technology. The interface analyzes the text entered by the user and performs semantic searches.

[0391] The search interface is connected to an emotion engine, which analyzes emotions from user input and usage and adjusts the display format and presentation order of search results. For example, if a user enters the query "Please tell me the latest maintenance records. I'm in a hurry," the emotion engine will determine the user's emotional state as "urgent" and prioritize displaying the most relevant and important information.

[0392] The server integrates and manages the operating history and maintenance records of factory robots in a database, and generates data for summarizing and displaying search results. Based on the analysis results of the emotion engine, it determines the summary format and display order of the search results. When the search results are displayed and the user requests more information, the terminal sends the request to the server, which then retrieves the information from the database and sends it to the terminal.

[0393] Additionally, audio data such as meeting recordings can be converted to text and stored in a database, enabling comprehensive searches with natural language queries. This process is performed using speech recognition software.

[0394] The hardware and software used specifically include the Python programming language, the pandas library, the TextBlob library, and scikit-learn, etc. Using this system, users can quickly obtain the information they need based on their emotional state, enabling the efficient operation of factory robots.

[0395] A specific example query is the prompt "Please tell me the latest maintenance records. I'm in a hurry." In this example, the emotion engine analyzes the user's emotion of "I'm in a hurry" and quickly displays the most important maintenance records.

[0396] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0397] Step 1:

[0398] The server reads executive appointment history, meeting records, company information, employee information, organizational charts, patent information, trademark information, service history, and event information from various CSV and Excel files, converts them into data frames (such as pandas), and stores them in a database. This enables integrated data management. The input is various information files, and the output is an integrated database.

[0399] Step 2:

[0400] A user accesses the search interface using a terminal (PC or mobile device) and inputs a search query in natural language, for example, by entering a prompt sentence such as "Please tell me the latest maintenance records. I'm very urgent." The input is the user's query text, which is passed to the search interface.

[0401] Step 3:

[0402] The search interface analyzes the user's input text using natural language processing (NLP) techniques. In this process, the input query text is broken down into tokens and semantic analysis is performed. The input is the query text, and the output is the analysis results (related keywords and meanings).

[0403] Step 4:

[0404] The emotion engine analyzes the user's emotional state from the input text. This analysis is performed using the TextBlob library, etc. For example, from the text "I'm in a hurry," the user's emotional state is determined to be "I'm in a hurry." The input is the user's query text, and the output is the emotional state.

[0405] Step 5:

[0406] The server searches the database based on the analysis results and the emotional state, extracting relevant information. During this process, it summarizes the search results and prioritizes the information according to the emotional state. The input is the analysis results and the emotional state, and the output is the summarized search results.

[0407] Step 6:

[0408] The terminal displays the summarized search results sent from the server in the most appropriate format based on the user's emotional state. The input is the summarized search results, and the output is the information displayed on the user's terminal screen.

[0409] Step 7:

[0410] When a user selects a specific item from the search results displayed in summary form, the terminal sends a request to the server to obtain detailed information about that item. The input is the item selected by the user, and the output is a request to obtain detailed information.

[0411] Step 8:

[0412] The server retrieves detailed information about the selected item from the database and sends it back to the terminal. The input is a request to retrieve detailed information, and the output is the detailed information.

[0413] Step 9:

[0414] The terminal displays the detailed information sent from the server to the user. This display is also adjusted based on the emotional state. The input is the detailed information, and the output is the detailed information displayed on the user's terminal screen.

[0415] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0416] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0417] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0418] [Second embodiment]

[0419] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0420] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0421] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0422] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0423] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0424] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0425] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0426] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0427] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0428] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0429] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0430] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0431] This invention provides a system that centrally manages in-house information and enables quick information retrieval regardless of changes in personnel or the passage of time. This system is composed of a server, terminals, and users, each playing their own role.

[0432] 1. Information intake

[0433] First, various information (executive appointment history, meeting records, company information, employee information, organizational chart, patent information, trademark information, service history, event information) is collected on the server. The server reads this information from CSV files, Excel files, etc. and stores it in a centrally managed database. Any missing data is also processed to ensure data integrity.

[0434] Example: Data on the history of executive appointments is read from a CSV file, and after necessary preprocessing, it is stored in the database. This creates a centrally managed database.

[0435] 2. Information Search

[0436] Users access the search interface using a terminal (PC or mobile device). The search interface supports searches in natural language, allowing users to intuitively enter search keywords. The entered search keywords are sent from the terminal to the server.

[0437] Example: When a user types "Yamada Taro executive history" into a search box, the search query is submitted and the server searches the database for relevant information.

[0438] 3. Summary display of search results

[0439] The server searches the database based on the search query and extracts relevant results. The extracted results are processed in summary form and sent to the terminal, where the user can view the summary of the search results on the terminal screen.

[0440] Example: The server searches the executive history database and extracts information related to "Yamada Taro." As a result, the name, title, and date of appointment of the relevant executive are displayed in summary form.

[0441] 4. Viewing detailed information

[0442] When a user selects a specific item from the summarized search results, the terminal sends a request to the server to obtain detailed information about that item. The server then retrieves the detailed information about the selected item from the database and sends it to the terminal.

[0443] Example: When a user clicks on the details of "Yamada Taro," detailed career information for that executive (such as date of appointment, position, department, etc.) is displayed.

[0444] 5. Converting Audio Data to Text

[0445] If there is audio data from the conference recording, the server converts the audio data into text, which is then stored in the database for search and display.

[0446] Example: Converting recordings of board meetings into text and storing the content in a database so that it can be searched in natural language.

[0447] 6. Natural Language Search

[0448] The search interface provides users with the ability to enter queries in natural language, allowing for analysis of specific keywords as well as detailed sentences to identify relevant information.

[0449] Example: If a user enters "meeting notes about new service announcements in 2019," the server will parse the query and search for new services announced in 2019 and related meeting notes.

[0450] This system allows for the fast and efficient management and search of various internal company information. Even if the person in charge changes, the centralized management of information ensures business continuity, and past information and future action items can be easily accessed. This significantly improves business efficiency and enables quick decision-making.

[0451] The processing flow will be explained below.

[0452] Step 1:

[0453] Information capture

[0454] The server loads information such as executive appointments and meeting records from external data sources such as CSV and Excel files, including the automatic loading of regularly updated data.

[0455] Example: The server reads "Executive Appointment History.csv" and converts information such as executive names, appointment dates, and positions into a data frame (e.g., pandas).

[0456] Step 2:

[0457] Data Preprocessing

[0458] The server preprocesses the data it loads, which includes imputing missing values, converting data types, and removing unnecessary data.

[0459] For example, the server will impute missing values ​​with 'N / A' and convert date fields to the appropriate data type (e.g. datetime).

[0460] Step 3:

[0461] Storage in the database

[0462] The server stores the preprocessed data in a centralized database, thereby realizing integrated management of information.

[0463] Example: The server inserts executive history data into an SQL database and creates the appropriate indexes.

[0464] Step 4:

[0465] Entering a query through the search interface

[0466] The user accesses the search interface on the terminal and inputs the information they want to search for in natural language.

[0467] Example: User enters "2021 Board Meeting Minutes."

[0468] Step 5:

[0469] Submitting a search query

[0470] The terminal transmits the input search query to the server, which then receives the query and begins analyzing it.

[0471] Example: A device sends a query for "2021 board meeting minutes" as an HTTP request to a server.

[0472] Step 6:

[0473] Database search

[0474] The server analyzes the incoming search query and searches a centralized database, sorting the results by relevance.

[0475] Example: The server uses an SQL query to extract data corresponding to "2021 Board of Directors Meeting Minutes."

[0476] Step 7:

[0477] Search result summary

[0478] The server generates data to summarize and display search results, allowing users to see concise information.

[0479] Example: The server extracts the most important fields from the search results (date, time, topic, participants, etc.) and presents them in a summarized format.

[0480] Step 8:

[0481] Displaying search results

[0482] The server sends the summarized search results to the terminal, which then displays them to the user, who can then check the list of search results.

[0483] Example: Display a list of summarized board meeting minutes in a device browser.

[0484] Step 9:

[0485] Request more information

[0486] When the user selects an item for which they wish to check details, the terminal sends a request to the server to obtain detailed information about that item.

[0487] Example: When a user clicks on a particular board meeting record, a request is sent to the server to get more information about it.

[0488] Step 10:

[0489] Get more information

[0490] The server retrieves detailed information about the selected item from the database and sends it to the terminal, allowing the user to view the detailed information.

[0491] Example: The server retrieves detailed information (e.g., text of speeches) of the selected board meeting record from the database and sends it to the terminal.

[0492] Step 11:

[0493] Viewing detailed information

[0494] The terminal displays the received detailed information to the user, allowing the user to check the details of the specific information.

[0495] Example: A detailed record of the selected board meeting is displayed on the terminal screen.

[0496] Step 12:

[0497] Converting audio data to text

[0498] If the meeting recording is provided as audio data, the server converts the audio data into text, which is then stored in a database and made searchable.

[0499] Example: The server uses speech recognition technology to convert the audio data of a meeting into text and stores the text in a database.

[0500] Step 13:

[0501] Providing natural language search

[0502] The server analyzes search queries entered in natural language and provides appropriate search results, enabling users to perform searches easily and intuitively.

[0503] Example: When a user enters a query for "all company events held in 2020," the server parses the natural language query and searches the database for relevant event information.

[0504] Example 1

[0505] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0506] Modern business activities require centralized management and search of a wide range of information, including executive history, meeting records, company information, employee information, organizational charts, intellectual property information, trademark information, service history information, and event information. However, when this information is stored in multiple different formats, it is difficult to automatically detect and correct missing information or format inconsistencies. Furthermore, there is a need for systems that can convert voice data into text and efficiently search using natural language, but current systems lack these capabilities.

[0507] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0508] In this invention, the server includes means for integrating and managing executive history information, meeting records, company information, employee information, organizational structure charts, intellectual property information, trademark information, service history information, and event information, a search interface for searching the information, means for summarizing and displaying search results, means for displaying detailed information of relevant search results, and means for automatically detecting and processing missing information and format inconsistencies in the information. This makes it possible to provide a system that efficiently manages a variety of information within a company, automatically ensures data consistency, and enables intuitive information searches using natural language.

[0509] "Officer history information" is detailed information about the positions held by company officers in the past and the periods of their service.

[0510] "Meeting records" are documents or data that record the content, minutes, participants, and decisions of meetings held within a company.

[0511] "Company information" is a set of data including basic information about a company, its management policies, performance, location, and contact information.

[0512] "Employee information" refers to data such as personal information, job title, role, salary, and career history for all employees belonging to a company.

[0513] An "organizational structure chart" is a diagram that visually shows the relationships between departments and positions within a company.

[0514] "Intellectual property information" refers to information relating to intellectual property rights owned by a company, such as patents, trademarks, copyrights, and trade secrets.

[0515] "Trademark Information" is detailed information about a company's trademarked brands, logos, slogans, etc.

[0516] "Service history information" is data that records history information and provision status regarding various services provided by a company.

[0517] "Event information" refers to information such as detailed information, dates, and participant lists of events hosted or participated in by a company.

[0518] The "search interface means" is a means for providing an interface that is used when a user searches for required information.

[0519] "Means for automatically detecting and handling missing data and format inconsistencies" refers to the system's functionality for automatically detecting missing data and format inconsistencies and handling them appropriately when they occur.

[0520] "Natural language search" is a function that understands search queries and provides appropriate data when users search for information using everyday language.

[0521] This invention provides a system that can centrally manage information within a company and search for it quickly and efficiently. The main components of the system include a server, terminals, and users.

[0522] Server Roles

[0523] The server operates a database for centrally managing various information within the company. The database management system used may be, for example, MySQL. The server reads executive history information, meeting records, company information, employee information, organizational structure charts, intellectual property information, trademark information, service history information, and event information from CSV files and Excel files, and stores this information in the database. It also has the ability to automatically detect missing information and format inconsistencies and handle them appropriately.

[0524] In addition, the server uses speech recognition software (such as Google Cloud Speech-to-Text) to convert the audio data of the meeting recording into text, and also stores the converted text data in a database.

[0525] Device Role

[0526] A terminal is a device that users use to operate the system, and can include PCs, mobile devices, etc. A search interface is provided on the terminal, allowing users to intuitively enter search keywords using natural language. These search keywords are sent from the terminal to the server.

[0527] User operations

[0528] A user uses a terminal to search for the information they need. For example, they type "Yamada Taro executive appointment history" into the search box. When this search query is sent to the server, the server searches the database for relevant information and sends the results in summary form to the terminal. The user can check the summary of the search results and click to view more detailed information.

[0529] Specific examples

[0530] For example, to search for the executive history of director Yamada Taro, a user would enter "Yamada Taro executive appointment history" into the search box on their device. The server receives this query, searches the database for relevant information, and sends Yamada Taro's position, appointment date, and other information in a summary format to the device. If the user wants to check more detailed information from the summary results, they can send a request from their device again, and the server will provide the detailed information.

[0531] As a concrete example of converting audio data from meeting records into text, the recording data of a board meeting is converted into text using Google Cloud Speech-to-Text and the content is stored in a database. This text data can later be searched as "Meeting records regarding the announcement of new services in 2019."

[0532] Prompt Sentence Examples

[0533] For example, enter the following prompt into the system:

[0534] "Find the conference notes for the 2019 New Service Launch."

[0535] In this way, the system can efficiently manage a wide variety of information according to user needs and support natural language searches, greatly improving business efficiency.

[0536] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0537] Step 1: Capture information

[0538] Input: CSV and Excel files containing executive history information, meeting notes, company information, employee information, organizational charts, intellectual property information, trademark information, service history information, and event information.

[0539] How it works: The server reads these files and detects missing data and format inconsistencies. It automatically handles any missing data and formats that are mismatched.

[0540] Data processing / calculation: Standardize data formats and fill in missing data to ensure consistency.

[0541] Output: Store the organized data in a database.

[0542] Specific operation: Read executive history information from a CSV file, fill in missing values, and store it in a MySQL database.

[0543] Step 2: Enter search keywords and submit

[0544] Input: Keywords entered by the user into the device's search interface (e.g., "Yamada Taro executive appointment history").

[0545] How it works: A user enters keywords into the search box on their device and clicks the search button. This input is sent from the device to the server.

[0546] Data processing / calculation: None in particular (sending keywords).

[0547] Output: Sending search keywords from the terminal to the server.

[0548] Specific operation: The user enters "Yamada Taro executive appointment history" and clicks the search button.

[0549] Step 3: Processing the search query and extracting search results

[0550] Input: Search keywords sent from the device (e.g., "Yamada Taro executive appointment history").

[0551] Operation: The server searches the database based on the received search keywords and extracts relevant information.

[0552] Data processing / calculation: Generate database queries and extract relevant data.

[0553] Output: Extraction of relevant search results.

[0554] Specific operation: The server searches and extracts executive history information related to "Yamada Taro" from the database.

[0555] Step 4: Summary of search results

[0556] Input: Search results extracted by the server (e.g., executive history information for "Yamada Taro").

[0557] Operation: The server sends the extracted results in summary form to the terminal.

[0558] Data processing / computation: summarizing and formatting information.

[0559] Output: Sends summarized search results to the terminal.

[0560] Specific operation: The server summarizes the job title and start date for Yamada Taro and sends it to the terminal.

[0561] Step 5: Request and view more information

[0562] Input: The item the user selects from the summary results (e.g., detailed information about "Yamada Taro").

[0563] How it works: When a user clicks on a specific item, the device sends a request to the server to get more information. The server retrieves the details from the database and sends them to the device.

[0564] Data processing / calculation: Obtain detailed information from the database.

[0565] Output: Send detailed information to the terminal for display.

[0566] Specific operation: The user clicks on the detailed information link for "Yamada Taro," and the server retrieves and displays the department he holds and his specific job history.

[0567] Step 6: Convert audio data to text

[0568] Input: Audio data from a meeting recording.

[0569] How it works: The server converts the audio data into text using Google Cloud Speech-to-Text.

[0570] Data processing / calculation: Converts voice data into text data.

[0571] Output: The converted text data.

[0572] Specific operation: The recording data of the board meeting is converted into text using Google Cloud Speech-to-Text and saved in the database as "Minutes of the board meeting held on January 1, 2019."

[0573] Step 7: Perform a natural language search

[0574] Input: A natural language query entered by a user into a search interface (e.g., "Conference notes on new service announcements in 2019").

[0575] How it works: The server uses natural language processing tools (such as NLTK) to parse the search query and search a database for relevant information.

[0576] Data processing / calculation: Search query analysis and query generation.

[0577] Output: Send relevant search results to your device.

[0578] Specific operation: The server analyzes the "Meeting records regarding the announcement of new services in 2019" using a natural language processing tool, searches for the relevant meeting records, and sends them to the terminal.

[0579] (Application example 1)

[0580] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0581] In conventional information management systems, important information such as factory manufacturing information, machine maintenance history, and work instructions is stored in a distributed manner, making it difficult to quickly search and centrally manage information. Furthermore, when managing the contents of meetings and discussions as voice data, there was a lack of systems that could convert the voice data into text and make it easily searchable. Furthermore, there was an insufficient system that supported searches in natural language, making intuitive searches difficult.

[0582] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0583] In this invention, the server includes means for integrating and managing executive appointment histories, meeting records, company information, employee information, organizational charts, patent information, trademark information, service history, and event information, means including a search interface for searching the information, means for summarizing and displaying search results, means for displaying detailed information of relevant search results, means for converting audio data of meeting records and meetings into text, and means for indexing and storing the text data in a database. This enables unified management of information, rapid search, and intuitive search in natural language.

[0584] "Officer appointment history" is data that records and manages information such as the appointment date, position, and term of office of an officer.

[0585] A "meeting record" is a document that records the contents of a meeting, speakers, remarks, agenda, conclusions, etc., and can be referenced later.

[0586] "Company information" refers to data including basic company information, date of establishment, location, business activities, capital, and board of directors composition.

[0587] "Employee information" refers to data that records and manages personal and work-related information such as an employee's name, department, position, date of joining, and contact information.

[0588] An "organizational chart" is a visual representation of the structure of a company or organization, showing each department and their relationships.

[0589] "Patent information" is data that records detailed information about patents, such as patent number, patent name, inventor, application date, registration date, and patent owner.

[0590] "Trademark information" is data that records detailed information such as the trademark name, trademark registration number, registration date, and trademark owner.

[0591] "Service history" is data that records information such as the type of service provided by a company, the start date of service, the person in charge, the service content, and the customer.

[0592] "Event information" is data that records information such as the name, date, time, location, participants, and content of events that a company has hosted or participated in.

[0593] A "search interface" is a portion of software that provides input functionality that a user uses to quickly search for specific information.

[0594] A "means for converting voice data to text" is a technique or device for taking voice input and converting it into textual data.

[0595] A "means for indexing and storing data in a database" is a technique or method for organizing and storing data in a database in an easily searchable format.

[0596] "Natural language search" is a search function that analyzes search queries entered by users in natural language and identifies and presents relevant information.

[0597] The present invention provides a smart manufacturing information management system that is installed on robots used in factories. This system centrally manages important information within the factory and supports searches using natural language. A specific embodiment of the system will be described below.

[0598] 1. System configuration

[0599] The system is composed of the roles of a server, a terminal, and a user.

[0600] server

[0601] The server maintains a database that centrally manages information collected within the factory, including executive appointment histories, meeting records, company information, employee information, organizational charts, patent information, trademark information, service history, and event information. The server also reads this information from CSV files and Excel files and stores it in the database. The server processes the voice data using software to convert the voice data to text (for example, the speech_recognition library).

[0602] Terminal

[0603] Users access the search interface using a terminal such as a PC or mobile device. The terminal accepts search query input and sends it to the server. The terminal also has the function of displaying search results in summary form and in detailed information.

[0604] 2. Information intake

[0605] The server reads various information used within the factory from CSV and Excel files, including information on newly appointed executives and the latest meeting records. The read data is stored in a database, and any missing or inconsistent data is dealt with appropriately.

[0606] 3. Search function

[0607] Users perform natural language searches through their devices. For example, they can enter "contents of the quality control meeting in September 2023," and the query is sent to the server. The server analyzes the query and searches the database for relevant information. Search results are displayed on the device in summary form, and detailed information is displayed when the user selects a specific item.

[0608] 4. Converting Audio Data to Text

[0609] The server converts recordings of meetings and discussions into text. For example, if you upload a recording of a quality control meeting, it will be converted into text using the speech_recognition library and the content will be stored in a database.

[0610] 5. Specific Examples

[0611] For example, if a quality control meeting is held at a factory and the audio data is recorded, the recording can be uploaded to the smart manufacturing information management system and the meeting content converted into text, and then by searching for "quality control meeting content in September 2023," the relevant meeting records can be easily retrieved.

[0612] Prompt Sentence Examples

[0613] "Upload the audio file, convert the meeting record to text, and save it in the database. Then, search for 'Contents of the Quality Control Meeting in September 2023' and display it."

[0614] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0615] Step 1: Capture information

[0616] The server reads executive appointment histories, meeting records, company information, employee information, organizational charts, patent information, trademark information, service history, and event information collected within the factory from CSV and Excel files. The server stores the read data in a database and processes any missing or inconsistent data appropriately. The input is CSV and Excel files, and the output is a centrally managed database.

[0617] Step 2: Enter a search query

[0618] A user accesses the search interface using a terminal (PC or mobile device) and inputs a search query in natural language. For example, they might input "Contents of the quality control meeting in September 2023." The input is the user's search query, and the output is a search request to the server.

[0619] Step 3: Parsing the search query and searching the data

[0620] The server analyzes the search query received from the user and searches for relevant information from the database. For example, if the query "Contents of the quality control meeting in September 2023" is entered, the server analyzes the query using a natural language processing algorithm and extracts the relevant meeting records from the database. The input is the search query, and the output is the search result data.

[0621] Step 4: Summarizing and displaying search results

[0622] The server summarizes the extracted search results and sends them to the terminal. The search results are displayed in summary form on the user's terminal screen. For example, the title and main topics of a particular meeting record are displayed as a summary. The input is the search result data, and the output is the summarized search results.

[0623] Step 5: Obtain and display detailed information

[0624] The user selects a specific item from the summarized search results and requests detailed information. The server retrieves the detailed information for the selected item from the database and sends it to the terminal. For example, if the user clicks on the detailed information for a meeting record, all statements and decisions made during the meeting are displayed as details. The input is the user's request for detailed information, and the output is the detailed information.

[0625] Step 6: Convert audio data to text

[0626] The server receives the audio data of the meeting or conference and converts it into text. It uses the speech_recognition library to analyze the audio data and convert it into text data. The input is the audio data and the output is the converted text data.

[0627] Step 7: Save the text data

[0628] The server indexes the converted text data in a database and stores it. For example, the audio recording of a quality control meeting is converted into text and the content is stored in a database. The input is text data, and the output is an indexed database.

[0629] Step 8: Perform a natural language search

[0630] The user then uses the device to perform a natural language search to quickly obtain the desired information. For example, if the user searches for "quality control meeting contents in September 2023," relevant meeting records are displayed as search results. The input is the user's search query, and the output is the relevant search results.

[0631] These processing steps enable centralized management of important information within the factory, rapid search, and intuitive search in natural language.

[0632] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0633] This invention is a system that centrally manages in-house information and can quickly search for information regardless of changes in personnel or the passage of time. By combining this system with an emotion engine, it becomes possible to provide information flexibly according to the user's emotional state.

[0634] 1. Information intake

[0635] First, various information (executive appointment history, meeting records, company information, employee information, organizational chart, patent information, trademark information, service history, event information) is collected on the server. The server reads this information from CSV files, Excel files, etc. and stores it in a centrally managed database. Any missing data is also processed to ensure data integrity.

[0636] Example: The server reads "Executive Appointment History.csv" and converts information such as executive names, appointment dates, and positions into a data frame (e.g., pandas).

[0637] 2. Information Search

[0638] Users access the search interface using a terminal (PC or mobile device) and enter the information they want to search for in natural language. The search interface is linked to an emotion engine that analyzes emotions from the user's input and usage status. The emotion engine analyzes emotions from the user's input and adjusts the search results.

[0639] Example: A user types in "2021 board meeting minutes," submits that search query, and the server searches the database for relevant information.

[0640] 3. Emotion Engine Analysis

[0641] The emotion engine analyzes user input and usage patterns to determine their emotions. Based on this analysis, it changes the display format and order of search results. It provides quick information when the user is in a hurry, and displays more details when more information is needed.

[0642] For example, if the emotion engine recognizes "impatience" from user input, it will prioritize and display the most relevant information, allowing the user to quickly obtain information.

[0643] 4. Summarizing and displaying search results

[0644] The server generates data to summarize and display the search results. Based on the analysis results of the emotion engine, it determines the summary format and display order of the search results. The server then sends the summarized search results to the terminal, which then displays them to the user.

[0645] Example: The server searches the executive appointment history database and extracts information related to "Yamada Taro." As a result, the name, title, appointment date, etc. of the relevant executive are displayed in summary form. If the emotion engine detects "doubt," it provides more detailed information.

[0646] 5. Viewing detailed information

[0647] When a user selects a specific item from the summarized search results, the device sends a request to the server to obtain detailed information about that item. The server retrieves the detailed information about the selected item from the database and sends it to the device. The emotion engine also monitors the user's emotions as they check the detailed information and provides additional information as needed.

[0648] Example: When a user clicks on the details of "Yamada Taro," detailed career information for that executive (such as date of appointment, position, department, etc.) is displayed.

[0649] 6. Audio to Text Conversion

[0650] If the conference record is provided as audio data, the server converts the audio data into text, which is then stored in the database for search and display.

[0651] Example: Converting recordings of board meetings into text and storing the content in a database so that it can be searched in natural language.

[0652] 7. Natural Language Search

[0653] The search interface allows users to input natural sentences and keywords, allowing the system to understand their intent and search for relevant information. This allows the system to analyze not only specific keywords but also detailed sentences to identify relevant information.

[0654] Example: When a user enters a query for "all company events held in 2020," the server parses the natural language query and searches the database for relevant event information.

[0655] This system can flexibly provide information according to the user's emotional state. This allows users to quickly obtain the information they need, improving work efficiency. In addition, by combining it with an emotion engine, the user experience is improved and the stress of information search is reduced.

[0656] The processing flow will be explained below.

[0657] Step 1:

[0658] Information capture

[0659] The server loads information such as executive appointments and meeting records from external data sources such as CSV and Excel files, including the automatic loading of regularly updated data.

[0660] Example: The server reads "Executive Appointment History.csv" and converts information such as executive names, appointment dates, and positions into a data frame (e.g., pandas).

[0661] Step 2:

[0662] Data Preprocessing

[0663] The server preprocesses the data it loads, which includes imputing missing values, converting data types, and removing unnecessary data.

[0664] For example, the server will impute missing values ​​with 'N / A' and convert date fields to the appropriate data type (e.g. datetime).

[0665] Step 3:

[0666] Storage in the database

[0667] The server stores the preprocessed data in a centralized database, thereby realizing integrated management of information.

[0668] Example: The server inserts executive history data into an SQL database and creates the appropriate indexes.

[0669] Step 4:

[0670] Entering a query through the search interface

[0671] The user accesses the search interface on the terminal and inputs the information they want to search for in natural language.

[0672] Example: User enters "2021 Board Meeting Minutes."

[0673] Step 5:

[0674] Submitting a search query

[0675] The terminal transmits the input search query to the server, which then receives the query and begins analyzing it.

[0676] Example: A device sends a query for "2021 board meeting minutes" as an HTTP request to a server.

[0677] Step 6:

[0678] Database search

[0679] The server analyzes the incoming search query and searches a centralized database, sorting the results by relevance.

[0680] Example: The server uses an SQL query to extract data corresponding to "2021 Board of Directors Meeting Minutes."

[0681] Step 7:

[0682] Emotion Engine Analysis

[0683] The server uses an emotion engine to analyze emotions from user input and usage, and adjusts search results based on the user's emotional state.

[0684] Example: If the server detects the emotion "anxiety," it will prioritize displaying the most relevant information.

[0685] Step 8:

[0686] Search result summary

[0687] The server generates data to summarize and display search results, allowing users to see concise information.

[0688] Example: The server extracts the most important fields from the search results (date, time, topic, participants, etc.) and presents them in a summarized format.

[0689] Step 9:

[0690] Displaying search results

[0691] The server sends the summarized search results to the terminal, which then displays them to the user, who can then check the list of search results.

[0692] Example: Display a list of summarized board meeting minutes in a device browser.

[0693] Step 10:

[0694] Request more information

[0695] When the user selects an item for which they wish to check details, the terminal sends a request to the server to obtain detailed information about that item.

[0696] Example: When a user clicks on a particular board meeting record, a request is sent to the server to get more information about it.

[0697] Step 11:

[0698] Get more information

[0699] The server retrieves detailed information about the selected item from the database and sends it to the terminal, allowing the user to view the detailed information.

[0700] Example: The server retrieves detailed information (e.g., text of speeches) of the selected board meeting record from the database and sends it to the terminal.

[0701] Step 12:

[0702] Viewing detailed information

[0703] The terminal displays the received detailed information to the user, allowing the user to check the details of the specific information.

[0704] Example: A detailed record of the selected board meeting is displayed on the terminal screen.

[0705] Step 13:

[0706] Converting audio data to text

[0707] If the conference recording is provided as audio data, the server converts the audio data into text, which is then stored in the database for search and display.

[0708] Example: The server uses speech recognition technology to convert the audio data of a meeting into text and stores the text in a database.

[0709] Step 14:

[0710] Providing natural language search

[0711] The server analyzes search queries entered in natural language and provides appropriate search results, enabling users to perform searches easily and intuitively.

[0712] Example: When a user enters a query for "all company events held in 2020," the server parses the natural language query and searches the database for relevant event information.

[0713] Step 15:

[0714] Emotional Engine Feedback

[0715] The server changes the display format and presentation order of search results depending on the user's emotional state. Furthermore, the emotion engine learns from user feedback and uses it to improve the accuracy of future search results.

[0716] For example, if the sentiment engine detects "doubt," it will prioritize more detailed search results.

[0717] Example 2

[0718] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0719] In conventional in-house information management systems, information is managed in a decentralized manner, making it difficult to quickly search for information when the person in charge changes or when time passes. Furthermore, the system is unable to provide information flexibly according to the user's emotional state, which can be stressful. Additionally, handling voice data is difficult, making it difficult to efficiently search and summarize voice data.

[0720] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for integrating and managing executive information, meeting information, company information, employee data, organizational structure, intellectual property information, trademark data, service history, and event records, a means including an interface for searching the information, a means for adjusting and displaying search results according to the user's emotional state, a means for generating and displaying summaries of the corresponding search results, and a means for displaying detailed information. This allows information to be centrally managed, enabling rapid information retrieval regardless of changes in personnel or the passage of time. Furthermore, appropriate information can be provided according to the user's emotional state, reducing user stress. Furthermore, by converting and managing audio data of meeting information into text, audio data can be efficiently searched and summarized.

[0721] "Officer information" is data relating to the appointment history and positions of officers in a company or organization.

[0722] "Meeting information" refers to data including minutes, participant lists, decisions, etc., of meetings held within a company or organization.

[0723] "Corporate information" refers to data that includes basic information such as a company overview, performance, business activities, and organizational structure.

[0724] "Employee Data" means data that includes personal information, job title, salary, work history, etc., of employees within a company or organization.

[0725] "Organizational structure" is information that indicates the arrangement of departments and divisions in a company or organization, the hierarchical structure of positions, and the like.

[0726] "Intellectual property information" refers to data relating to intellectual property such as patents, trademarks, and copyrights held by a company or organization.

[0727] "Trademark Data" means data relating to trademarks registered by a company or organization and their details.

[0728] "Service history" refers to data such as usage history of services and products provided by a company or organization, and customer feedback.

[0729] "Event records" are detailed information about events, seminars, exhibitions, etc. held by a company or organization.

[0730] "Interface" refers to the input screen or operation panel on a PC or mobile device that users use to search for information.

[0731] "Means for adjusting and displaying search results according to the user's emotional state" refers to a function that analyzes the user's emotions and adjusts the display format and order of search results based on the results.

[0732] "Means for generating and displaying summaries" refers to the function of concisely summarizing search results and displaying them in a format that is easy for the user to view.

[0733] "Means for displaying detailed information" refers to a function that displays detailed information when a user selects a specific search result.

[0734] "Means for converting voice data into text" refers to the function of recognizing voice data such as meeting records, converting it into text format, and saving it.

[0735] This invention is a system for centrally managing in-house information and for quickly and flexibly searching and displaying the information. This system is implemented as follows.

[0736] The server imports various information, such as executive information, meeting information, company information, employee data, organizational structure, intellectual property information, trademark data, service history, and event records, in the form of CSV files or Excel files, converts it into a data frame using the pandas library, and stores it in a centrally managed database. During this process, it detects and automatically supplements or corrects missing data and format inconsistencies. It also uses Google Speech-to-Text or similar speech recognition software to convert the audio data of the meeting information into text and store it in the database.

[0737] A user accesses a search interface using a PC or mobile device. The user enters a search query in natural language, which is then sent from the device to the server. For example, a query such as "2021 board meeting minutes" might be considered. The server receives the query and analyzes the user's emotional state using an emotion engine, such as the transformers library.

[0738] The emotion engine analyzes the user's emotions and adjusts the display format and order of search results based on the results. For example, if the user is feeling impatient, the most relevant information will be displayed first, providing information quickly. The emotion analysis results are also reflected in the generation of summaries of search results, which are sent to the device in an appropriate format depending on the user's emotional state.

[0739] The server searches the database, extracts the relevant information, formats it, and generates a summary. The generated summary is sent to the device in a format that is easy for the user to view, based on the analysis results of the emotion engine. When the user selects a specific item from the search results displayed in summary form, the device sends a request for detailed information to the server. The server retrieves the detailed information, sends it to the device in an appropriate format, and displays it to the user.

[0740] For example, if a user wants to find detailed information about an executive named "Yamada Taro," the user enters "details about Yamada Taro" into the search interface. This query is sent to the server, which retrieves the relevant executive information from the database and displays it as detailed information.

[0741] This system centralizes information management, making it possible to search for information quickly and effectively without being affected by changes in personnel or the passage of time.In addition, the introduction of an emotion engine enables flexible and prompt provision of information according to the user's emotions, reducing user stress.

[0742] Examples of prompts include "2021 board meeting minutes," "More information about Taro Yamada," and "All company events held in 2020." By entering these prompts, the system can quickly search for the relevant information and provide it to the user.

[0743] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0744] Step 1: Capture information

[0745] The server imports various information such as executive information, meeting information, company information, employee data, organizational structure, intellectual property information, trademark data, service history, and event records from CSV and Excel files. Specifically, it uses the pandas library to convert these files into data frames and stores them in a centrally managed database. The input is a CSV or Excel file, and the output is the data stored in the database.

[0746] Step 2: Data integrity check and completion

[0747] The server detects missing or inconsistent values ​​in the imported data and automatically completes or corrects them, for example by filling blank cells with a default value such as "unknown." The input is a data frame, and the output is the consistent data.

[0748] Step 3: Convert audio data to text

[0749] The server receives the audio data of the conference information and converts it into text using speech recognition software (e.g., Google Speech-to-Text). The converted text data is also stored in a database. The input is audio data, and the output is the converted text data.

[0750] Step 4: Finding information

[0751] A user uses a terminal to access a search interface and inputs the information they want to search for in natural language. For example, they input a query such as "2021 board meeting minutes." The terminal then sends this query to the server. The input is the user's search query, and the output is the transmission of the search query to the server.

[0752] Step 5: Analysis by Emotion Engine

[0753] The server analyzes the received search query and detects the user's emotions using an emotion engine. The emotion engine uses libraries such as the transformers library. Based on the analysis results, the server adjusts the display format and order of search results. The input is the search query, and the output is the emotion analysis results.

[0754] Step 6: Generating and formatting search results

[0755] The server retrieves search results from the database and generates a summary based on the analysis results of the emotion engine. For example, it summarizes information related to "Yamada Taro" and sends it to the terminal in the form of job title, date of appointment, etc. The input is the database search results and emotion analysis results, and the output is the summarized search results.

[0756] Step 7: Viewing search results

[0757] The terminal displays the search results in summary form received from the server to the user. The user checks the displayed information and selects detailed information as necessary. The input is the summarized search results, and the output is the display to the user.

[0758] Step 8: Request and view more information

[0759] When a user selects a specific item from the search results displayed in summary form, the terminal requests detailed information from the server. The server retrieves the detailed information from the database and sends it to the terminal. The terminal displays the detailed information to the user. The input is a request for detailed information, and the output is the display of the detailed information.

[0760] (Application example 2)

[0761] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0762] Conventional factory robot management systems have had problems such as difficulty in centrally managing robot operation history and maintenance records, and difficulty for operators in quickly obtaining the information they need. Furthermore, information search does not provide flexible information according to the user's emotional state, which means operators cannot obtain appropriate information in emergencies or during normal times, resulting in reduced work efficiency.

[0763] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for integrating and managing executive appointment histories, meeting records, company information, employee information, organizational charts, patent information, trademark information, service history, and event information; means including a search interface for searching for the information; means for summarizing and displaying search results; means for displaying detailed information of relevant search results; means for adjusting the search results according to the user's emotional state using an emotion engine; means for integrating and managing the operation history and maintenance records of factory robots in a database; and means for searching for related information based on a query in the user's natural language. This enables the user to flexibly and quickly obtain necessary information according to their emotional state, thereby achieving efficient operation of factory robots.

[0764] "Officer appointment history" refers to historical information on the appointment dates and positions of company officers.

[0765] "Meeting records" refers to recorded information such as minutes, audio data, and attendee information related to meetings within a company.

[0766] "Company information" refers to basic information about a company, such as the date of establishment, the name of the representative, and the main business activities.

[0767] "Employee information" refers to individual information about employees within a company, such as name, department, position, and start date.

[0768] An "organizational chart" is a diagram that shows the hierarchical internal structure of a company, visually representing the relationships between departments and positions.

[0769] "Patent information" refers to information about patents held by a company, including, for example, patent number, invention name, application date, registration date, etc.

[0770] "Trademark information" refers to information about trademarks held by a company, including, for example, trademark registration number, trademark name, registration date, etc.

[0771] "Service history" refers to the performance and history information of various services provided by a company, such as the date of service execution, customer information, and service content.

[0772] "Event information" refers to information about an event held by a company, such as the event date, participant list, and event content.

[0773] "Search interface" refers to the UI (user interface) that a user uses to search for specific information.

[0774] An "emotion engine" is a system that analyzes a user's emotional state (e.g., urgency, impatience, doubt, etc.) and adjusts the information provided based on that.

[0775] "Operation history" refers to the operating status and history information of a factory robot, such as work content, operating time, and error records.

[0776] "Maintenance records" refers to historical information about maintenance performed on a factory robot, such as inspection dates, repair details, and replacement parts.

[0777] A "natural language query" refers to a search request written in everyday language or sentence format, and refers to a technology that analyzes the entire sentence, not just specific keywords, to search for related information.

[0778] This invention builds a system that centrally manages the operation history and maintenance records of factory robots and is equipped with an emotion engine to provide flexible information according to the user's emotional state.

[0779] The server creates a database to integrate and manage executive appointment histories, meeting records, company information, employee information, organizational charts, patent information, trademark information, service history, and event information. This information is read from CSV files and Excel files, converted into a data frame (e.g., pandas), and stored. The server maintains data integrity and takes appropriate action if there is missing data.

[0780] Users access the search interface using a terminal (PC or mobile device) and enter search queries in natural language. To enable natural language searches, the server employs Natural Language Processing (NLP) technology. The interface analyzes the text entered by the user and performs semantic searches.

[0781] The search interface is connected to an emotion engine, which analyzes emotions from user input and usage and adjusts the display format and presentation order of search results. For example, if a user enters the query "Please tell me the latest maintenance records. I'm in a hurry," the emotion engine will determine the user's emotional state as "urgent" and prioritize displaying the most relevant and important information.

[0782] The server integrates and manages the operating history and maintenance records of factory robots in a database, and generates data for summarizing and displaying search results. Based on the analysis results of the emotion engine, it determines the summary format and display order of the search results. When the search results are displayed and the user requests more information, the terminal sends the request to the server, which then retrieves the information from the database and sends it to the terminal.

[0783] Additionally, audio data such as meeting recordings can be converted to text and stored in a database, enabling comprehensive searches with natural language queries. This process is performed using speech recognition software.

[0784] The hardware and software used specifically include the Python programming language, the pandas library, the TextBlob library, and scikit-learn, etc. Using this system, users can quickly obtain the information they need based on their emotional state, enabling the efficient operation of factory robots.

[0785] A specific example query is the prompt "Please tell me the latest maintenance records. I'm in a hurry." In this example, the emotion engine analyzes the user's emotion of "I'm in a hurry" and quickly displays the most important maintenance records.

[0786] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0787] Step 1:

[0788] The server reads executive appointment history, meeting records, company information, employee information, organizational charts, patent information, trademark information, service history, and event information from various CSV and Excel files, converts them into data frames (such as pandas), and stores them in a database. This enables integrated data management. The input is various information files, and the output is an integrated database.

[0789] Step 2:

[0790] A user accesses the search interface using a terminal (PC or mobile device) and inputs a search query in natural language, for example, by entering a prompt sentence such as "Please tell me the latest maintenance records. I'm very urgent." The input is the user's query text, which is passed to the search interface.

[0791] Step 3:

[0792] The search interface analyzes the user's input text using natural language processing (NLP) techniques. In this process, the input query text is broken down into tokens and semantic analysis is performed. The input is the query text, and the output is the analysis results (related keywords and meanings).

[0793] Step 4:

[0794] The emotion engine analyzes the user's emotional state from the input text. This analysis is performed using the TextBlob library, etc. For example, from the text "I'm in a hurry," the user's emotional state is determined to be "I'm in a hurry." The input is the user's query text, and the output is the emotional state.

[0795] Step 5:

[0796] The server searches the database based on the analysis results and the emotional state, extracting relevant information. During this process, it summarizes the search results and prioritizes the information according to the emotional state. The input is the analysis results and the emotional state, and the output is the summarized search results.

[0797] Step 6:

[0798] The terminal displays the summarized search results sent from the server in the most appropriate format based on the user's emotional state. The input is the summarized search results, and the output is the information displayed on the user's terminal screen.

[0799] Step 7:

[0800] When a user selects a specific item from the search results displayed in summary form, the terminal sends a request to the server to obtain detailed information about that item. The input is the item selected by the user, and the output is a request to obtain detailed information.

[0801] Step 8:

[0802] The server retrieves detailed information about the selected item from the database and sends it back to the terminal. The input is a request to retrieve detailed information, and the output is the detailed information.

[0803] Step 9:

[0804] The terminal displays the detailed information sent from the server to the user. This display is also adjusted based on the emotional state. The input is the detailed information, and the output is the detailed information displayed on the user's terminal screen.

[0805] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0806] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0807] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0808] [Third embodiment]

[0809] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0810] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0811] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0812] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0813] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0814] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0815] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0816] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0817] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0818] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0819] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0820] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0821] This invention provides a system that centrally manages in-house information and enables quick information retrieval regardless of changes in personnel or the passage of time. This system is composed of a server, terminals, and users, each playing their own role.

[0822] 1. Information intake

[0823] First, various information (executive appointment history, meeting records, company information, employee information, organizational chart, patent information, trademark information, service history, event information) is collected on the server. The server reads this information from CSV files, Excel files, etc. and stores it in a centrally managed database. Any missing data is also processed to ensure data integrity.

[0824] Example: Data on the history of executive appointments is read from a CSV file, and after necessary preprocessing, it is stored in the database. This creates a centrally managed database.

[0825] 2. Information Search

[0826] Users access the search interface using a terminal (PC or mobile device). The search interface supports searches in natural language, allowing users to intuitively enter search keywords. The entered search keywords are sent from the terminal to the server.

[0827] Example: When a user types "Yamada Taro executive history" into a search box, the search query is submitted and the server searches the database for relevant information.

[0828] 3. Summary display of search results

[0829] The server searches the database based on the search query and extracts relevant results. The extracted results are processed in summary form and sent to the terminal, where the user can view the summary of the search results on the terminal screen.

[0830] Example: The server searches the executive history database and extracts information related to "Yamada Taro." As a result, the name, title, and date of appointment of the relevant executive are displayed in summary form.

[0831] 4. Viewing detailed information

[0832] When a user selects a specific item from the summarized search results, the terminal sends a request to the server to obtain detailed information about that item. The server then retrieves the detailed information about the selected item from the database and sends it to the terminal.

[0833] Example: When a user clicks on the details of "Yamada Taro," detailed career information for that executive (such as date of appointment, position, department, etc.) is displayed.

[0834] 5. Converting Audio Data to Text

[0835] If there is audio data from the conference recording, the server converts the audio data into text, which is then stored in the database for search and display.

[0836] Example: Converting recordings of board meetings into text and storing the content in a database so that it can be searched in natural language.

[0837] 6. Natural Language Search

[0838] The search interface provides users with the ability to enter queries in natural language, allowing for analysis of specific keywords as well as detailed sentences to identify relevant information.

[0839] Example: If a user enters "meeting notes about new service announcements in 2019," the server will parse the query and search for new services announced in 2019 and related meeting notes.

[0840] This system allows for the fast and efficient management and search of various internal company information. Even if the person in charge changes, the centralized management of information ensures business continuity, and past information and future action items can be easily accessed. This significantly improves business efficiency and enables quick decision-making.

[0841] The processing flow will be explained below.

[0842] Step 1:

[0843] Information capture

[0844] The server loads information such as executive appointments and meeting records from external data sources such as CSV and Excel files, including the automatic loading of regularly updated data.

[0845] Example: The server reads "Executive Appointment History.csv" and converts information such as executive names, appointment dates, and positions into a data frame (e.g., pandas).

[0846] Step 2:

[0847] Data Preprocessing

[0848] The server preprocesses the data it loads, which includes imputing missing values, converting data types, and removing unnecessary data.

[0849] For example, the server will impute missing values ​​with 'N / A' and convert date fields to the appropriate data type (e.g. datetime).

[0850] Step 3:

[0851] Storage in the database

[0852] The server stores the preprocessed data in a centralized database, thereby realizing integrated management of information.

[0853] Example: The server inserts executive history data into an SQL database and creates the appropriate indexes.

[0854] Step 4:

[0855] Entering a query through the search interface

[0856] The user accesses the search interface on the terminal and inputs the information they want to search for in natural language.

[0857] Example: User enters "2021 Board Meeting Minutes."

[0858] Step 5:

[0859] Submitting a search query

[0860] The terminal transmits the input search query to the server, which then receives the query and begins analyzing it.

[0861] Example: A device sends a query for "2021 board meeting minutes" as an HTTP request to a server.

[0862] Step 6:

[0863] Database search

[0864] The server analyzes the incoming search query and searches a centralized database, sorting the results by relevance.

[0865] Example: The server uses an SQL query to extract data corresponding to "2021 Board of Directors Meeting Minutes."

[0866] Step 7:

[0867] Search result summary

[0868] The server generates data to summarize and display search results, allowing users to see concise information.

[0869] Example: The server extracts the most important fields from the search results (date, time, topic, participants, etc.) and presents them in a summarized format.

[0870] Step 8:

[0871] Displaying search results

[0872] The server sends the summarized search results to the terminal, which then displays them to the user, who can then check the list of search results.

[0873] Example: Display a list of summarized board meeting minutes in a device browser.

[0874] Step 9:

[0875] Request more information

[0876] When the user selects an item for which they wish to check details, the terminal sends a request to the server to obtain detailed information about that item.

[0877] Example: When a user clicks on a particular board meeting record, a request is sent to the server to get more information about it.

[0878] Step 10:

[0879] Get more information

[0880] The server retrieves detailed information about the selected item from the database and sends it to the terminal, allowing the user to view the detailed information.

[0881] Example: The server retrieves detailed information (e.g., text of speeches) of the selected board meeting record from the database and sends it to the terminal.

[0882] Step 11:

[0883] Viewing detailed information

[0884] The terminal displays the received detailed information to the user, allowing the user to check the details of the specific information.

[0885] Example: A detailed record of the selected board meeting is displayed on the terminal screen.

[0886] Step 12:

[0887] Converting audio data to text

[0888] If the meeting recording is provided as audio data, the server converts the audio data into text, which is then stored in a database and made searchable.

[0889] Example: The server uses speech recognition technology to convert the audio data of a meeting into text and stores the text in a database.

[0890] Step 13:

[0891] Providing natural language search

[0892] The server analyzes search queries entered in natural language and provides appropriate search results, enabling users to perform searches easily and intuitively.

[0893] Example: When a user enters a query for "all company events held in 2020," the server parses the natural language query and searches the database for relevant event information.

[0894] Example 1

[0895] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0896] Modern business activities require centralized management and search of a wide range of information, including executive history, meeting records, company information, employee information, organizational charts, intellectual property information, trademark information, service history information, and event information. However, when this information is stored in multiple different formats, it is difficult to automatically detect and correct missing information or format inconsistencies. Furthermore, there is a need for systems that can convert voice data into text and efficiently search using natural language, but current systems lack these capabilities.

[0897] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0898] In this invention, the server includes means for integrating and managing executive history information, meeting records, company information, employee information, organizational structure charts, intellectual property information, trademark information, service history information, and event information, a search interface for searching the information, means for summarizing and displaying search results, means for displaying detailed information of relevant search results, and means for automatically detecting and processing missing information and format inconsistencies in the information. This makes it possible to provide a system that efficiently manages a variety of information within a company, automatically ensures data consistency, and enables intuitive information searches using natural language.

[0899] "Officer history information" is detailed information about the positions held by company officers in the past and the periods of their service.

[0900] "Meeting records" are documents or data that record the content, minutes, participants, and decisions of meetings held within a company.

[0901] "Company information" is a set of data including basic information about a company, its management policies, performance, location, and contact information.

[0902] "Employee information" refers to data such as personal information, job title, role, salary, and career history for all employees belonging to a company.

[0903] An "organizational structure chart" is a diagram that visually shows the relationships between departments and positions within a company.

[0904] "Intellectual property information" refers to information relating to intellectual property rights owned by a company, such as patents, trademarks, copyrights, and trade secrets.

[0905] "Trademark Information" is detailed information about a company's trademarked brands, logos, slogans, etc.

[0906] "Service history information" is data that records history information and provision status regarding various services provided by a company.

[0907] "Event information" refers to information such as detailed information, dates, and participant lists of events hosted or participated in by a company.

[0908] The "search interface means" is a means for providing an interface that is used when a user searches for required information.

[0909] "Means for automatically detecting and handling missing data and format inconsistencies" refers to the system's functionality for automatically detecting missing data and format inconsistencies and handling them appropriately when they occur.

[0910] "Natural language search" is a function that understands search queries and provides appropriate data when users search for information using everyday language.

[0911] This invention provides a system that can centrally manage information within a company and search for it quickly and efficiently. The main components of the system include a server, terminals, and users.

[0912] Server Roles

[0913] The server operates a database for centrally managing various information within the company. The database management system used may be, for example, MySQL. The server reads executive history information, meeting records, company information, employee information, organizational structure charts, intellectual property information, trademark information, service history information, and event information from CSV files and Excel files, and stores this information in the database. It also has the ability to automatically detect missing information and format inconsistencies and handle them appropriately.

[0914] In addition, the server uses speech recognition software (such as Google Cloud Speech-to-Text) to convert the audio data of the meeting recording into text, and also stores the converted text data in a database.

[0915] Device Role

[0916] A terminal is a device that users use to operate the system, and can include PCs, mobile devices, etc. A search interface is provided on the terminal, allowing users to intuitively enter search keywords using natural language. These search keywords are sent from the terminal to the server.

[0917] User operations

[0918] A user uses a terminal to search for the information they need. For example, they type "Yamada Taro executive appointment history" into the search box. When this search query is sent to the server, the server searches the database for relevant information and sends the results in summary form to the terminal. The user can check the summary of the search results and click to view more detailed information.

[0919] Specific examples

[0920] For example, to search for the executive history of director Yamada Taro, a user would enter "Yamada Taro executive appointment history" into the search box on their device. The server receives this query, searches the database for relevant information, and sends Yamada Taro's position, appointment date, and other information in a summary format to the device. If the user wants to check more detailed information from the summary results, they can send a request from their device again, and the server will provide the detailed information.

[0921] As a concrete example of converting audio data from meeting records into text, the recording data of a board meeting is converted into text using Google Cloud Speech-to-Text and the content is stored in a database. This text data can later be searched as "Meeting records regarding the announcement of new services in 2019."

[0922] Prompt Sentence Examples

[0923] For example, enter the following prompt into the system:

[0924] "Find the conference notes for the 2019 New Service Launch."

[0925] In this way, the system can efficiently manage a wide variety of information according to user needs and support natural language searches, greatly improving business efficiency.

[0926] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0927] Step 1: Capture information

[0928] Input: CSV and Excel files containing executive history information, meeting notes, company information, employee information, organizational charts, intellectual property information, trademark information, service history information, and event information.

[0929] How it works: The server reads these files and detects missing data and format inconsistencies. It automatically handles any missing data and formats that are mismatched.

[0930] Data processing / calculation: Standardize data formats and fill in missing data to ensure consistency.

[0931] Output: Store the organized data in a database.

[0932] Specific operation: Read executive history information from a CSV file, fill in missing values, and store it in a MySQL database.

[0933] Step 2: Enter search keywords and submit

[0934] Input: Keywords entered by the user into the device's search interface (e.g., "Yamada Taro executive appointment history").

[0935] How it works: A user enters keywords into the search box on their device and clicks the search button. This input is sent from the device to the server.

[0936] Data processing / calculation: None in particular (sending keywords).

[0937] Output: Sending search keywords from the terminal to the server.

[0938] Specific operation: The user enters "Yamada Taro executive appointment history" and clicks the search button.

[0939] Step 3: Processing the search query and extracting search results

[0940] Input: Search keywords sent from the device (e.g., "Yamada Taro executive appointment history").

[0941] Operation: The server searches the database based on the received search keywords and extracts relevant information.

[0942] Data processing / calculation: Generate database queries and extract relevant data.

[0943] Output: Extraction of relevant search results.

[0944] Specific operation: The server searches and extracts executive history information related to "Yamada Taro" from the database.

[0945] Step 4: Summary of search results

[0946] Input: Search results extracted by the server (e.g., executive history information for "Yamada Taro").

[0947] Operation: The server sends the extracted results in summary form to the terminal.

[0948] Data processing / computation: summarizing and formatting information.

[0949] Output: Sends summarized search results to the terminal.

[0950] Specific operation: The server summarizes the job title and start date for Yamada Taro and sends it to the terminal.

[0951] Step 5: Request and view more information

[0952] Input: The item the user selects from the summary results (e.g., detailed information about "Yamada Taro").

[0953] How it works: When a user clicks on a specific item, the device sends a request to the server to get more information. The server retrieves the details from the database and sends them to the device.

[0954] Data processing / calculation: Obtain detailed information from the database.

[0955] Output: Send detailed information to the terminal for display.

[0956] Specific operation: The user clicks on the detailed information link for "Yamada Taro," and the server retrieves and displays the department he holds and his specific job history.

[0957] Step 6: Convert audio data to text

[0958] Input: Audio data from a meeting recording.

[0959] How it works: The server converts the audio data into text using Google Cloud Speech-to-Text.

[0960] Data processing / calculation: Converts voice data into text data.

[0961] Output: The converted text data.

[0962] Specific operation: The recording data of the board meeting is converted into text using Google Cloud Speech-to-Text and saved in the database as "Minutes of the board meeting held on January 1, 2019."

[0963] Step 7: Perform a natural language search

[0964] Input: A natural language query entered by a user into a search interface (e.g., "Conference notes on new service announcements in 2019").

[0965] How it works: The server uses natural language processing tools (such as NLTK) to parse the search query and search a database for relevant information.

[0966] Data processing / calculation: Search query analysis and query generation.

[0967] Output: Send relevant search results to your device.

[0968] Specific operation: The server analyzes the "Meeting records regarding the announcement of new services in 2019" using a natural language processing tool, searches for the relevant meeting records, and sends them to the terminal.

[0969] (Application example 1)

[0970] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0971] In conventional information management systems, important information such as factory manufacturing information, machine maintenance history, and work instructions is stored in a distributed manner, making it difficult to quickly search and centrally manage information. Furthermore, when managing the contents of meetings and discussions as voice data, there was a lack of systems that could convert the voice data into text and make it easily searchable. Furthermore, there was an insufficient system that supported searches in natural language, making intuitive searches difficult.

[0972] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0973] In this invention, the server includes means for integrating and managing executive appointment histories, meeting records, company information, employee information, organizational charts, patent information, trademark information, service history, and event information, means including a search interface for searching the information, means for summarizing and displaying search results, means for displaying detailed information of relevant search results, means for converting audio data of meeting records and meetings into text, and means for indexing and storing the text data in a database. This enables unified management of information, rapid search, and intuitive search in natural language.

[0974] "Officer appointment history" is data that records and manages information such as the appointment date, position, and term of office of an officer.

[0975] A "meeting record" is a document that records the contents of a meeting, speakers, remarks, agenda, conclusions, etc., and can be referenced later.

[0976] "Company information" refers to data including basic company information, date of establishment, location, business activities, capital, and board of directors composition.

[0977] "Employee information" refers to data that records and manages personal and work-related information such as an employee's name, department, position, date of joining, and contact information.

[0978] An "organizational chart" is a visual representation of the structure of a company or organization, showing each department and their relationships.

[0979] "Patent information" is data that records detailed information about patents, such as patent number, patent name, inventor, application date, registration date, and patent owner.

[0980] "Trademark information" is data that records detailed information such as the trademark name, trademark registration number, registration date, and trademark owner.

[0981] "Service history" is data that records information such as the type of service provided by a company, the start date of service, the person in charge, the service content, and the customer.

[0982] "Event information" is data that records information such as the name, date, time, location, participants, and content of events that a company has hosted or participated in.

[0983] A "search interface" is a portion of software that provides input functionality that a user uses to quickly search for specific information.

[0984] A "means for converting voice data to text" is a technique or device for taking voice input and converting it into textual data.

[0985] A "means for indexing and storing data in a database" is a technique or method for organizing and storing data in a database in an easily searchable format.

[0986] "Natural language search" is a search function that analyzes search queries entered by users in natural language and identifies and presents relevant information.

[0987] The present invention provides a smart manufacturing information management system that is installed on robots used in factories. This system centrally manages important information within the factory and supports searches using natural language. A specific embodiment of the system will be described below.

[0988] 1. System configuration

[0989] The system is composed of the roles of a server, a terminal, and a user.

[0990] server

[0991] The server maintains a database that centrally manages information collected within the factory, including executive appointment histories, meeting records, company information, employee information, organizational charts, patent information, trademark information, service history, and event information. The server also reads this information from CSV files and Excel files and stores it in the database. The server processes the voice data using software to convert the voice data to text (for example, the speech_recognition library).

[0992] Terminal

[0993] Users access the search interface using a terminal such as a PC or mobile device. The terminal accepts search query input and sends it to the server. The terminal also has the function of displaying search results in summary form and in detailed information.

[0994] 2. Information intake

[0995] The server reads various information used within the factory from CSV and Excel files, including information on newly appointed executives and the latest meeting records. The read data is stored in a database, and any missing or inconsistent data is dealt with appropriately.

[0996] 3. Search function

[0997] Users perform natural language searches through their devices. For example, they can enter "contents of the quality control meeting in September 2023," and the query is sent to the server. The server analyzes the query and searches the database for relevant information. Search results are displayed on the device in summary form, and detailed information is displayed when the user selects a specific item.

[0998] 4. Converting Audio Data to Text

[0999] The server converts recordings of meetings and discussions into text. For example, if you upload a recording of a quality control meeting, it will be converted into text using the speech_recognition library and the content will be stored in a database.

[1000] 5. Specific Examples

[1001] For example, if a quality control meeting is held at a factory and the audio data is recorded, the recording can be uploaded to the smart manufacturing information management system and the meeting content converted into text, and then by searching for "quality control meeting content in September 2023," the relevant meeting records can be easily retrieved.

[1002] Prompt Sentence Examples

[1003] "Upload the audio file, convert the meeting record to text, and save it in the database. Then, search for 'Contents of the Quality Control Meeting in September 2023' and display it."

[1004] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1005] Step 1: Capture information

[1006] The server reads executive appointment histories, meeting records, company information, employee information, organizational charts, patent information, trademark information, service history, and event information collected within the factory from CSV and Excel files. The server stores the read data in a database and processes any missing or inconsistent data appropriately. The input is CSV and Excel files, and the output is a centrally managed database.

[1007] Step 2: Enter a search query

[1008] A user accesses the search interface using a terminal (PC or mobile device) and inputs a search query in natural language. For example, they might input "Contents of the quality control meeting in September 2023." The input is the user's search query, and the output is a search request to the server.

[1009] Step 3: Parsing the search query and searching the data

[1010] The server analyzes the search query received from the user and searches for relevant information from the database. For example, if the query "Contents of the quality control meeting in September 2023" is entered, the server analyzes the query using a natural language processing algorithm and extracts the relevant meeting records from the database. The input is the search query, and the output is the search result data.

[1011] Step 4: Summarizing and displaying search results

[1012] The server summarizes the extracted search results and sends them to the terminal. The search results are displayed in summary form on the user's terminal screen. For example, the title and main topics of a particular meeting record are displayed as a summary. The input is the search result data, and the output is the summarized search results.

[1013] Step 5: Obtain and display detailed information

[1014] The user selects a specific item from the summarized search results and requests detailed information. The server retrieves the detailed information for the selected item from the database and sends it to the terminal. For example, if the user clicks on the detailed information for a meeting record, all statements and decisions made during the meeting are displayed as details. The input is the user's request for detailed information, and the output is the detailed information.

[1015] Step 6: Convert audio data to text

[1016] The server receives the audio data of the meeting or conference and converts it into text. It uses the speech_recognition library to analyze the audio data and convert it into text data. The input is the audio data and the output is the converted text data.

[1017] Step 7: Save the text data

[1018] The server indexes the converted text data in a database and stores it. For example, the audio recording of a quality control meeting is converted into text and the content is stored in a database. The input is text data, and the output is an indexed database.

[1019] Step 8: Perform a natural language search

[1020] The user then uses the device to perform a natural language search to quickly obtain the desired information. For example, if the user searches for "quality control meeting contents in September 2023," relevant meeting records are displayed as search results. The input is the user's search query, and the output is the relevant search results.

[1021] These processing steps enable centralized management of important information within the factory, rapid search, and intuitive search in natural language.

[1022] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1023] This invention is a system that centrally manages in-house information and can quickly search for information regardless of changes in personnel or the passage of time. By combining this system with an emotion engine, it becomes possible to provide information flexibly according to the user's emotional state.

[1024] 1. Information intake

[1025] First, various information (executive appointment history, meeting records, company information, employee information, organizational chart, patent information, trademark information, service history, event information) is collected on the server. The server reads this information from CSV files, Excel files, etc. and stores it in a centrally managed database. Any missing data is also processed to ensure data integrity.

[1026] Example: The server reads "Executive Appointment History.csv" and converts information such as executive names, appointment dates, and positions into a data frame (e.g., pandas).

[1027] 2. Information Search

[1028] Users access the search interface using a terminal (PC or mobile device) and enter the information they want to search for in natural language. The search interface is linked to an emotion engine that analyzes emotions from the user's input and usage status. The emotion engine analyzes emotions from the user's input and adjusts the search results.

[1029] Example: A user types in "2021 board meeting minutes," submits that search query, and the server searches the database for relevant information.

[1030] 3. Emotion Engine Analysis

[1031] The emotion engine analyzes user input and usage patterns to determine their emotions. Based on this analysis, it changes the display format and order of search results. It provides quick information when the user is in a hurry, and displays more details when more information is needed.

[1032] For example, if the emotion engine recognizes "impatience" from user input, it will prioritize and display the most relevant information, allowing the user to quickly obtain information.

[1033] 4. Summarizing and displaying search results

[1034] The server generates data to summarize and display the search results. Based on the analysis results of the emotion engine, it determines the summary format and display order of the search results. The server then sends the summarized search results to the terminal, which then displays them to the user.

[1035] Example: The server searches the executive appointment history database and extracts information related to "Yamada Taro." As a result, the name, title, appointment date, etc. of the relevant executive are displayed in summary form. If the emotion engine detects "doubt," it provides more detailed information.

[1036] 5. Viewing detailed information

[1037] When a user selects a specific item from the summarized search results, the device sends a request to the server to obtain detailed information about that item. The server retrieves the detailed information about the selected item from the database and sends it to the device. The emotion engine also monitors the user's emotions as they check the detailed information and provides additional information as needed.

[1038] Example: When a user clicks on the details of "Yamada Taro," detailed career information for that executive (such as date of appointment, position, department, etc.) is displayed.

[1039] 6. Audio to Text Conversion

[1040] If the conference record is provided as audio data, the server converts the audio data into text, which is then stored in the database for search and display.

[1041] Example: Converting recordings of board meetings into text and storing the content in a database so that it can be searched in natural language.

[1042] 7. Natural Language Search

[1043] The search interface allows users to input natural sentences and keywords, allowing the system to understand their intent and search for relevant information. This allows the system to analyze not only specific keywords but also detailed sentences to identify relevant information.

[1044] Example: When a user enters a query for "all company events held in 2020," the server parses the natural language query and searches the database for relevant event information.

[1045] This system can flexibly provide information according to the user's emotional state. This allows users to quickly obtain the information they need, improving work efficiency. In addition, by combining it with an emotion engine, the user experience is improved and the stress of information search is reduced.

[1046] The processing flow will be explained below.

[1047] Step 1:

[1048] Information capture

[1049] The server loads information such as executive appointments and meeting records from external data sources such as CSV and Excel files, including the automatic loading of regularly updated data.

[1050] Example: The server reads "Executive Appointment History.csv" and converts information such as executive names, appointment dates, and positions into a data frame (e.g., pandas).

[1051] Step 2:

[1052] Data Preprocessing

[1053] The server preprocesses the data it loads, which includes imputing missing values, converting data types, and removing unnecessary data.

[1054] For example, the server will impute missing values ​​with 'N / A' and convert date fields to the appropriate data type (e.g. datetime).

[1055] Step 3:

[1056] Storage in the database

[1057] The server stores the preprocessed data in a centralized database, thereby realizing integrated management of information.

[1058] Example: The server inserts executive history data into an SQL database and creates the appropriate indexes.

[1059] Step 4:

[1060] Entering a query through the search interface

[1061] The user accesses the search interface on the terminal and inputs the information they want to search for in natural language.

[1062] Example: User enters "2021 Board Meeting Minutes."

[1063] Step 5:

[1064] Submitting a search query

[1065] The terminal transmits the input search query to the server, which then receives the query and begins analyzing it.

[1066] Example: A device sends a query for "2021 board meeting minutes" as an HTTP request to a server.

[1067] Step 6:

[1068] Database search

[1069] The server analyzes the incoming search query and searches a centralized database, sorting the results by relevance.

[1070] Example: The server uses an SQL query to extract data corresponding to "2021 Board of Directors Meeting Minutes."

[1071] Step 7:

[1072] Emotion Engine Analysis

[1073] The server uses an emotion engine to analyze emotions from user input and usage, and adjusts search results based on the user's emotional state.

[1074] Example: If the server detects the emotion "anxiety," it will prioritize displaying the most relevant information.

[1075] Step 8:

[1076] Search result summary

[1077] The server generates data to summarize and display search results, allowing users to see concise information.

[1078] Example: The server extracts the most important fields from the search results (date, time, topic, participants, etc.) and presents them in a summarized format.

[1079] Step 9:

[1080] Displaying search results

[1081] The server sends the summarized search results to the terminal, which then displays them to the user, who can then check the list of search results.

[1082] Example: Display a list of summarized board meeting minutes in a device browser.

[1083] Step 10:

[1084] Request more information

[1085] When the user selects an item for which they wish to check details, the terminal sends a request to the server to obtain detailed information about that item.

[1086] Example: When a user clicks on a particular board meeting record, a request is sent to the server to get more information about it.

[1087] Step 11:

[1088] Get more information

[1089] The server retrieves detailed information about the selected item from the database and sends it to the terminal, allowing the user to view the detailed information.

[1090] Example: The server retrieves detailed information (e.g., text of speeches) of the selected board meeting record from the database and sends it to the terminal.

[1091] Step 12:

[1092] Viewing detailed information

[1093] The terminal displays the received detailed information to the user, allowing the user to check the details of the specific information.

[1094] Example: A detailed record of the selected board meeting is displayed on the terminal screen.

[1095] Step 13:

[1096] Converting audio data to text

[1097] If the conference recording is provided as audio data, the server converts the audio data into text, which is then stored in the database for search and display.

[1098] Example: The server uses speech recognition technology to convert the audio data of a meeting into text and stores the text in a database.

[1099] Step 14:

[1100] Providing natural language search

[1101] The server analyzes search queries entered in natural language and provides appropriate search results, enabling users to perform searches easily and intuitively.

[1102] Example: When a user enters a query for "all company events held in 2020," the server parses the natural language query and searches the database for relevant event information.

[1103] Step 15:

[1104] Emotional Engine Feedback

[1105] The server changes the display format and presentation order of search results depending on the user's emotional state. Furthermore, the emotion engine learns from user feedback and uses it to improve the accuracy of future search results.

[1106] For example, if the sentiment engine detects "doubt," it will prioritize more detailed search results.

[1107] Example 2

[1108] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1109] In conventional in-house information management systems, information is managed in a decentralized manner, making it difficult to quickly search for information when the person in charge changes or when time passes. Furthermore, the system is unable to provide information flexibly according to the user's emotional state, which can be stressful. Additionally, handling voice data is difficult, making it difficult to efficiently search and summarize voice data.

[1110] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for integrating and managing executive information, meeting information, company information, employee data, organizational structure, intellectual property information, trademark data, service history, and event records, a means including an interface for searching the information, a means for adjusting and displaying search results according to the user's emotional state, a means for generating and displaying summaries of the corresponding search results, and a means for displaying detailed information. This allows information to be centrally managed, enabling rapid information retrieval regardless of changes in personnel or the passage of time. Furthermore, appropriate information can be provided according to the user's emotional state, reducing user stress. Furthermore, by converting and managing audio data of meeting information into text, audio data can be efficiently searched and summarized.

[1111] "Officer information" is data relating to the appointment history and positions of officers in a company or organization.

[1112] "Meeting information" refers to data including minutes, participant lists, decisions, etc., of meetings held within a company or organization.

[1113] "Corporate information" refers to data that includes basic information such as a company overview, performance, business activities, and organizational structure.

[1114] "Employee Data" means data that includes personal information, job title, salary, work history, etc., of employees within a company or organization.

[1115] "Organizational structure" is information that indicates the arrangement of departments and divisions in a company or organization, the hierarchical structure of positions, and the like.

[1116] "Intellectual property information" refers to data relating to intellectual property such as patents, trademarks, and copyrights held by a company or organization.

[1117] "Trademark Data" means data relating to trademarks registered by a company or organization and their details.

[1118] "Service history" refers to data such as usage history of services and products provided by a company or organization, and customer feedback.

[1119] "Event records" are detailed information about events, seminars, exhibitions, etc. held by a company or organization.

[1120] "Interface" refers to the input screen or operation panel on a PC or mobile device that users use to search for information.

[1121] "Means for adjusting and displaying search results according to the user's emotional state" refers to a function that analyzes the user's emotions and adjusts the display format and order of search results based on the results.

[1122] "Means for generating and displaying summaries" refers to the function of concisely summarizing search results and displaying them in a format that is easy for the user to view.

[1123] "Means for displaying detailed information" refers to a function that displays detailed information when a user selects a specific search result.

[1124] "Means for converting voice data into text" refers to the function of recognizing voice data such as meeting records, converting it into text format, and saving it.

[1125] This invention is a system for centrally managing in-house information and for quickly and flexibly searching and displaying the information. This system is implemented as follows.

[1126] The server imports various information, such as executive information, meeting information, company information, employee data, organizational structure, intellectual property information, trademark data, service history, and event records, in the form of CSV files or Excel files, converts it into a data frame using the pandas library, and stores it in a centrally managed database. During this process, it detects and automatically supplements or corrects missing data and format inconsistencies. It also uses Google Speech-to-Text or similar speech recognition software to convert the audio data of the meeting information into text and store it in the database.

[1127] A user accesses a search interface using a PC or mobile device. The user enters a search query in natural language, which is then sent from the device to the server. For example, a query such as "2021 board meeting minutes" might be considered. The server receives the query and analyzes the user's emotional state using an emotion engine, such as the transformers library.

[1128] The emotion engine analyzes the user's emotions and adjusts the display format and order of search results based on the results. For example, if the user is feeling impatient, the most relevant information will be displayed first, providing information quickly. The emotion analysis results are also reflected in the generation of summaries of search results, which are sent to the device in an appropriate format depending on the user's emotional state.

[1129] The server searches the database, extracts the relevant information, formats it, and generates a summary. The generated summary is sent to the device in a format that is easy for the user to view, based on the analysis results of the emotion engine. When the user selects a specific item from the search results displayed in summary form, the device sends a request for detailed information to the server. The server retrieves the detailed information, sends it to the device in an appropriate format, and displays it to the user.

[1130] For example, if a user wants to find detailed information about an executive named "Yamada Taro," the user enters "details about Yamada Taro" into the search interface. This query is sent to the server, which retrieves the relevant executive information from the database and displays it as detailed information.

[1131] This system centralizes information management, making it possible to search for information quickly and effectively without being affected by changes in personnel or the passage of time.In addition, the introduction of an emotion engine enables flexible and prompt provision of information according to the user's emotions, reducing user stress.

[1132] Examples of prompts include "2021 board meeting minutes," "More information about Taro Yamada," and "All company events held in 2020." By entering these prompts, the system can quickly search for the relevant information and provide it to the user.

[1133] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1134] Step 1: Capture information

[1135] The server imports various information such as executive information, meeting information, company information, employee data, organizational structure, intellectual property information, trademark data, service history, and event records from CSV and Excel files. Specifically, it uses the pandas library to convert these files into data frames and stores them in a centrally managed database. The input is a CSV or Excel file, and the output is the data stored in the database.

[1136] Step 2: Data integrity check and completion

[1137] The server detects missing or inconsistent values ​​in the imported data and automatically completes or corrects them, for example by filling blank cells with a default value such as "unknown." The input is a data frame, and the output is the consistent data.

[1138] Step 3: Convert audio data to text

[1139] The server receives the audio data of the conference information and converts it into text using speech recognition software (e.g., Google Speech-to-Text). The converted text data is also stored in a database. The input is audio data, and the output is the converted text data.

[1140] Step 4: Finding information

[1141] A user uses a terminal to access a search interface and inputs the information they want to search for in natural language. For example, they input a query such as "2021 board meeting minutes." The terminal then sends this query to the server. The input is the user's search query, and the output is the transmission of the search query to the server.

[1142] Step 5: Analysis by Emotion Engine

[1143] The server analyzes the received search query and detects the user's emotions using an emotion engine. The emotion engine uses libraries such as the transformers library. Based on the analysis results, the server adjusts the display format and order of search results. The input is the search query, and the output is the emotion analysis results.

[1144] Step 6: Generating and formatting search results

[1145] The server retrieves search results from the database and generates a summary based on the analysis results of the emotion engine. For example, it summarizes information related to "Yamada Taro" and sends it to the terminal in the form of job title, date of appointment, etc. The input is the database search results and emotion analysis results, and the output is the summarized search results.

[1146] Step 7: Viewing search results

[1147] The terminal displays the search results in summary form received from the server to the user. The user checks the displayed information and selects detailed information as necessary. The input is the summarized search results, and the output is the display to the user.

[1148] Step 8: Request and view more information

[1149] When a user selects a specific item from the search results displayed in summary form, the terminal requests detailed information from the server. The server retrieves the detailed information from the database and sends it to the terminal. The terminal displays the detailed information to the user. The input is a request for detailed information, and the output is the display of the detailed information.

[1150] (Application example 2)

[1151] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1152] Conventional factory robot management systems have had problems such as difficulty in centrally managing robot operation history and maintenance records, and difficulty for operators in quickly obtaining the information they need. Furthermore, information search does not provide flexible information according to the user's emotional state, which means operators cannot obtain appropriate information in emergencies or during normal times, resulting in reduced work efficiency.

[1153] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for integrating and managing executive appointment histories, meeting records, company information, employee information, organizational charts, patent information, trademark information, service history, and event information; means including a search interface for searching for the information; means for summarizing and displaying search results; means for displaying detailed information of relevant search results; means for adjusting the search results according to the user's emotional state using an emotion engine; means for integrating and managing the operation history and maintenance records of factory robots in a database; and means for searching for related information based on a query in the user's natural language. This enables the user to flexibly and quickly obtain necessary information according to their emotional state, thereby achieving efficient operation of factory robots.

[1154] "Officer appointment history" refers to historical information on the appointment dates and positions of company officers.

[1155] "Meeting records" refers to recorded information such as minutes, audio data, and attendee information related to meetings within a company.

[1156] "Company information" refers to basic information about a company, such as the date of establishment, the name of the representative, and the main business activities.

[1157] "Employee information" refers to individual information about employees within a company, such as name, department, position, and start date.

[1158] An "organizational chart" is a diagram that shows the hierarchical internal structure of a company, visually representing the relationships between departments and positions.

[1159] "Patent information" refers to information about patents held by a company, including, for example, patent number, invention name, application date, registration date, etc.

[1160] "Trademark information" refers to information about trademarks held by a company, including, for example, trademark registration number, trademark name, registration date, etc.

[1161] "Service history" refers to the performance and history information of various services provided by a company, such as the date of service execution, customer information, and service content.

[1162] "Event information" refers to information about an event held by a company, such as the event date, participant list, and event content.

[1163] "Search interface" refers to the UI (user interface) that a user uses to search for specific information.

[1164] An "emotion engine" is a system that analyzes a user's emotional state (e.g., urgency, impatience, doubt, etc.) and adjusts the information provided based on that.

[1165] "Operation history" refers to the operating status and history information of a factory robot, such as work content, operating time, and error records.

[1166] "Maintenance records" refers to historical information about maintenance performed on a factory robot, such as inspection dates, repair details, and replacement parts.

[1167] A "natural language query" refers to a search request written in everyday language or sentence format, and refers to a technology that analyzes the entire sentence, not just specific keywords, to search for related information.

[1168] This invention builds a system that centrally manages the operation history and maintenance records of factory robots and is equipped with an emotion engine to provide flexible information according to the user's emotional state.

[1169] The server creates a database to integrate and manage executive appointment histories, meeting records, company information, employee information, organizational charts, patent information, trademark information, service history, and event information. This information is read from CSV files and Excel files, converted into a data frame (e.g., pandas), and stored. The server maintains data integrity and takes appropriate action if there is missing data.

[1170] Users access the search interface using a terminal (PC or mobile device) and enter search queries in natural language. To enable natural language searches, the server employs Natural Language Processing (NLP) technology. The interface analyzes the text entered by the user and performs semantic searches.

[1171] The search interface is connected to an emotion engine, which analyzes emotions from user input and usage and adjusts the display format and presentation order of search results. For example, if a user enters the query "Please tell me the latest maintenance records. I'm in a hurry," the emotion engine will determine the user's emotional state as "urgent" and prioritize displaying the most relevant and important information.

[1172] The server integrates and manages the operating history and maintenance records of factory robots in a database, and generates data for summarizing and displaying search results. Based on the analysis results of the emotion engine, it determines the summary format and display order of the search results. When the search results are displayed and the user requests more information, the terminal sends the request to the server, which then retrieves the information from the database and sends it to the terminal.

[1173] Additionally, audio data such as meeting recordings can be converted to text and stored in a database, enabling comprehensive searches with natural language queries. This process is performed using speech recognition software.

[1174] The hardware and software used specifically include the Python programming language, the pandas library, the TextBlob library, and scikit-learn, etc. Using this system, users can quickly obtain the information they need based on their emotional state, enabling the efficient operation of factory robots.

[1175] A specific example query is the prompt "Please tell me the latest maintenance records. I'm in a hurry." In this example, the emotion engine analyzes the user's emotion of "I'm in a hurry" and quickly displays the most important maintenance records.

[1176] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1177] Step 1:

[1178] The server reads executive appointment history, meeting records, company information, employee information, organizational charts, patent information, trademark information, service history, and event information from various CSV and Excel files, converts them into data frames (such as pandas), and stores them in a database. This enables integrated data management. The input is various information files, and the output is an integrated database.

[1179] Step 2:

[1180] A user accesses the search interface using a terminal (PC or mobile device) and inputs a search query in natural language, for example, by entering a prompt sentence such as "Please tell me the latest maintenance records. I'm very urgent." The input is the user's query text, which is passed to the search interface.

[1181] Step 3:

[1182] The search interface analyzes the user's input text using natural language processing (NLP) techniques. In this process, the input query text is broken down into tokens and semantic analysis is performed. The input is the query text, and the output is the analysis results (related keywords and meanings).

[1183] Step 4:

[1184] The emotion engine analyzes the user's emotional state from the input text. This analysis is performed using the TextBlob library, etc. For example, from the text "I'm in a hurry," the user's emotional state is determined to be "I'm in a hurry." The input is the user's query text, and the output is the emotional state.

[1185] Step 5:

[1186] The server searches the database based on the analysis results and the emotional state, extracting relevant information. During this process, it summarizes the search results and prioritizes the information according to the emotional state. The input is the analysis results and the emotional state, and the output is the summarized search results.

[1187] Step 6:

[1188] The terminal displays the summarized search results sent from the server in the most appropriate format based on the user's emotional state. The input is the summarized search results, and the output is the information displayed on the user's terminal screen.

[1189] Step 7:

[1190] When a user selects a specific item from the search results displayed in summary form, the terminal sends a request to the server to obtain detailed information about that item. The input is the item selected by the user, and the output is a request to obtain detailed information.

[1191] Step 8:

[1192] The server retrieves detailed information about the selected item from the database and sends it back to the terminal. The input is a request to retrieve detailed information, and the output is the detailed information.

[1193] Step 9:

[1194] The terminal displays the detailed information sent from the server to the user. This display is also adjusted based on the emotional state. The input is the detailed information, and the output is the detailed information displayed on the user's terminal screen.

[1195] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1196] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1197] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1198] [Fourth embodiment]

[1199] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1200] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1201] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1202] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1203] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1204] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1205] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1206] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1207] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1208] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1209] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1210] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1211] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1212] This invention provides a system that centrally manages in-house information and enables quick information retrieval regardless of changes in personnel or the passage of time. This system is composed of a server, terminals, and users, each playing their own role.

[1213] 1. Information intake

[1214] First, various information (executive appointment history, meeting records, company information, employee information, organizational chart, patent information, trademark information, service history, event information) is collected on the server. The server reads this information from CSV files, Excel files, etc. and stores it in a centrally managed database. Any missing data is also processed to ensure data integrity.

[1215] Example: Data on the history of executive appointments is read from a CSV file, and after necessary preprocessing, it is stored in the database. This creates a centrally managed database.

[1216] 2. Information Search

[1217] Users access the search interface using a terminal (PC or mobile device). The search interface supports searches in natural language, allowing users to intuitively enter search keywords. The entered search keywords are sent from the terminal to the server.

[1218] Example: When a user types "Yamada Taro executive history" into a search box, the search query is submitted and the server searches the database for relevant information.

[1219] 3. Summary display of search results

[1220] The server searches the database based on the search query and extracts relevant results. The extracted results are processed in summary form and sent to the terminal, where the user can view the summary of the search results on the terminal screen.

[1221] Example: The server searches the executive history database and extracts information related to "Yamada Taro." As a result, the name, title, and date of appointment of the relevant executive are displayed in summary form.

[1222] 4. Viewing detailed information

[1223] When a user selects a specific item from the summarized search results, the terminal sends a request to the server to obtain detailed information about that item. The server then retrieves the detailed information about the selected item from the database and sends it to the terminal.

[1224] Example: When a user clicks on the details of "Yamada Taro," detailed career information for that executive (such as date of appointment, position, department, etc.) is displayed.

[1225] 5. Converting Audio Data to Text

[1226] If there is audio data from the conference recording, the server converts the audio data into text, which is then stored in the database for search and display.

[1227] Example: Converting recordings of board meetings into text and storing the content in a database so that it can be searched in natural language.

[1228] 6. Natural Language Search

[1229] The search interface provides users with the ability to enter queries in natural language, allowing for analysis of specific keywords as well as detailed sentences to identify relevant information.

[1230] Example: If a user enters "meeting notes about new service announcements in 2019," the server will parse the query and search for new services announced in 2019 and related meeting notes.

[1231] This system allows for the fast and efficient management and search of various internal company information. Even if the person in charge changes, the centralized management of information ensures business continuity, and past information and future action items can be easily accessed. This significantly improves business efficiency and enables quick decision-making.

[1232] The processing flow will be explained below.

[1233] Step 1:

[1234] Information capture

[1235] The server loads information such as executive appointments and meeting records from external data sources such as CSV and Excel files, including the automatic loading of regularly updated data.

[1236] Example: The server reads "Executive Appointment History.csv" and converts information such as executive names, appointment dates, and positions into a data frame (e.g., pandas).

[1237] Step 2:

[1238] Data Preprocessing

[1239] The server preprocesses the data it loads, which includes imputing missing values, converting data types, and removing unnecessary data.

[1240] For example, the server will impute missing values ​​with 'N / A' and convert date fields to the appropriate data type (e.g. datetime).

[1241] Step 3:

[1242] Storage in the database

[1243] The server stores the preprocessed data in a centralized database, thereby realizing integrated management of information.

[1244] Example: The server inserts executive history data into an SQL database and creates the appropriate indexes.

[1245] Step 4:

[1246] Entering a query through the search interface

[1247] The user accesses the search interface on the terminal and inputs the information they want to search for in natural language.

[1248] Example: User enters "2021 Board Meeting Minutes."

[1249] Step 5:

[1250] Submitting a search query

[1251] The terminal transmits the input search query to the server, which then receives the query and begins analyzing it.

[1252] Example: A device sends a query for "2021 board meeting minutes" as an HTTP request to a server.

[1253] Step 6:

[1254] Database search

[1255] The server analyzes the incoming search query and searches a centralized database, sorting the results by relevance.

[1256] Example: The server uses an SQL query to extract data corresponding to "2021 Board of Directors Meeting Minutes."

[1257] Step 7:

[1258] Search result summary

[1259] The server generates data to summarize and display search results, allowing users to see concise information.

[1260] Example: The server extracts the most important fields from the search results (date, time, topic, participants, etc.) and presents them in a summarized format.

[1261] Step 8:

[1262] Displaying search results

[1263] The server sends the summarized search results to the terminal, which then displays them to the user, who can then check the list of search results.

[1264] Example: Display a list of summarized board meeting minutes in a device browser.

[1265] Step 9:

[1266] Request more information

[1267] When the user selects an item for which they wish to check details, the terminal sends a request to the server to obtain detailed information about that item.

[1268] Example: When a user clicks on a particular board meeting record, a request is sent to the server to get more information about it.

[1269] Step 10:

[1270] Get more information

[1271] The server retrieves detailed information about the selected item from the database and sends it to the terminal, allowing the user to view the detailed information.

[1272] Example: The server retrieves detailed information (e.g., text of speeches) of the selected board meeting record from the database and sends it to the terminal.

[1273] Step 11:

[1274] Viewing detailed information

[1275] The terminal displays the received detailed information to the user, allowing the user to check the details of the specific information.

[1276] Example: A detailed record of the selected board meeting is displayed on the terminal screen.

[1277] Step 12:

[1278] Converting audio data to text

[1279] If the meeting recording is provided as audio data, the server converts the audio data into text, which is then stored in a database and made searchable.

[1280] Example: The server uses speech recognition technology to convert the audio data of a meeting into text and stores the text in a database.

[1281] Step 13:

[1282] Providing natural language search

[1283] The server analyzes search queries entered in natural language and provides appropriate search results, enabling users to perform searches easily and intuitively.

[1284] Example: When a user enters a query for "all company events held in 2020," the server parses the natural language query and searches the database for relevant event information.

[1285] Example 1

[1286] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1287] Modern business activities require centralized management and search of a wide range of information, including executive history, meeting records, company information, employee information, organizational charts, intellectual property information, trademark information, service history information, and event information. However, when this information is stored in multiple different formats, it is difficult to automatically detect and correct missing information or format inconsistencies. Furthermore, there is a need for systems that can convert voice data into text and efficiently search using natural language, but current systems lack these capabilities.

[1288] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1289] In this invention, the server includes means for integrating and managing executive history information, meeting records, company information, employee information, organizational structure charts, intellectual property information, trademark information, service history information, and event information, a search interface for searching the information, means for summarizing and displaying search results, means for displaying detailed information of relevant search results, and means for automatically detecting and processing missing information and format inconsistencies in the information. This makes it possible to provide a system that efficiently manages a variety of information within a company, automatically ensures data consistency, and enables intuitive information searches using natural language.

[1290] "Officer history information" is detailed information about the positions held by company officers in the past and the periods of their service.

[1291] "Meeting records" are documents or data that record the content, minutes, participants, and decisions of meetings held within a company.

[1292] "Company information" is a set of data including basic information about a company, its management policies, performance, location, and contact information.

[1293] "Employee information" refers to data such as personal information, job title, role, salary, and career history for all employees belonging to a company.

[1294] An "organizational structure chart" is a diagram that visually shows the relationships between departments and positions within a company.

[1295] "Intellectual property information" refers to information relating to intellectual property rights owned by a company, such as patents, trademarks, copyrights, and trade secrets.

[1296] "Trademark Information" is detailed information about a company's trademarked brands, logos, slogans, etc.

[1297] "Service history information" is data that records history information and provision status regarding various services provided by a company.

[1298] "Event information" refers to information such as detailed information, dates, and participant lists of events hosted or participated in by a company.

[1299] The "search interface means" is a means for providing an interface that is used when a user searches for required information.

[1300] "Means for automatically detecting and handling missing data and format inconsistencies" refers to the system's functionality for automatically detecting missing data and format inconsistencies and handling them appropriately when they occur.

[1301] "Natural language search" is a function that understands search queries and provides appropriate data when users search for information using everyday language.

[1302] This invention provides a system that can centrally manage information within a company and search for it quickly and efficiently. The main components of the system include a server, terminals, and users.

[1303] Server Roles

[1304] The server operates a database for centrally managing various information within the company. The database management system used may be, for example, MySQL. The server reads executive history information, meeting records, company information, employee information, organizational structure charts, intellectual property information, trademark information, service history information, and event information from CSV files and Excel files, and stores this information in the database. It also has the ability to automatically detect missing information and format inconsistencies and handle them appropriately.

[1305] In addition, the server uses speech recognition software (such as Google Cloud Speech-to-Text) to convert the audio data of the meeting recording into text, and also stores the converted text data in a database.

[1306] Device Role

[1307] A terminal is a device that users use to operate the system, and can include PCs, mobile devices, etc. A search interface is provided on the terminal, allowing users to intuitively enter search keywords using natural language. These search keywords are sent from the terminal to the server.

[1308] User operations

[1309] A user uses a terminal to search for the information they need. For example, they type "Yamada Taro executive appointment history" into the search box. When this search query is sent to the server, the server searches the database for relevant information and sends the results in summary form to the terminal. The user can check the summary of the search results and click to view more detailed information.

[1310] Specific examples

[1311] For example, to search for the executive history of director Yamada Taro, a user would enter "Yamada Taro executive appointment history" into the search box on their device. The server receives this query, searches the database for relevant information, and sends Yamada Taro's position, appointment date, and other information in a summary format to the device. If the user wants to check more detailed information from the summary results, they can send a request from their device again, and the server will provide the detailed information.

[1312] As a concrete example of converting audio data from meeting records into text, the recording data of a board meeting is converted into text using Google Cloud Speech-to-Text and the content is stored in a database. This text data can later be searched as "Meeting records regarding the announcement of new services in 2019."

[1313] Prompt Sentence Examples

[1314] For example, enter the following prompt into the system:

[1315] "Find the conference notes for the 2019 New Service Launch."

[1316] In this way, the system can efficiently manage a wide variety of information according to user needs and support natural language searches, greatly improving business efficiency.

[1317] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1318] Step 1: Capture information

[1319] Input: CSV and Excel files containing executive history information, meeting notes, company information, employee information, organizational charts, intellectual property information, trademark information, service history information, and event information.

[1320] How it works: The server reads these files and detects missing data and format inconsistencies. It automatically handles any missing data and formats that are mismatched.

[1321] Data processing / calculation: Standardize data formats and fill in missing data to ensure consistency.

[1322] Output: Store the organized data in a database.

[1323] Specific operation: Read executive history information from a CSV file, fill in missing values, and store it in a MySQL database.

[1324] Step 2: Enter search keywords and submit

[1325] Input: Keywords entered by the user into the device's search interface (e.g., "Yamada Taro executive appointment history").

[1326] How it works: A user enters keywords into the search box on their device and clicks the search button. This input is sent from the device to the server.

[1327] Data processing / calculation: None in particular (sending keywords).

[1328] Output: Sending search keywords from the terminal to the server.

[1329] Specific operation: The user enters "Yamada Taro executive appointment history" and clicks the search button.

[1330] Step 3: Processing the search query and extracting search results

[1331] Input: Search keywords sent from the device (e.g., "Yamada Taro executive appointment history").

[1332] Operation: The server searches the database based on the received search keywords and extracts relevant information.

[1333] Data processing / calculation: Generate database queries and extract relevant data.

[1334] Output: Extraction of relevant search results.

[1335] Specific operation: The server searches and extracts executive history information related to "Yamada Taro" from the database.

[1336] Step 4: Summary of search results

[1337] Input: Search results extracted by the server (e.g., executive history information for "Yamada Taro").

[1338] Operation: The server sends the extracted results in summary form to the terminal.

[1339] Data processing / computation: summarizing and formatting information.

[1340] Output: Sends summarized search results to the terminal.

[1341] Specific operation: The server summarizes the job title and start date for Yamada Taro and sends it to the terminal.

[1342] Step 5: Request and view more information

[1343] Input: The item the user selects from the summary results (e.g., detailed information about "Yamada Taro").

[1344] How it works: When a user clicks on a specific item, the device sends a request to the server to get more information. The server retrieves the details from the database and sends them to the device.

[1345] Data processing / calculation: Obtain detailed information from the database.

[1346] Output: Send detailed information to the terminal for display.

[1347] Specific operation: The user clicks on the detailed information link for "Yamada Taro," and the server retrieves and displays the department he holds and his specific job history.

[1348] Step 6: Convert audio data to text

[1349] Input: Audio data from a meeting recording.

[1350] How it works: The server converts the audio data into text using Google Cloud Speech-to-Text.

[1351] Data processing / calculation: Converts voice data into text data.

[1352] Output: The converted text data.

[1353] Specific operation: The recording data of the board meeting is converted into text using Google Cloud Speech-to-Text and saved in the database as "Minutes of the board meeting held on January 1, 2019."

[1354] Step 7: Perform a natural language search

[1355] Input: A natural language query entered by a user into a search interface (e.g., "Conference notes on new service announcements in 2019").

[1356] How it works: The server uses natural language processing tools (such as NLTK) to parse the search query and search a database for relevant information.

[1357] Data processing / calculation: Search query analysis and query generation.

[1358] Output: Send relevant search results to your device.

[1359] Specific operation: The server analyzes the "Meeting records regarding the announcement of new services in 2019" using a natural language processing tool, searches for the relevant meeting records, and sends them to the terminal.

[1360] (Application example 1)

[1361] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1362] In conventional information management systems, important information such as factory manufacturing information, machine maintenance history, and work instructions is stored in a distributed manner, making it difficult to quickly search and centrally manage information. Furthermore, when managing the contents of meetings and discussions as voice data, there was a lack of systems that could convert the voice data into text and make it easily searchable. Furthermore, there was an insufficient system that supported searches in natural language, making intuitive searches difficult.

[1363] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1364] In this invention, the server includes means for integrating and managing executive appointment histories, meeting records, company information, employee information, organizational charts, patent information, trademark information, service history, and event information, means including a search interface for searching the information, means for summarizing and displaying search results, means for displaying detailed information of relevant search results, means for converting audio data of meeting records and meetings into text, and means for indexing and storing the text data in a database. This enables unified management of information, rapid search, and intuitive search in natural language.

[1365] "Officer appointment history" is data that records and manages information such as the appointment date, position, and term of office of an officer.

[1366] A "meeting record" is a document that records the contents of a meeting, speakers, remarks, agenda, conclusions, etc., and can be referenced later.

[1367] "Company information" refers to data including basic company information, date of establishment, location, business activities, capital, and board of directors composition.

[1368] "Employee information" refers to data that records and manages personal and work-related information such as an employee's name, department, position, date of joining, and contact information.

[1369] An "organizational chart" is a visual representation of the structure of a company or organization, showing each department and their relationships.

[1370] "Patent information" is data that records detailed information about patents, such as patent number, patent name, inventor, application date, registration date, and patent owner.

[1371] "Trademark information" is data that records detailed information such as the trademark name, trademark registration number, registration date, and trademark owner.

[1372] "Service history" is data that records information such as the type of service provided by a company, the start date of service, the person in charge, the service content, and the customer.

[1373] "Event information" is data that records information such as the name, date, time, location, participants, and content of events that a company has hosted or participated in.

[1374] A "search interface" is a portion of software that provides input functionality that a user uses to quickly search for specific information.

[1375] A "means for converting voice data to text" is a technique or device for taking voice input and converting it into textual data.

[1376] A "means for indexing and storing data in a database" is a technique or method for organizing and storing data in a database in an easily searchable format.

[1377] "Natural language search" is a search function that analyzes search queries entered by users in natural language and identifies and presents relevant information.

[1378] The present invention provides a smart manufacturing information management system that is installed on robots used in factories. This system centrally manages important information within the factory and supports searches using natural language. A specific embodiment of the system will be described below.

[1379] 1. System configuration

[1380] The system is composed of the roles of a server, a terminal, and a user.

[1381] server

[1382] The server maintains a database that centrally manages information collected within the factory, including executive appointment histories, meeting records, company information, employee information, organizational charts, patent information, trademark information, service history, and event information. The server also reads this information from CSV files and Excel files and stores it in the database. The server processes the voice data using software to convert the voice data to text (for example, the speech_recognition library).

[1383] Terminal

[1384] Users access the search interface using a terminal such as a PC or mobile device. The terminal accepts search query input and sends it to the server. The terminal also has the function of displaying search results in summary form and in detailed information.

[1385] 2. Information intake

[1386] The server reads various information used within the factory from CSV and Excel files, including information on newly appointed executives and the latest meeting records. The read data is stored in a database, and any missing or inconsistent data is dealt with appropriately.

[1387] 3. Search function

[1388] Users perform natural language searches through their devices. For example, they can enter "contents of the quality control meeting in September 2023," and the query is sent to the server. The server analyzes the query and searches the database for relevant information. Search results are displayed on the device in summary form, and detailed information is displayed when the user selects a specific item.

[1389] 4. Converting Audio Data to Text

[1390] The server converts recordings of meetings and discussions into text. For example, if you upload a recording of a quality control meeting, it will be converted into text using the speech_recognition library and the content will be stored in a database.

[1391] 5. Specific Examples

[1392] For example, if a quality control meeting is held at a factory and the audio data is recorded, the recording can be uploaded to the smart manufacturing information management system and the meeting content converted into text, and then by searching for "quality control meeting content in September 2023," the relevant meeting records can be easily retrieved.

[1393] Prompt Sentence Examples

[1394] "Upload the audio file, convert the meeting record to text, and save it in the database. Then, search for 'Contents of the Quality Control Meeting in September 2023' and display it."

[1395] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1396] Step 1: Capture information

[1397] The server reads executive appointment histories, meeting records, company information, employee information, organizational charts, patent information, trademark information, service history, and event information collected within the factory from CSV and Excel files. The server stores the read data in a database and processes any missing or inconsistent data appropriately. The input is CSV and Excel files, and the output is a centrally managed database.

[1398] Step 2: Enter a search query

[1399] A user accesses the search interface using a terminal (PC or mobile device) and inputs a search query in natural language. For example, they might input "Contents of the quality control meeting in September 2023." The input is the user's search query, and the output is a search request to the server.

[1400] Step 3: Parsing the search query and searching the data

[1401] The server analyzes the search query received from the user and searches for relevant information from the database. For example, if the query "Contents of the quality control meeting in September 2023" is entered, the server analyzes the query using a natural language processing algorithm and extracts the relevant meeting records from the database. The input is the search query, and the output is the search result data.

[1402] Step 4: Summarizing and displaying search results

[1403] The server summarizes the extracted search results and sends them to the terminal. The search results are displayed in summary form on the user's terminal screen. For example, the title and main topics of a particular meeting record are displayed as a summary. The input is the search result data, and the output is the summarized search results.

[1404] Step 5: Obtain and display detailed information

[1405] The user selects a specific item from the summarized search results and requests detailed information. The server retrieves the detailed information for the selected item from the database and sends it to the terminal. For example, if the user clicks on the detailed information for a meeting record, all statements and decisions made during the meeting are displayed as details. The input is the user's request for detailed information, and the output is the detailed information.

[1406] Step 6: Convert audio data to text

[1407] The server receives the audio data of the meeting or conference and converts it into text. It uses the speech_recognition library to analyze the audio data and convert it into text data. The input is the audio data and the output is the converted text data.

[1408] Step 7: Save the text data

[1409] The server indexes the converted text data in a database and stores it. For example, the audio recording of a quality control meeting is converted into text and the content is stored in a database. The input is text data, and the output is an indexed database.

[1410] Step 8: Perform a natural language search

[1411] The user then uses the device to perform a natural language search to quickly obtain the desired information. For example, if the user searches for "quality control meeting contents in September 2023," relevant meeting records are displayed as search results. The input is the user's search query, and the output is the relevant search results.

[1412] These processing steps enable centralized management of important information within the factory, rapid search, and intuitive search in natural language.

[1413] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1414] This invention is a system that centrally manages in-house information and can quickly search for information regardless of changes in personnel or the passage of time. By combining this system with an emotion engine, it becomes possible to provide information flexibly according to the user's emotional state.

[1415] 1. Information intake

[1416] First, various information (executive appointment history, meeting records, company information, employee information, organizational chart, patent information, trademark information, service history, event information) is collected on the server. The server reads this information from CSV files, Excel files, etc. and stores it in a centrally managed database. Any missing data is also processed to ensure data integrity.

[1417] Example: The server reads "Executive Appointment History.csv" and converts information such as executive names, appointment dates, and positions into a data frame (e.g., pandas).

[1418] 2. Information Search

[1419] Users access the search interface using a terminal (PC or mobile device) and enter the information they want to search for in natural language. The search interface is linked to an emotion engine that analyzes emotions from the user's input and usage status. The emotion engine analyzes emotions from the user's input and adjusts the search results.

[1420] Example: A user types in "2021 board meeting minutes," submits that search query, and the server searches the database for relevant information.

[1421] 3. Emotion Engine Analysis

[1422] The emotion engine analyzes user input and usage patterns to determine their emotions. Based on this analysis, it changes the display format and order of search results. It provides quick information when the user is in a hurry, and displays more details when more information is needed.

[1423] For example, if the emotion engine recognizes "impatience" from user input, it will prioritize and display the most relevant information, allowing the user to quickly obtain information.

[1424] 4. Summarizing and displaying search results

[1425] The server generates data to summarize and display the search results. Based on the analysis results of the emotion engine, it determines the summary format and display order of the search results. The server then sends the summarized search results to the terminal, which then displays them to the user.

[1426] Example: The server searches the executive appointment history database and extracts information related to "Yamada Taro." As a result, the name, title, appointment date, etc. of the relevant executive are displayed in summary form. If the emotion engine detects "doubt," it provides more detailed information.

[1427] 5. Viewing detailed information

[1428] When a user selects a specific item from the summarized search results, the device sends a request to the server to obtain detailed information about that item. The server retrieves the detailed information about the selected item from the database and sends it to the device. The emotion engine also monitors the user's emotions as they check the detailed information and provides additional information as needed.

[1429] Example: When a user clicks on the details of "Yamada Taro," detailed career information for that executive (such as date of appointment, position, department, etc.) is displayed.

[1430] 6. Audio to Text Conversion

[1431] If the conference record is provided as audio data, the server converts the audio data into text, which is then stored in the database for search and display.

[1432] Example: Converting recordings of board meetings into text and storing the content in a database so that it can be searched in natural language.

[1433] 7. Natural Language Search

[1434] The search interface allows users to input natural sentences and keywords, allowing the system to understand their intent and search for relevant information. This allows the system to analyze not only specific keywords but also detailed sentences to identify relevant information.

[1435] Example: When a user enters a query for "all company events held in 2020," the server parses the natural language query and searches the database for relevant event information.

[1436] This system can flexibly provide information according to the user's emotional state. This allows users to quickly obtain the information they need, improving work efficiency. In addition, by combining it with an emotion engine, the user experience is improved and the stress of information search is reduced.

[1437] The processing flow will be explained below.

[1438] Step 1:

[1439] Information capture

[1440] The server loads information such as executive appointments and meeting records from external data sources such as CSV and Excel files, including the automatic loading of regularly updated data.

[1441] Example: The server reads "Executive Appointment History.csv" and converts information such as executive names, appointment dates, and positions into a data frame (e.g., pandas).

[1442] Step 2:

[1443] Data Preprocessing

[1444] The server preprocesses the data it loads, which includes imputing missing values, converting data types, and removing unnecessary data.

[1445] For example, the server will impute missing values ​​with 'N / A' and convert date fields to the appropriate data type (e.g. datetime).

[1446] Step 3:

[1447] Storage in the database

[1448] The server stores the preprocessed data in a centralized database, thereby realizing integrated management of information.

[1449] Example: The server inserts executive history data into an SQL database and creates the appropriate indexes.

[1450] Step 4:

[1451] Entering a query through the search interface

[1452] The user accesses the search interface on the terminal and inputs the information they want to search for in natural language.

[1453] Example: User enters "2021 Board Meeting Minutes."

[1454] Step 5:

[1455] Submitting a search query

[1456] The terminal transmits the input search query to the server, which then receives the query and begins analyzing it.

[1457] Example: A device sends a query for "2021 board meeting minutes" as an HTTP request to a server.

[1458] Step 6:

[1459] Database search

[1460] The server analyzes the incoming search query and searches a centralized database, sorting the results by relevance.

[1461] Example: The server uses an SQL query to extract data corresponding to "2021 Board of Directors Meeting Minutes."

[1462] Step 7:

[1463] Emotion Engine Analysis

[1464] The server uses an emotion engine to analyze emotions from user input and usage, and adjusts search results based on the user's emotional state.

[1465] Example: If the server detects the emotion "anxiety," it will prioritize displaying the most relevant information.

[1466] Step 8:

[1467] Search result summary

[1468] The server generates data to summarize and display search results, allowing users to see concise information.

[1469] Example: The server extracts the most important fields from the search results (date, time, topic, participants, etc.) and presents them in a summarized format.

[1470] Step 9:

[1471] Displaying search results

[1472] The server sends the summarized search results to the terminal, which then displays them to the user, who can then check the list of search results.

[1473] Example: Display a list of summarized board meeting minutes in a device browser.

[1474] Step 10:

[1475] Request more information

[1476] When the user selects an item for which they wish to check details, the terminal sends a request to the server to obtain detailed information about that item.

[1477] Example: When a user clicks on a particular board meeting record, a request is sent to the server to get more information about it.

[1478] Step 11:

[1479] Get more information

[1480] The server retrieves detailed information about the selected item from the database and sends it to the terminal, allowing the user to view the detailed information.

[1481] Example: The server retrieves detailed information (e.g., text of speeches) of the selected board meeting record from the database and sends it to the terminal.

[1482] Step 12:

[1483] Viewing detailed information

[1484] The terminal displays the received detailed information to the user, allowing the user to check the details of the specific information.

[1485] Example: A detailed record of the selected board meeting is displayed on the terminal screen.

[1486] Step 13:

[1487] Converting audio data to text

[1488] If the conference recording is provided as audio data, the server converts the audio data into text, which is then stored in the database for search and display.

[1489] Example: The server uses speech recognition technology to convert the audio data of a meeting into text and stores the text in a database.

[1490] Step 14:

[1491] Providing natural language search

[1492] The server analyzes search queries entered in natural language and provides appropriate search results, enabling users to perform searches easily and intuitively.

[1493] Example: When a user enters a query for "all company events held in 2020," the server parses the natural language query and searches the database for relevant event information.

[1494] Step 15:

[1495] Emotional Engine Feedback

[1496] The server changes the display format and presentation order of search results depending on the user's emotional state. Furthermore, the emotion engine learns from user feedback and uses it to improve the accuracy of future search results.

[1497] For example, if the sentiment engine detects "doubt," it will prioritize more detailed search results.

[1498] Example 2

[1499] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1500] In conventional in-house information management systems, information is managed in a decentralized manner, making it difficult to quickly search for information when the person in charge changes or when time passes. Furthermore, the system is unable to provide information flexibly according to the user's emotional state, which can be stressful. Additionally, handling voice data is difficult, making it difficult to efficiently search and summarize voice data.

[1501] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for integrating and managing executive information, meeting information, company information, employee data, organizational structure, intellectual property information, trademark data, service history, and event records, a means including an interface for searching the information, a means for adjusting and displaying search results according to the user's emotional state, a means for generating and displaying summaries of the corresponding search results, and a means for displaying detailed information. This allows information to be centrally managed, enabling rapid information retrieval regardless of changes in personnel or the passage of time. Furthermore, appropriate information can be provided according to the user's emotional state, reducing user stress. Furthermore, by converting and managing audio data of meeting information into text, audio data can be efficiently searched and summarized.

[1502] "Officer information" is data relating to the appointment history and positions of officers in a company or organization.

[1503] "Meeting information" refers to data including minutes, participant lists, decisions, etc., of meetings held within a company or organization.

[1504] "Corporate information" refers to data that includes basic information such as a company overview, performance, business activities, and organizational structure.

[1505] "Employee Data" means data that includes personal information, job title, salary, work history, etc., of employees within a company or organization.

[1506] "Organizational structure" is information that indicates the arrangement of departments and divisions in a company or organization, the hierarchical structure of positions, and the like.

[1507] "Intellectual property information" refers to data relating to intellectual property such as patents, trademarks, and copyrights held by a company or organization.

[1508] "Trademark Data" means data relating to trademarks registered by a company or organization and their details.

[1509] "Service history" refers to data such as usage history of services and products provided by a company or organization, and customer feedback.

[1510] "Event records" are detailed information about events, seminars, exhibitions, etc. held by a company or organization.

[1511] "Interface" refers to the input screen or operation panel on a PC or mobile device that users use to search for information.

[1512] "Means for adjusting and displaying search results according to the user's emotional state" refers to a function that analyzes the user's emotions and adjusts the display format and order of search results based on the results.

[1513] "Means for generating and displaying summaries" refers to the function of concisely summarizing search results and displaying them in a format that is easy for the user to view.

[1514] "Means for displaying detailed information" refers to a function that displays detailed information when a user selects a specific search result.

[1515] "Means for converting voice data into text" refers to the function of recognizing voice data such as meeting records, converting it into text format, and saving it.

[1516] This invention is a system for centrally managing in-house information and for quickly and flexibly searching and displaying the information. This system is implemented as follows.

[1517] The server imports various information, such as executive information, meeting information, company information, employee data, organizational structure, intellectual property information, trademark data, service history, and event records, in the form of CSV files or Excel files, converts it into a data frame using the pandas library, and stores it in a centrally managed database. During this process, it detects and automatically supplements or corrects missing data and format inconsistencies. It also uses Google Speech-to-Text or similar speech recognition software to convert the audio data of the meeting information into text and store it in the database.

[1518] A user accesses a search interface using a PC or mobile device. The user enters a search query in natural language, which is then sent from the device to the server. For example, a query such as "2021 board meeting minutes" might be considered. The server receives the query and analyzes the user's emotional state using an emotion engine, such as the transformers library.

[1519] The emotion engine analyzes the user's emotions and adjusts the display format and order of search results based on the results. For example, if the user is feeling impatient, the most relevant information will be displayed first, providing information quickly. The emotion analysis results are also reflected in the generation of summaries of search results, which are sent to the device in an appropriate format depending on the user's emotional state.

[1520] The server searches the database, extracts the relevant information, formats it, and generates a summary. The generated summary is sent to the device in a format that is easy for the user to view, based on the analysis results of the emotion engine. When the user selects a specific item from the search results displayed in summary form, the device sends a request for detailed information to the server. The server retrieves the detailed information, sends it to the device in an appropriate format, and displays it to the user.

[1521] For example, if a user wants to find detailed information about an executive named "Yamada Taro," the user enters "details about Yamada Taro" into the search interface. This query is sent to the server, which retrieves the relevant executive information from the database and displays it as detailed information.

[1522] This system centralizes information management, making it possible to search for information quickly and effectively without being affected by changes in personnel or the passage of time.In addition, the introduction of an emotion engine enables flexible and prompt provision of information according to the user's emotions, reducing user stress.

[1523] Examples of prompts include "2021 board meeting minutes," "More information about Taro Yamada," and "All company events held in 2020." By entering these prompts, the system can quickly search for the relevant information and provide it to the user.

[1524] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1525] Step 1: Capture information

[1526] The server imports various information such as executive information, meeting information, company information, employee data, organizational structure, intellectual property information, trademark data, service history, and event records from CSV and Excel files. Specifically, it uses the pandas library to convert these files into data frames and stores them in a centrally managed database. The input is a CSV or Excel file, and the output is the data stored in the database.

[1527] Step 2: Data integrity check and completion

[1528] The server detects missing or inconsistent values ​​in the imported data and automatically completes or corrects them, for example by filling blank cells with a default value such as "unknown." The input is a data frame, and the output is the consistent data.

[1529] Step 3: Convert audio data to text

[1530] The server receives the audio data of the conference information and converts it into text using speech recognition software (e.g., Google Speech-to-Text). The converted text data is also stored in a database. The input is audio data, and the output is the converted text data.

[1531] Step 4: Finding information

[1532] A user uses a terminal to access a search interface and inputs the information they want to search for in natural language. For example, they input a query such as "2021 board meeting minutes." The terminal then sends this query to the server. The input is the user's search query, and the output is the transmission of the search query to the server.

[1533] Step 5: Analysis by Emotion Engine

[1534] The server analyzes the received search query and detects the user's emotions using an emotion engine. The emotion engine uses libraries such as the transformers library. Based on the analysis results, the server adjusts the display format and order of search results. The input is the search query, and the output is the emotion analysis results.

[1535] Step 6: Generating and formatting search results

[1536] The server retrieves search results from the database and generates a summary based on the analysis results of the emotion engine. For example, it summarizes information related to "Yamada Taro" and sends it to the terminal in the form of job title, date of appointment, etc. The input is the database search results and emotion analysis results, and the output is the summarized search results.

[1537] Step 7: Viewing search results

[1538] The terminal displays the search results in summary form received from the server to the user. The user checks the displayed information and selects detailed information as necessary. The input is the summarized search results, and the output is the display to the user.

[1539] Step 8: Request and view more information

[1540] When a user selects a specific item from the search results displayed in summary form, the terminal requests detailed information from the server. The server retrieves the detailed information from the database and sends it to the terminal. The terminal displays the detailed information to the user. The input is a request for detailed information, and the output is the display of the detailed information.

[1541] (Application example 2)

[1542] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1543] Conventional factory robot management systems have had problems such as difficulty in centrally managing robot operation history and maintenance records, and difficulty for operators in quickly obtaining the information they need. Furthermore, information search does not provide flexible information according to the user's emotional state, which means operators cannot obtain appropriate information in emergencies or during normal times, resulting in reduced work efficiency.

[1544] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for integrating and managing executive appointment histories, meeting records, company information, employee information, organizational charts, patent information, trademark information, service history, and event information; means including a search interface for searching for the information; means for summarizing and displaying search results; means for displaying detailed information of relevant search results; means for adjusting the search results according to the user's emotional state using an emotion engine; means for integrating and managing the operation history and maintenance records of factory robots in a database; and means for searching for related information based on a query in the user's natural language. This enables the user to flexibly and quickly obtain necessary information according to their emotional state, thereby achieving efficient operation of factory robots.

[1545] "Officer appointment history" refers to historical information on the appointment dates and positions of company officers.

[1546] "Meeting records" refers to recorded information such as minutes, audio data, and attendee information related to meetings within a company.

[1547] "Company information" refers to basic information about a company, such as the date of establishment, the name of the representative, and the main business activities.

[1548] "Employee information" refers to individual information about employees within a company, such as name, department, position, and start date.

[1549] An "organizational chart" is a diagram that shows the hierarchical internal structure of a company, visually representing the relationships between departments and positions.

[1550] "Patent information" refers to information about patents held by a company, including, for example, patent number, invention name, application date, registration date, etc.

[1551] "Trademark information" refers to information about trademarks held by a company, including, for example, trademark registration number, trademark name, registration date, etc.

[1552] "Service history" refers to the performance and history information of various services provided by a company, such as the date of service execution, customer information, and service content.

[1553] "Event information" refers to information about an event held by a company, such as the event date, participant list, and event content.

[1554] "Search interface" refers to the UI (user interface) that a user uses to search for specific information.

[1555] An "emotion engine" is a system that analyzes a user's emotional state (e.g., urgency, impatience, doubt, etc.) and adjusts the information provided based on that.

[1556] "Operation history" refers to the operating status and history information of a factory robot, such as work content, operating time, and error records.

[1557] "Maintenance records" refers to historical information about maintenance performed on a factory robot, such as inspection dates, repair details, and replacement parts.

[1558] A "natural language query" refers to a search request written in everyday language or sentence format, and refers to a technology that analyzes the entire sentence, not just specific keywords, to search for related information.

[1559] This invention builds a system that centrally manages the operation history and maintenance records of factory robots and is equipped with an emotion engine to provide flexible information according to the user's emotional state.

[1560] The server creates a database to integrate and manage executive appointment histories, meeting records, company information, employee information, organizational charts, patent information, trademark information, service history, and event information. This information is read from CSV files and Excel files, converted into a data frame (e.g., pandas), and stored. The server maintains data integrity and takes appropriate action if there is missing data.

[1561] Users access the search interface using a terminal (PC or mobile device) and enter search queries in natural language. To enable natural language searches, the server employs Natural Language Processing (NLP) technology. The interface analyzes the text entered by the user and performs semantic searches.

[1562] The search interface is connected to an emotion engine, which analyzes emotions from user input and usage and adjusts the display format and presentation order of search results. For example, if a user enters the query "Please tell me the latest maintenance records. I'm in a hurry," the emotion engine will determine the user's emotional state as "urgent" and prioritize displaying the most relevant and important information.

[1563] The server integrates and manages the operating history and maintenance records of factory robots in a database, and generates data for summarizing and displaying search results. Based on the analysis results of the emotion engine, it determines the summary format and display order of the search results. When the search results are displayed and the user requests more information, the terminal sends the request to the server, which then retrieves the information from the database and sends it to the terminal.

[1564] Additionally, audio data such as meeting recordings can be converted to text and stored in a database, enabling comprehensive searches with natural language queries. This process is performed using speech recognition software.

[1565] The hardware and software used specifically include the Python programming language, the pandas library, the TextBlob library, and scikit-learn, etc. Using this system, users can quickly obtain the information they need based on their emotional state, enabling the efficient operation of factory robots.

[1566] A specific example query is the prompt "Please tell me the latest maintenance records. I'm in a hurry." In this example, the emotion engine analyzes the user's emotion of "I'm in a hurry" and quickly displays the most important maintenance records.

[1567] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1568] Step 1:

[1569] The server reads executive appointment history, meeting records, company information, employee information, organizational charts, patent information, trademark information, service history, and event information from various CSV and Excel files, converts them into data frames (such as pandas), and stores them in a database. This enables integrated data management. The input is various information files, and the output is an integrated database.

[1570] Step 2:

[1571] A user accesses the search interface using a terminal (PC or mobile device) and inputs a search query in natural language, for example, by entering a prompt sentence such as "Please tell me the latest maintenance records. I'm very urgent." The input is the user's query text, which is passed to the search interface.

[1572] Step 3:

[1573] The search interface analyzes the user's input text using natural language processing (NLP) techniques. In this process, the input query text is broken down into tokens and semantic analysis is performed. The input is the query text, and the output is the analysis results (related keywords and meanings).

[1574] Step 4:

[1575] The emotion engine analyzes the user's emotional state from the input text. This analysis is performed using the TextBlob library, etc. For example, from the text "I'm in a hurry," the user's emotional state is determined to be "I'm in a hurry." The input is the user's query text, and the output is the emotional state.

[1576] Step 5:

[1577] The server searches the database based on the analysis results and the emotional state, extracting relevant information. During this process, it summarizes the search results and prioritizes the information according to the emotional state. The input is the analysis results and the emotional state, and the output is the summarized search results.

[1578] Step 6:

[1579] The terminal displays the summarized search results sent from the server in the most appropriate format based on the user's emotional state. The input is the summarized search results, and the output is the information displayed on the user's terminal screen.

[1580] Step 7:

[1581] When a user selects a specific item from the search results displayed in summary form, the terminal sends a request to the server to obtain detailed information about that item. The input is the item selected by the user, and the output is a request to obtain detailed information.

[1582] Step 8:

[1583] The server retrieves detailed information about the selected item from the database and sends it back to the terminal. The input is a request to retrieve detailed information, and the output is the detailed information.

[1584] Step 9:

[1585] The terminal displays the detailed information sent from the server to the user. This display is also adjusted based on the emotional state. The input is the detailed information, and the output is the detailed information displayed on the user's terminal screen.

[1586] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1587] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1588] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1589] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1590] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1591] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1592] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1593] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1594] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1595] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1596] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1597] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1598] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1599] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1600] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1601] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1602] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1603] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1604] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1605] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1606] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1607] The following is further disclosed regarding the above embodiment.

[1608] (Claim 1)

[1609] A means for integrating and managing executive appointment history, meeting records, company information, employee information, organizational charts, patent information, trademark information, service history, and event information;

[1610] means including a search interface for searching said information;

[1611] a means for summarizing and displaying search results;

[1612] a means for displaying detailed information about the search results;

[1613] A system including:

[1614] (Claim 2)

[1615] 10. The system of claim 1, further comprising means for converting audio data of the conference recording into text.

[1616] (Claim 3)

[1617] The system of claim 1 , wherein the search interface includes means for enabling natural language searches.

[1618] "Example 1"

[1619] (Claim 1)

[1620] A means for integrating and managing executive history information, meeting records, company information, employee information, organizational structure charts, intellectual property information, trademark information, service history information, and event information;

[1621] a search interface means for searching the information;

[1622] a means for summarizing and displaying search results;

[1623] a means for displaying detailed information about the search results;

[1624] means for automatically detecting and handling missing or format inconsistent information;

[1625] A system including:

[1626] (Claim 2)

[1627] 10. The system of claim 1, further comprising means for converting audio data of the conference recording into text.

[1628] (Claim 3)

[1629] The system of claim 1 , wherein the search interface includes means for enabling natural language searches.

[1630] "Application Example 1"

[1631] (Claim 1)

[1632] A means for integrating and managing executive appointment history, meeting records, company information, employee information, organizational charts, patent information, trademark information, service history, and event information;

[1633] means including a search interface for searching said information;

[1634] a means for summarizing and displaying search results;

[1635] a means for displaying detailed information about the search results;

[1636] A means to convert meeting records and audio data into text,

[1637] a means for indexing and storing the text data in a database;

[1638] A system including:

[1639] (Claim 2)

[1640] 10. The system of claim 1, further comprising means for voice input of conversations or instructions and converting the same into text.

[1641] (Claim 3)

[1642] The system of claim 1 , wherein the search interface includes means for enabling natural language searches.

[1643] "Example 2: Combining Emotion Engines"

[1644] (Claim 1)

[1645] A means for integrating and managing executive information, meeting information, company information, employee data, organizational structure, intellectual property information, trademark data, service history, and event records;

[1646] means including an interface...

Claims

1. A means for integrating and managing executive appointment history, meeting records, company information, employee information, organizational charts, patent information, trademark information, service history, and event information; means including a search interface for searching said information; a means for summarizing and displaying search results; a means for displaying detailed information about the search results; A system including:

2. 2. The system of claim 1, further comprising means for converting audio data of the conference recording into text.

3. The system of claim 1 , wherein the search interface includes means for enabling natural language searching.

Images