System

Abstract

An object of the system according to the embodiment is to convert a voice into a text, summarize and record the text, and propose an action based on the record.SOLUTION: A system according to an embodiment includes a speech recognizer, a summary generator, a record manager, and an action suggester. The voice recognizer converts a user's voice into a text using a voice recognizing AI. The summary generator summarizes the text generated by the speech recognizer. The record management unit stores the summary generated by the summary generation unit as a record. The action suggestion unit suggests an action based on the record stored by the record management unit.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional technology has the drawback of being inefficient because the process of converting speech into text, summarizing and recording that text, and then suggesting actions based on that recording is not automated.

[0005] The system according to the embodiment aims to convert speech to text, summarize and record the text, and then suggest actions based on the recording. [Means for solving the problem]

[0006] The system according to the embodiment includes a speech recognition unit, a summary generation unit, a record management unit, and an action suggestion unit. The speech recognition unit converts a user's speech into text using a speech recognition AI. The summary generation unit summarizes the text generated by the speech recognition unit. The record management unit stores the summary generated by the summary generation unit as a record. The action suggestion unit suggests an action based on the record stored by the record management unit. [Effects of the Invention]

[0007] An embodiment of the system can convert speech to text, summarize and record the text, and suggest actions based on the recording. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10]1 shows an emotion map onto which multiple emotions are mapped. DETAILED DESCRIPTION OF THE INVENTION

[0009] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0010] First, the terms used in the following description will be explained.

[0011] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, the processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (Tensor Processing Unit).

[0012] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0013] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0014] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), and Bluetooth (registered trademark).

[0015] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0016] [First embodiment] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0017] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0018] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0019] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0020] The reception device 38 includes a touch panel 38A and a microphone 38B, and receives user input. The touch panel 38A detects contact with a pointer (for example, a pen or a finger) to receive user input by the touch of the pointer. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 (see FIG. 2) acquires the data indicating the user input.

[0021] Output device 40 includes a display 40A and a speaker 40B, and presents data to a user by outputting the data in a form of expression that the user can perceive (e.g., audio and / or text). Display 40A displays visible information such as text and images in accordance with instructions from processor 46. Speaker 40B outputs audio in accordance with instructions from processor 46. Camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0022] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0023] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0024] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0025] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate a user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotion, including estimation and prediction of the user's emotion, but is not limited to these examples. Furthermore, the estimation and prediction of emotion also includes, for example, emotion analysis.

[0026] In the smart device 14, the specific processing is performed by the processor 46. The storage 50 stores a specific processing program 60. The specific processing program 60 is used together with the specific processing program 56 by the data processing system 10. The processor 46 reads the specific processing program 60 from the storage 50 and executes the read specific processing program 60 on the RAM 48. The specific processing is realized by the processor 46 operating as the control unit 46A in accordance with the specific processing program 60 executed on the RAM 48. Note that the smart device 14 has a data generation model and an emotion identification model similar to the data generation model 58 and the emotion identification model 59, and can also perform processing similar to that of the specific processing unit 290 using these models.

[0027] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device (e.g., a generation server) may have the data generation model 58. In this case, the data processing device 12 obtains a processing result (prediction result, etc.) using the data generation model 58 by communicating with the server device having the data generation model 58. Furthermore, the data processing device 12 may be a server device, or may be a terminal device owned by a user (e.g., a mobile phone, a robot, a home appliance, etc.). Next, an example of processing by the data processing system 10 according to the first embodiment will be described.

[0028] (Example 1) An application according to an embodiment of the present invention combines speech recognition AI and generative AI. This application is a system that automatically summarizes what a user says, stores it as a record, and suggests actions based on the recorded content. This allows the application to efficiently record the user's voice and suggest subsequent actions.

[0029] An application according to an embodiment includes a speech recognition unit, a summary generation unit, a record management unit, and an action suggestion unit. The speech recognition unit converts a user's speech into text. For example, the speech recognition unit converts speech into text with high accuracy using deep learning. The speech recognition unit can also analyze speech using a hidden Markov model (HMM). The speech recognition unit can also convert speech into text using phoneme recognition technology. The summary generation unit summarizes the text generated by the speech recognition unit. For example, the summary generation unit summarizes the text using a generation AI (e.g., an LLM). The summary generation unit can also extract and summarize important parts of a sentence. The summary generation unit can also generate a summary by inputting a prompt containing instructions on what the user wants the generation AI to do. The record management unit stores the summary generated by the summary generation unit as a record. For example, the record management unit can store the summary in a database. The record management unit can also store the summary in cloud storage. The record management unit can also create an index to enable users to search for records. The action suggestion unit suggests actions based on the records stored by the record management unit. For example, the action suggestion unit suggests an agenda for the next meeting using the generation AI. The action suggestion unit can also generate a list of tasks. The action suggestion unit can also suggest actions by inputting a prompt containing instructions on what the user wants the generation AI to do. This allows the application according to the embodiment to efficiently record the user's voice and suggest subsequent actions. For example, the application can accurately record the contents of a meeting and efficiently prepare for the next meeting. The application can also easily manage daily notes and quickly find necessary information.

[0030] The speech recognition unit can identify multiple speakers and generate a different text file for each speaker. For example, the speech recognition unit may identify multiple speakers during a conference and save each speaker's utterances in a separate text file. For example, speaker A's utterances may be saved in file A and speaker B's utterances in file B. The speech recognition unit may also identify multiple speakers during an interview and save each speaker's utterances in a separate text file. For example, the interviewer's utterances may be saved in file A and the interviewee's utterances in file B. The speech recognition unit may also identify multiple speakers during a discussion and save each speaker's utterances in a separate text file. For example, speaker X's utterances may be saved in file X and speaker Y's utterances in file Y. This allows the utterances of multiple speakers to be managed individually.

[0031] The speech recognition unit can automatically filter background sounds to generate text with less noise. For example, the speech recognition unit automatically filters background sounds during a meeting to generate text with less noise. For example, it removes the sound of an air conditioner or keyboard typing sounds. The speech recognition unit also automatically filters background sounds during recording while away from home to generate text with less noise. For example, it removes the sound of cars or wind. The speech recognition unit also automatically filters background sounds during a conference call to generate text with less noise. For example, it removes other people's voices and other noises. This makes it possible to generate text with less noise.

[0032] The speech recognition unit can automatically translate different languages ​​and generate text in multiple languages ​​simultaneously. For example, the speech recognition unit translates statements made during a meeting in real time and generates text in multiple languages ​​simultaneously. For example, Japanese statements are translated into English and Chinese and converted into text. The speech recognition unit also translates statements made during an interview in real time and generates text in multiple languages ​​simultaneously. For example, English statements are translated into French and Spanish and converted into text. The speech recognition unit also translates statements made during a discussion in real time and generates text in multiple languages ​​simultaneously. For example, German statements are translated into Italian and Russian and converted into text. This makes it possible to generate text in multiple languages ​​simultaneously.

[0033] The speech recognition unit can convert speech data into text in real time and display it as live subtitles. For example, the speech recognition unit converts remarks made during a meeting into text in real time and displays them as live subtitles. For example, subtitles are displayed as the meeting progresses. The speech recognition unit can also convert remarks made during a lecture into text in real time and display them as live subtitles. For example, the remarks of a speaker are instantly displayed as subtitles. The speech recognition unit can also convert remarks made during a television program into text in real time and display them as live subtitles. For example, the remarks of a news anchor are displayed as subtitles. This allows live subtitles to be displayed in real time.

[0034] The summary generation unit can learn from past summary data and extract patterns to improve the accuracy of summaries. For example, the summary generation unit learns from past meeting summary data and extracts patterns to improve the accuracy of summaries. For example, it learns important points and frequently occurring keywords. The summary generation unit can also learn from past interview summary data and extract patterns to improve the accuracy of summaries. For example, it can learn the flow of the interview and important questions. The summary generation unit can also learn from past discussion summary data and extract patterns to improve the accuracy of summaries. For example, it can learn the development of the discussion and important conclusions. In this way, it is possible to learn from past data and improve the accuracy of summaries.

[0035] When generating a summary, the summary generation unit can refer to the user's past behavioral history and provide a personalized summary. The summary generation unit, for example, refers to the user's past conference participation history and provides a personalized summary. For example, topics that interest the user are prioritized for summarization. The summary generation unit also refers to the user's past interview history and provides a personalized summary. For example, questions and answers that the user considers important are summarized with emphasis. The summary generation unit also refers to the user's past discussion history and provides a personalized summary. For example, points of discussion that the user particularly paid attention to are summarized with emphasis. In this way, a personalized summary can be provided based on the user's past behavioral history.

[0036] The summary generation unit can automatically add visual elements when generating a summary. For example, the summary generation unit automatically adds visual elements when generating a summary of a meeting. For example, it adds a timeline showing the progress of the discussion or a graph showing important data. The summary generation unit also automatically adds visual elements when generating a summary of an interview. For example, it adds a diagram showing the profile of the interviewee or statistical data of the answers. The summary generation unit also automatically adds visual elements when generating a summary of a discussion. For example, it adds a mind map showing the structure of the discussion or a flowchart showing the conclusion. In this way, adding visual elements makes the summary easier to understand visually.

[0037] The summary generation unit can output a summary in different formats when generating it. For example, when generating a summary of a meeting, the summary generation unit outputs it in a presentation format. For example, the summary may be displayed in slide format to make it easier to understand visually. When generating a summary of an interview, the summary generation unit outputs it in a report format. For example, the summary may be displayed in text format to provide detailed information. When generating a summary of a discussion, the summary generation unit outputs it in a memo format. For example, the summary may be displayed in bullet point format to convey information concisely. This allows summaries to be output in different formats.

[0038] The record management unit can automatically tag recorded data to improve searchability. For example, the record management unit automatically tags summary data of recorded meetings. For example, it adds the agenda and names of participants as tags. The record management unit also automatically tags summary data of recorded interviews. For example, it adds the topic of the question and the occupation of the respondent as tags. The record management unit also automatically tags summary data of recorded discussions. For example, it adds the topic and conclusion of the discussion as tags. This can improve searchability of recorded data.

[0039] The record management unit can encrypt the recorded data to enhance security. For example, the record management unit encrypts summary data of a recorded meeting to enhance security. For example, the AES encryption algorithm is used to protect the data. The record management unit also encrypts summary data of a recorded interview to enhance security. For example, the RSA encryption algorithm is used to protect the data. The record management unit also encrypts summary data of a recorded discussion to enhance security. For example, the TLS protocol is used to protect transmission and reception of data. This enhances the security of the recorded data.

[0040] The records management unit periodically backs up recorded data to prevent data loss. For example, the records management unit periodically backs up summary data of recorded meetings to prevent data loss. For example, the backup is automatically performed daily to cloud storage. The records management unit also periodically backs up summary data of recorded interviews to prevent data loss. For example, the backup is automatically performed weekly to an external hard drive. The records management unit also periodically backs up summary data of recorded discussions to prevent data loss. For example, the backup is automatically performed monthly to network storage. This prevents the loss of recorded data.

[0041] The record management unit can link the recorded data with cloud storage to make it accessible from anywhere. For example, the record management unit links the summary data of recorded meetings with cloud storage to make it accessible from anywhere. For example, the record management unit stores the data in Google (registered trademark) Drive or Dropbox. The record management unit also links the summary data of recorded interviews with cloud storage to make it accessible from anywhere. For example, the record management unit stores the data in OneDrive or iCloud. The record management unit also links the summary data of recorded discussions with cloud storage to make it accessible from anywhere. For example, the record management unit stores the data in Amazon S3 or Box. This makes it possible to access the recorded data from anywhere.

[0042] The record management unit can link the recorded data with other applications. For example, the record management unit links the summary data of recorded meetings with a calendar app to automatically update the meeting schedule. For example, the record management unit synchronizes the data with Google (registered trademark) Calendar or Outlook Calendar. The record management unit also links the summary data of recorded interviews with a task management app to automatically generate follow-up tasks for the interviews. For example, the record management unit synchronizes the data with Trello or Asana. The record management unit also links the summary data of recorded discussions with a project management app to reflect the results of the discussions in the project plan. For example, the record management unit synchronizes the data with Jira or Basecamp. This allows the recorded data to be linked with other applications.

[0043] The action suggestion unit can refer to past data based on the recorded content and suggest the optimal action. For example, the action suggestion unit can refer to past meeting data based on a meeting summary and suggest the optimal action. For example, it can suggest actions that were successful in past meetings. The action suggestion unit can also refer to past interview data based on an interview summary and suggest the optimal action. For example, it can suggest actions that were effective in past interviews. The action suggestion unit can also refer to past discussion data based on a discussion summary and suggest the optimal action. For example, it can suggest actions that were effective in past discussions. In this way, it is possible to suggest the optimal action by referring to past data.

[0044] The action suggestion unit can generate multiple action candidates based on the recorded content and provide the user with options. The action suggestion unit can generate multiple action candidates based on, for example, a meeting summary and provide the user with options. For example, the action suggestion unit can suggest an agenda or a task list for the next meeting. The action suggestion unit can also generate multiple action candidates based on an interview summary and provide the user with options. For example, the action suggestion unit can suggest follow-up questions or a topic for the next interview. The action suggestion unit can also generate multiple action candidates based on a discussion summary and provide the user with options. For example, the action suggestion unit can suggest a project plan or a task list based on the results of the discussion. In this way, multiple action candidates can be generated and the user can be provided with options.

[0045] The action suggestion unit can suggest collaboration with other users based on the recorded content. The action suggestion unit can suggest collaboration with other users based on, for example, a summary of a meeting. For example, it can suggest a user who is suitable for a specific task. The action suggestion unit can also suggest collaboration with other users based on a summary of an interview. For example, it can suggest a user who is suitable for following up on the interview. The action suggestion unit can also suggest collaboration with other users based on a summary of a discussion. For example, it can suggest a user who is suitable for a project based on the results of the discussion. This makes it possible to suggest collaboration with other users.

[0046] The action suggestion unit can refer to external resources and provide additional information based on the recorded content. For example, the action suggestion unit can refer to external resources and provide additional information based on a summary of a meeting, for example, by suggesting related websites or literature. Also, the action suggestion unit can refer to external resources and provide additional information based on a summary of an interview, for example, by suggesting related research papers or articles. Also, the action suggestion unit can refer to external resources and provide additional information based on a summary of a discussion, for example, by suggesting related news or reports. In this way, additional information can be provided by referring to external resources.

[0047] The user interface unit can add a voice command function to enable hands-free operation. The user interface unit can add, for example, a voice command function to enable hands-free operation. For example, it can be made possible to start or stop recording by voice. The user interface unit can also add a voice command function to enable hands-free operation. For example, it can be made possible to generate or save a summary by voice. The user interface unit can also add a voice command function to enable hands-free operation. For example, it can be made possible to search or display records by voice. This makes it possible to enable hands-free operation.

[0048] The user interface unit can incorporate an interactive tutorial to enable the user to easily learn how to operate the device. The user interface unit, for example, can incorporate an interactive tutorial to enable the user to easily learn how to operate the device. For example, a step-by-step guide can be provided. The user interface unit can also incorporate an interactive tutorial to enable the user to easily learn how to operate the device. For example, a demonstration video can be provided. The user interface unit can also incorporate an interactive tutorial to enable the user to easily learn how to operate the device. For example, a function can be provided that can simulate actual operations. This can enable the user to easily learn how to operate the device.

[0049] The user interface unit can be made compatible with different devices. For example, the user interface unit can be made compatible with tablets, allowing operation on a larger screen. For example, a layout specifically for tablets can be provided. The user interface unit can also be made compatible with smart watches, allowing for easy operation. For example, a simple operation screen specifically for smart watches can be provided. The user interface unit can also be made compatible with desktop PCs, allowing for more versatile operation. For example, a detailed settings screen specifically for desktops can be provided. This allows the unit to be made compatible with different devices.

[0050] The user interface unit can add a customization function to enable the user to change the layout to suit their preferences. The user interface unit, for example, can add a customization function to enable the user to change the layout to suit their preferences. For example, the user can freely change the placement of widgets. The user interface unit can also add a customization function to enable the user to change the layout to suit their preferences. For example, the user can change the theme color or font size. The user interface unit can also add a customization function to enable the user to change the layout to suit their preferences. For example, the user can change the type and order of information to be displayed. This allows the user to change the layout to suit their preferences.

[0051] The system according to the embodiment is not limited to the above-described example, and various modifications are possible, for example, as follows.

[0052] The speech recognition unit can identify multiple speakers and generate a different text file for each speaker. For example, it can identify multiple speakers during a meeting and save each speaker's utterances in a separate text file. For example, speaker A's utterances are saved in file A and speaker B's utterances in file B. The speech recognition unit can also identify multiple speakers during an interview and save each speaker's utterances in a separate text file. For example, the interviewer's utterances are saved in file A and the interviewee's utterances are saved in file B. The speech recognition unit can also identify multiple speakers during a discussion and save each speaker's utterances in a separate text file. For example, speaker X's utterances are saved in file X and speaker Y's utterances are saved in file Y. This allows the utterances of multiple speakers to be managed individually.

[0053] The speech recognition unit can automatically filter background sounds to generate text with less noise. For example, background sounds during a meeting can be automatically filtered to generate text with less noise. For example, the sound of an air conditioner and the sound of keyboard typing can be removed. The speech recognition unit can also automatically filter background sounds during recording while on the go to generate text with less noise. For example, the sound of cars and wind can be removed. The speech recognition unit can also automatically filter background sounds during a conference call to generate text with less noise. For example, the voices of other people talking and other noise can be removed. This allows text with less noise to be generated.

[0054] The speech recognition unit can automatically translate different languages ​​and generate text in multiple languages ​​simultaneously. For example, speech during a meeting can be translated in real time to generate text in multiple languages ​​simultaneously. For example, Japanese speech can be translated into English and Chinese and converted into text. The speech recognition unit can also translate speech during an interview in real time to generate text in multiple languages ​​simultaneously. For example, English speech can be translated into French and Spanish and converted into text. The speech recognition unit can also translate speech during a discussion in real time to generate text in multiple languages ​​simultaneously. For example, German speech can be translated into Italian and Russian and converted into text. This makes it possible to generate text in multiple languages ​​simultaneously.

[0055] The speech recognition unit can convert speech data into text in real time and display it as live subtitles. For example, remarks made during a meeting can be converted into text in real time and displayed as live subtitles. For example, subtitles are displayed as the meeting progresses. The speech recognition unit can also convert remarks made during a lecture into text in real time and display them as live subtitles. For example, the remarks of a speaker can be instantly displayed as subtitles. The speech recognition unit can also convert remarks made during a television program into text in real time and display them as live subtitles. For example, the remarks of a news anchor can be displayed as subtitles. This allows live subtitles to be displayed in real time.

[0056] The summary generation unit can learn from past summary data and extract patterns to improve the accuracy of summaries. For example, it can learn from past meeting summary data and extract patterns to improve the accuracy of summaries. For example, it can learn important points and frequently occurring keywords. The summary generation unit can also learn from past interview summary data and extract patterns to improve the accuracy of summaries. For example, it can learn the flow of the interview and important questions. The summary generation unit can also learn from past discussion summary data and extract patterns to improve the accuracy of summaries. For example, it can learn the development of the discussion and important conclusions. In this way, it is possible to learn from past data and improve the accuracy of summaries.

[0057] The records management unit can automatically tag recorded data to improve searchability. For example, it can automatically tag summary data of recorded meetings. For example, it can add the agenda and names of participants as tags. The records management unit can also automatically tag summary data of recorded interviews. For example, it can add the topic of the question and the occupation of the respondent as tags. The records management unit can also automatically tag summary data of recorded discussions. For example, it can add the topic and conclusion of the discussion as tags. This can improve searchability of recorded data.

[0058] The processing flow of the first embodiment will be briefly explained below.

[0059] Step 1: The speech recognition unit converts the user's speech into text. For example, the speech recognition unit can convert speech into text with high accuracy using deep learning. The speech recognition unit can also analyze speech using HMM (Hidden Markov Model). The speech recognition unit can also convert speech into text using phoneme recognition technology. Step 2: The summary generator summarizes the text generated by the speech recognizer. For example, the summary generator summarizes the text using a generation AI (e.g., LLM). The summary generator can also extract and summarize important parts of a sentence. The summary generator can also generate a summary by inputting a prompt containing instructions on what the user wants the generation AI to do. Step 3: The records manager stores the summaries generated by the summary generator as records. For example, the records manager may store the summaries in a database. Alternatively, the records manager may store the summaries in cloud storage. The records manager may also create an index to allow users to search for the records. Step 4: The action suggestion unit suggests actions based on the records stored by the record management unit. For example, the action suggestion unit may use the generation AI to suggest an agenda for the next meeting. The action suggestion unit may also generate a list of tasks. The action suggestion unit may also suggest actions by inputting a prompt containing instructions on what the user wants the generation AI to do.

[0060] (Example 2) An application according to an embodiment of the present invention combines speech recognition AI and generative AI. This application is a system that automatically summarizes what a user says, stores it as a record, and suggests actions based on the recorded content. This allows the application to efficiently record the user's voice and suggest subsequent actions.

[0061] An application according to an embodiment includes a speech recognition unit, a summary generation unit, a record management unit, and an action suggestion unit. The speech recognition unit converts a user's speech into text. For example, the speech recognition unit converts speech into text with high accuracy using deep learning. The speech recognition unit can also analyze speech using a hidden Markov model (HMM). The speech recognition unit can also convert speech into text using phoneme recognition technology. The summary generation unit summarizes the text generated by the speech recognition unit. For example, the summary generation unit summarizes the text using a generation AI (e.g., an LLM). The summary generation unit can also extract and summarize important parts of a sentence. The summary generation unit can also generate a summary by inputting a prompt containing instructions on what the user wants the generation AI to do. The record management unit stores the summary generated by the summary generation unit as a record. For example, the record management unit can store the summary in a database. The record management unit can also store the summary in cloud storage. The record management unit can also create an index to enable users to search for records. The action suggestion unit suggests actions based on the records stored by the record management unit. For example, the action suggestion unit suggests an agenda for the next meeting using the generation AI. The action suggestion unit can also generate a list of tasks. The action suggestion unit can also suggest actions by inputting a prompt containing instructions on what the user wants the generation AI to do. This allows the application according to the embodiment to efficiently record the user's voice and suggest subsequent actions. For example, the application can accurately record the contents of a meeting and efficiently prepare for the next meeting. The application can also easily manage daily notes and quickly find necessary information.

[0062] The speech recognition unit analyzes the tone and speed of the user's voice, infers the emotion using an emotion estimation function, and can adjust the nuance of the text based on the emotion. The speech recognition unit, for example, analyzes the tone and speed of the user's voice in real time and infers the emotion using the emotion estimation function. For example, if the user is speaking excitedly, the emotion is reflected in the text. The speech recognition unit also analyzes the tone and speed of the user's voice and infers the emotion using the emotion estimation function. For example, if the user is speaking calmly, the emotion is reflected in the text. The speech recognition unit also analyzes the tone and speed of the user's voice and infers the emotion using the emotion estimation function. For example, if the user is speaking angrily, the emotion is reflected in the text. In this way, text that reflects the user's emotion can be generated.

[0063] The speech recognition unit can identify multiple speakers and generate a different text file for each speaker. For example, the speech recognition unit may identify multiple speakers during a conference and save each speaker's utterances in a separate text file. For example, speaker A's utterances may be saved in file A and speaker B's utterances in file B. The speech recognition unit may also identify multiple speakers during an interview and save each speaker's utterances in a separate text file. For example, the interviewer's utterances may be saved in file A and the interviewee's utterances in file B. The speech recognition unit may also identify multiple speakers during a discussion and save each speaker's utterances in a separate text file. For example, speaker X's utterances may be saved in file X and speaker Y's utterances in file Y. This allows the utterances of multiple speakers to be managed individually.

[0064] The speech recognition unit can automatically filter background sounds to generate text with less noise. For example, the speech recognition unit automatically filters background sounds during a meeting to generate text with less noise. For example, it removes the sound of an air conditioner or keyboard typing sounds. The speech recognition unit also automatically filters background sounds during recording while away from home to generate text with less noise. For example, it removes the sound of cars or wind. The speech recognition unit also automatically filters background sounds during a conference call to generate text with less noise. For example, it removes other people's voices and other noises. This makes it possible to generate text with less noise.

[0065] The speech recognition unit can automatically translate different languages ​​and generate text in multiple languages ​​simultaneously. For example, the speech recognition unit translates statements made during a meeting in real time and generates text in multiple languages ​​simultaneously. For example, Japanese statements are translated into English and Chinese and converted into text. The speech recognition unit also translates statements made during an interview in real time and generates text in multiple languages ​​simultaneously. For example, English statements are translated into French and Spanish and converted into text. The speech recognition unit also translates statements made during a discussion in real time and generates text in multiple languages ​​simultaneously. For example, German statements are translated into Italian and Russian and converted into text. This makes it possible to generate text in multiple languages ​​simultaneously.

[0066] The speech recognition unit can convert speech data into text in real time and display it as live subtitles. For example, the speech recognition unit converts remarks made during a meeting into text in real time and displays them as live subtitles. For example, subtitles are displayed as the meeting progresses. The speech recognition unit can also convert remarks made during a lecture into text in real time and display them as live subtitles. For example, the remarks of a speaker are instantly displayed as subtitles. The speech recognition unit can also convert remarks made during a television program into text in real time and display them as live subtitles. For example, the remarks of a news anchor are displayed as subtitles. This allows live subtitles to be displayed in real time.

[0067] The voice recognition unit can use the emotion estimation function to analyze the emotion of the user when speaking in real time and provide feedback according to the emotion. For example, the voice recognition unit analyzes the user's utterance in real time and analyzes the emotion using the emotion estimation function. For example, if the user is nervous, advice to relax is provided. The voice recognition unit also analyzes the user's utterance in real time and analyzes the emotion using the emotion estimation function. For example, if the user is excited, advice to stay calm is provided. The voice recognition unit also analyzes the user's utterance in real time and analyzes the emotion using the emotion estimation function. For example, if the user is sad, an encouraging message is provided. This makes it possible to provide feedback according to the user's emotion.

[0068] The summary generation unit can use the emotion estimation function to emphasize important emotional elements when generating a summary. For example, when generating a summary of a meeting, the summary generation unit uses the emotion estimation function to emphasize important emotional elements. For example, the summary is generated by particularly emphasizing parts where the speaker's emotions are strongly expressed. Furthermore, when generating a summary of an interview, the summary generation unit uses the emotion estimation function to emphasize important emotional elements. For example, the summary is generated by particularly emphasizing parts where the interviewee's emotions were heightened. Furthermore, when generating a summary of a discussion, the summary generation unit uses the emotion estimation function to emphasize important emotional elements. For example, the summary is generated by particularly emphasizing parts where the discussion became heated. In this way, a summary that emphasizes important emotional elements can be generated.

[0069] The summary generation unit can learn from past summary data and extract patterns to improve the accuracy of summaries. For example, the summary generation unit learns from past meeting summary data and extracts patterns to improve the accuracy of summaries. For example, it learns important points and frequently occurring keywords. The summary generation unit can also learn from past interview summary data and extract patterns to improve the accuracy of summaries. For example, it can learn the flow of the interview and important questions. The summary generation unit can also learn from past discussion summary data and extract patterns to improve the accuracy of summaries. For example, it can learn the development of the discussion and important conclusions. In this way, it is possible to learn from past data and improve the accuracy of summaries.

[0070] When generating a summary, the summary generation unit can refer to the user's past behavioral history and provide a personalized summary. The summary generation unit, for example, refers to the user's past conference participation history and provides a personalized summary. For example, topics that interest the user are prioritized for summarization. The summary generation unit also refers to the user's past interview history and provides a personalized summary. For example, questions and answers that the user considers important are summarized with emphasis. The summary generation unit also refers to the user's past discussion history and provides a personalized summary. For example, points of discussion that the user particularly paid attention to are summarized with emphasis. In this way, a personalized summary can be provided based on the user's past behavioral history.

[0071] The summary generation unit can automatically add visual elements when generating a summary. For example, the summary generation unit automatically adds visual elements when generating a summary of a meeting. For example, it adds a timeline showing the progress of the discussion or a graph showing important data. The summary generation unit also automatically adds visual elements when generating a summary of an interview. For example, it adds a diagram showing the profile of the interviewee or statistical data of the answers. The summary generation unit also automatically adds visual elements when generating a summary of a discussion. For example, it adds a mind map showing the structure of the discussion or a flowchart showing the conclusion. In this way, adding visual elements makes the summary easier to understand visually.

[0072] The summary generation unit can output a summary in different formats when generating it. For example, when generating a summary of a meeting, the summary generation unit outputs it in a presentation format. For example, the summary may be displayed in slide format to make it easier to understand visually. When generating a summary of an interview, the summary generation unit outputs it in a report format. For example, the summary may be displayed in text format to provide detailed information. When generating a summary of a discussion, the summary generation unit outputs it in a memo format. For example, the summary may be displayed in bullet point format to convey information concisely. This allows summaries to be output in different formats.

[0073] The summary generation unit collects the user's emotional reactions to the summarized content, thereby improving the accuracy of the summary. For example, after generating a summary, the summary generation unit collects the user's emotional reactions using an emotion estimation function. For example, if the user has a positive reaction to the summary, the summary generation unit learns the summarization method. Furthermore, after generating a summary, the summary generation unit collects the user's emotional reactions using the emotion estimation function. For example, if the user has a negative reaction to the summary, the summarization method is improved. Furthermore, after generating a summary, the summary generation unit collects the user's emotional reactions using the emotion estimation function. For example, if the user has a neutral reaction to the summary, the summarization method is adjusted. In this way, the user's emotional reactions can be collected and the accuracy of the summary can be improved.

[0074] The record management unit can automatically tag recorded data to improve searchability. For example, the record management unit automatically tags summary data of recorded meetings. For example, it adds the agenda and names of participants as tags. The record management unit also automatically tags summary data of recorded interviews. For example, it adds the topic of the question and the occupation of the respondent as tags. The record management unit also automatically tags summary data of recorded discussions. For example, it adds the topic and conclusion of the discussion as tags. This can improve searchability of recorded data.

[0075] The record management unit can encrypt the recorded data to enhance security. For example, the record management unit encrypts summary data of a recorded meeting to enhance security. For example, the AES encryption algorithm is used to protect the data. The record management unit also encrypts summary data of a recorded interview to enhance security. For example, the RSA encryption algorithm is used to protect the data. The record management unit also encrypts summary data of a recorded discussion to enhance security. For example, the TLS protocol is used to protect transmission and reception of data. This enhances the security of the recorded data.

[0076] The records management unit periodically backs up recorded data to prevent data loss. For example, the records management unit periodically backs up summary data of recorded meetings to prevent data loss. For example, the backup is automatically performed daily to cloud storage. The records management unit also periodically backs up summary data of recorded interviews to prevent data loss. For example, the backup is automatically performed weekly to an external hard drive. The records management unit also periodically backs up summary data of recorded discussions to prevent data loss. For example, the backup is automatically performed monthly to network storage. This prevents the loss of recorded data.

[0077] The record management unit can link the recorded data with cloud storage to make it accessible from anywhere. For example, the record management unit links the summary data of recorded meetings with cloud storage to make it accessible from anywhere. For example, the record management unit stores the data in Google (registered trademark) Drive or Dropbox. The record management unit also links the summary data of recorded interviews with cloud storage to make it accessible from anywhere. For example, the record management unit stores the data in OneDrive or iCloud. The record management unit also links the summary data of recorded discussions with cloud storage to make it accessible from anywhere. For example, the record management unit stores the data in Amazon S3 or Box. This makes it possible to access the recorded data from anywhere.

[0078] The record management unit can link the recorded data with other applications. For example, the record management unit links the summary data of recorded meetings with a calendar app to automatically update the meeting schedule. For example, the record management unit synchronizes the data with Google (registered trademark) Calendar or Outlook Calendar. The record management unit also links the summary data of recorded interviews with a task management app to automatically generate follow-up tasks for the interviews. For example, the record management unit synchronizes the data with Trello or Asana. The record management unit also links the summary data of recorded discussions with a project management app to reflect the results of the discussions in the project plan. For example, the record management unit synchronizes the data with Jira or Basecamp. This allows the recorded data to be linked with other applications.

[0079] The record management unit can use the emotion estimation function to analyze the user's emotional response to the recorded data and evaluate the importance of the data. For example, the record management unit uses the emotion estimation function to analyze the user's emotional response to recorded meeting summary data and evaluate the importance of the data. For example, data with a large number of positive emotional responses is marked as high importance. The record management unit also uses the emotion estimation function to analyze the user's emotional response to recorded interview summary data and evaluate the importance of the data. For example, data with a large number of negative emotional responses is marked as low importance. The record management unit also uses the emotion estimation function to analyze the user's emotional response to recorded discussion summary data and evaluate the importance of the data. For example, data with a large number of neutral emotional responses is marked as medium importance. In this way, the importance of the recorded data can be evaluated.

[0080] The action suggestion unit can use the emotion estimation function to suggest an action according to the user's emotion based on the recorded content. The action suggestion unit can use the emotion estimation function to suggest an action according to the user's emotion based on, for example, a summary of a meeting. For example, if the user is feeling stressed, it can suggest an action to relax. The action suggestion unit can also use the emotion estimation function to suggest an action according to the user's emotion based on a summary of an interview. For example, if the user is excited, it can suggest an action to calm down. The action suggestion unit can also use the emotion estimation function to suggest an action according to the user's emotion based on a summary of a discussion. For example, if the user is sad, it can suggest an encouraging message. This makes it possible to suggest an action according to the user's emotion.

[0081] The action suggestion unit can refer to past data based on the recorded content and suggest the optimal action. For example, the action suggestion unit can refer to past meeting data based on a meeting summary and suggest the optimal action. For example, it can suggest actions that were successful in past meetings. The action suggestion unit can also refer to past interview data based on an interview summary and suggest the optimal action. For example, it can suggest actions that were effective in past interviews. The action suggestion unit can also refer to past discussion data based on a discussion summary and suggest the optimal action. For example, it can suggest actions that were effective in past discussions. In this way, it is possible to suggest the optimal action by referring to past data.

[0082] The action suggestion unit can generate multiple action candidates based on the recorded content and provide the user with options. The action suggestion unit can generate multiple action candidates based on, for example, a meeting summary and provide the user with options. For example, the action suggestion unit can suggest an agenda or a task list for the next meeting. The action suggestion unit can also generate multiple action candidates based on an interview summary and provide the user with options. For example, the action suggestion unit can suggest follow-up questions or a topic for the next interview. The action suggestion unit can also generate multiple action candidates based on a discussion summary and provide the user with options. For example, the action suggestion unit can suggest a project plan or a task list based on the results of the discussion. In this way, multiple action candidates can be generated and the user can be provided with options.

[0083] The action suggestion unit can suggest collaboration with other users based on the recorded content. The action suggestion unit can suggest collaboration with other users based on, for example, a summary of a meeting. For example, it can suggest a user who is suitable for a specific task. The action suggestion unit can also suggest collaboration with other users based on a summary of an interview. For example, it can suggest a user who is suitable for following up on the interview. The action suggestion unit can also suggest collaboration with other users based on a summary of a discussion. For example, it can suggest a user who is suitable for a project based on the results of the discussion. This makes it possible to suggest collaboration with other users.

[0084] The action suggestion unit can refer to external resources and provide additional information based on the recorded content. For example, the action suggestion unit can refer to external resources and provide additional information based on a summary of a meeting, for example, by suggesting related websites or literature. Also, the action suggestion unit can refer to external resources and provide additional information based on a summary of an interview, for example, by suggesting related research papers or articles. Also, the action suggestion unit can refer to external resources and provide additional information based on a summary of a discussion, for example, by suggesting related news or reports. In this way, additional information can be provided by referring to external resources.

[0085] The action suggestion unit can use the emotion estimation function to collect users' emotional reactions to proposed actions and improve the accuracy of the actions. The action suggestion unit, for example, collects users' emotional reactions to proposed meeting actions and improves the accuracy of the actions. For example, it preferentially suggests actions with a large number of positive emotional reactions. The action suggestion unit also collects users' emotional reactions to proposed interview actions and improves the accuracy of the actions. For example, it improves actions with a large number of negative emotional reactions. The action suggestion unit also collects users' emotional reactions to proposed discussion actions and improves the accuracy of the actions. For example, it adjusts actions with a large number of neutral emotional reactions. In this way, it is possible to collect users' emotional reactions to proposed actions and improve the accuracy of the actions.

[0086] The user interface unit incorporates an emotion estimation function and can customize the interface according to the user's emotion. The user interface unit, for example, incorporates an emotion estimation function and customizes the interface according to the user's emotion. For example, if the user is feeling stressed, the color of the interface is changed to a calm color. The user interface unit also incorporates an emotion estimation function and customizes the interface according to the user's emotion. For example, if the user is excited, the interface layout is simplified. The user interface unit also incorporates an emotion estimation function and customizes the interface according to the user's emotion. For example, if the user is sad, an encouraging message is displayed. This makes it possible to customize the interface according to the user's emotion.

[0087] The user interface unit can add a voice command function to enable hands-free operation. The user interface unit can add, for example, a voice command function to enable hands-free operation. For example, it can be made possible to start or stop recording by voice. The user interface unit can also add a voice command function to enable hands-free operation. For example, it can be made possible to generate or save a summary by voice. The user interface unit can also add a voice command function to enable hands-free operation. For example, it can be made possible to search or display records by voice. This makes it possible to enable hands-free operation.

[0088] The user interface unit can incorporate an interactive tutorial to enable the user to easily learn how to operate the device. The user interface unit, for example, can incorporate an interactive tutorial to enable the user to easily learn how to operate the device. For example, a step-by-step guide can be provided. The user interface unit can also incorporate an interactive tutorial to enable the user to easily learn how to operate the device. For example, a demonstration video can be provided. The user interface unit can also incorporate an interactive tutorial to enable the user to easily learn how to operate the device. For example, a function can be provided that can simulate actual operations. This can enable the user to easily learn how to operate the device.

[0089] The user interface unit can be made compatible with different devices. For example, the user interface unit can be made compatible with tablets, allowing operation on a larger screen. For example, a layout specifically for tablets can be provided. The user interface unit can also be made compatible with smart watches, allowing for easy operation. For example, a simple operation screen specifically for smart watches can be provided. The user interface unit can also be made compatible with desktop PCs, allowing for more versatile operation. For example, a detailed settings screen specifically for desktops can be provided. This allows the unit to be made compatible with different devices.

[0090] The user interface unit can add a customization function to enable the user to change the layout to suit their preferences. The user interface unit, for example, can add a customization function to enable the user to change the layout to suit their preferences. For example, the user can freely change the placement of widgets. The user interface unit can also add a customization function to enable the user to change the layout to suit their preferences. For example, the user can change the theme color or font size. The user interface unit can also add a customization function to enable the user to change the layout to suit their preferences. For example, the user can change the type and order of information to be displayed. This allows the user to change the layout to suit their preferences.

[0091] The user interface unit can use the emotion estimation function to analyze the usage status of the user interface and propose an optimal interface design. The user interface unit, for example, uses the emotion estimation function to analyze the usage status of the user interface and propose an optimal interface design. For example, if the user is feeling stressed, a simple design is proposed. The user interface unit also uses the emotion estimation function to analyze the usage status of the user interface and propose an optimal interface design. For example, if the user is satisfied, the design is maintained. The user interface unit also uses the emotion estimation function to analyze the usage status of the user interface and propose an optimal interface design. For example, if the user is confused, the navigation is simplified. In this way, the usage status of the user interface can be analyzed and an optimal interface design can be proposed.

[0092] The system according to the embodiment is not limited to the above-described example, and various modifications are possible, for example, as follows.

[0093] The speech recognition unit analyzes the tone and speed of the user's voice, infers their emotion using the emotion estimation function, and can adjust the nuance of the text based on that emotion. For example, if the user is speaking excitedly, that emotion is reflected in the text. The speech recognition unit also analyzes the tone and speed of the user's voice, and infers their emotion using the emotion estimation function. For example, if the user is speaking calmly, that emotion is reflected in the text. The speech recognition unit also analyzes the tone and speed of the user's voice, and infers their emotion using the emotion estimation function. For example, if the user is speaking angrily, that emotion is reflected in the text. In this way, text that reflects the user's emotion can be generated.

[0094] The speech recognition unit can identify multiple speakers and generate a different text file for each speaker. For example, it can identify multiple speakers during a meeting and save each speaker's utterances in a separate text file. For example, speaker A's utterances are saved in file A and speaker B's utterances in file B. The speech recognition unit can also identify multiple speakers during an interview and save each speaker's utterances in a separate text file. For example, the interviewer's utterances are saved in file A and the interviewee's utterances are saved in file B. The speech recognition unit can also identify multiple speakers during a discussion and save each speaker's utterances in a separate text file. For example, speaker X's utterances are saved in file X and speaker Y's utterances are saved in file Y. This allows the utterances of multiple speakers to be managed individually.

[0095] The speech recognition unit can automatically filter background sounds to generate text with less noise. For example, background sounds during a meeting can be automatically filtered to generate text with less noise. For example, the sound of an air conditioner and the sound of keyboard typing can be removed. The speech recognition unit can also automatically filter background sounds during recording while on the go to generate text with less noise. For example, the sound of cars and wind can be removed. The speech recognition unit can also automatically filter background sounds during a conference call to generate text with less noise. For example, the voices of other people talking and other noise can be removed. This allows text with less noise to be generated.

[0096] The speech recognition unit can automatically translate different languages ​​and generate text in multiple languages ​​simultaneously. For example, speech during a meeting can be translated in real time to generate text in multiple languages ​​simultaneously. For example, Japanese speech can be translated into English and Chinese and converted into text. The speech recognition unit can also translate speech during an interview in real time to generate text in multiple languages ​​simultaneously. For example, English speech can be translated into French and Spanish and converted into text. The speech recognition unit can also translate speech during a discussion in real time to generate text in multiple languages ​​simultaneously. For example, German speech can be translated into Italian and Russian and converted into text. This makes it possible to generate text in multiple languages ​​simultaneously.

[0097] The speech recognition unit can convert speech data into text in real time and display it as live subtitles. For example, remarks made during a meeting can be converted into text in real time and displayed as live subtitles. For example, subtitles are displayed as the meeting progresses. The speech recognition unit can also convert remarks made during a lecture into text in real time and display them as live subtitles. For example, the remarks of a speaker can be instantly displayed as subtitles. The speech recognition unit can also convert remarks made during a television program into text in real time and display them as live subtitles. For example, the remarks of a news anchor can be displayed as subtitles. This allows live subtitles to be displayed in real time.

[0098] The voice recognition unit can use the emotion estimation function to analyze the emotion of the user when he or she speaks in real time and provide feedback according to the emotion. For example, the voice recognition unit can analyze the user's speech in real time and analyze the emotion using the emotion estimation function. For example, if the user is nervous, advice to relax is provided. The voice recognition unit can also analyze the user's speech in real time and analyze the emotion using the emotion estimation function. For example, if the user is excited, advice to stay calm is provided. The voice recognition unit can also analyze the user's speech in real time and analyze the emotion using the emotion estimation function. For example, if the user is sad, an encouraging message is provided. This makes it possible to provide feedback according to the user's emotion.

[0099] The summary generation unit can use the emotion estimation function to emphasize important emotional elements when generating a summary. For example, when generating a summary of a meeting, the emotion estimation function is used to emphasize important emotional elements. For example, the summary is particularly emphasized in the summary when parts where the speaker's emotions are strongly expressed. Furthermore, when generating a summary of an interview, the summary generation unit uses the emotion estimation function to emphasize important emotional elements. For example, the summary is particularly emphasized in the summary when parts where the interviewee's emotions were heightened. Furthermore, when generating a summary of a discussion, the summary generation unit uses the emotion estimation function to emphasize important emotional elements. For example, the summary is particularly emphasized in the summary when parts of the discussion became heated. In this way, a summary that emphasizes important emotional elements can be generated.

[0100] The summary generation unit can learn from past summary data and extract patterns to improve the accuracy of summaries. For example, it can learn from past meeting summary data and extract patterns to improve the accuracy of summaries. For example, it can learn important points and frequently occurring keywords. The summary generation unit can also learn from past interview summary data and extract patterns to improve the accuracy of summaries. For example, it can learn the flow of the interview and important questions. The summary generation unit can also learn from past discussion summary data and extract patterns to improve the accuracy of summaries. For example, it can learn the development of the discussion and important conclusions. In this way, it is possible to learn from past data and improve the accuracy of summaries.

[0101] The summary generation unit can collect the user's emotional reactions to the summarized content and improve the accuracy of the summary. For example, after generating a summary, the emotional reactions of the user are collected using an emotion estimation function. For example, if the user has a positive reaction to the summary, the summarization method is learned. Furthermore, after generating a summary, the summary generation unit can collect the user's emotional reactions using the emotion estimation function. For example, if the user has a negative reaction to the summary, the summarization method is improved. Furthermore, after generating a summary, the summary generation unit can collect the user's emotional reactions using the emotion estimation function. For example, if the user has a neutral reaction to the summary, the summarization method is adjusted. In this way, the user's emotional reactions can be collected and the accuracy of the summary can be improved.

[0102] The records management unit can automatically tag recorded data to improve searchability. For example, it can automatically tag summary data of recorded meetings. For example, it can add the agenda and names of participants as tags. The records management unit can also automatically tag summary data of recorded interviews. For example, it can add the topic of the question and the occupation of the respondent as tags. The records management unit can also automatically tag summary data of recorded discussions. For example, it can add the topic and conclusion of the discussion as tags. This can improve searchability of recorded data.

[0103] The processing flow of the second embodiment will be briefly explained below.

[0104] Step 1: The speech recognition unit converts the user's speech into text. For example, the speech recognition unit can convert speech into text with high accuracy using deep learning. The speech recognition unit can also analyze speech using HMM (Hidden Markov Model). The speech recognition unit can also convert speech into text using phoneme recognition technology. Step 2: The summary generator summarizes the text generated by the speech recognizer. For example, the summary generator summarizes the text using a generation AI (e.g., LLM). The summary generator can also extract and summarize important parts of a sentence. The summary generator can also generate a summary by inputting a prompt containing instructions on what the user wants the generation AI to do. Step 3: The records manager stores the summaries generated by the summary generator as records. For example, the records manager may store the summaries in a database. Alternatively, the records manager may store the summaries in cloud storage. The records manager may also create an index to allow users to search for the records. Step 4: The action suggestion unit suggests actions based on the records stored by the record management unit. For example, the action suggestion unit may use the generation AI to suggest an agenda for the next meeting. The action suggestion unit may also generate a list of tasks. The action suggestion unit may also suggest actions by inputting a prompt containing instructions on what the user wants the generation AI to do.

[0105] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0106] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> Examples of generative AIs include the data generation model 58, such as a neural network model (e.g., a neural network model), and a neural network model (e.g., a neural network model). The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating speech, text data indicating text, and image data indicating an image is also input to the data generation model 58. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specification processing unit 290 performs the above-mentioned specification processing using the data generation model 58. The data generation model 58 may be a fine-tuned model so as to output an inference result from a prompt that does not include an instruction. In this case, the data generation model 58 can output an inference result from a prompt that does not include an instruction. The data processing device 12 and the like include multiple types of data generation models 58, and the data generation model 58 includes AIs other than the generative AI. The AI ​​other than the generative AI may be, for example, linear regression, logistic regression, decision tree, random forest, support vector machine (SVM), k-means clustering, convolutional neural network (CNN), recurrent neural network (RNN), generative adversarial network (GAN), or naive Bayes, and can perform various processes, but is not limited to these examples. The AI ​​may also be an AI agent. When the processes of each of the above-mentioned parts are performed by AI, the processes may be performed in part or entirely by AI, but are not limited to these examples. The processes performed by AI, including the generative AI, may be replaced with rule-based processes.

[0107] Furthermore, the processing by the data processing system 10 described above is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart device 14, but may also be executed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart device 14. Furthermore, the specific processing unit 290 of the data processing device 12 acquires or collects information necessary for processing from the smart device 14 or an external device, and the smart device 14 acquires or collects information necessary for processing from the data processing device 12 or an external device.

[0108] [Second embodiment] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0109] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0110] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN and / or a LAN.

[0111] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0112] The microphone 238 receives instructions and the like from the user by receiving voice uttered by the user. The microphone 238 captures the voice uttered by the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to instructions from the processor 46.

[0113] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the user's surroundings (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0114] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0115] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0116] The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0117] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate a user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotion, including estimation and prediction of the user's emotion, but is not limited to these examples. Furthermore, the estimation and prediction of emotion also includes, for example, emotion analysis.

[0118] In the smart glasses 214, the specific processing is performed by the processor 46. A specific processing program 60 is stored in the storage 50. The processor 46 reads the specific processing program 60 from the storage 50 and executes the read specific processing program 60 on the RAM 48. The specific processing is realized by the processor 46 operating as the control unit 46A in accordance with the specific processing program 60 executed on the RAM 48. The smart glasses 214 also have a data generation model and an emotion identification model similar to the data generation model 58 and the emotion identification model 59, and can perform processing similar to that of the specific processing unit 290 using these models.

[0119] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 communicates with the server device having the data generation model 58 to obtain a processing result (such as a prediction result) using the data generation model 58. Furthermore, the data processing device 12 may be a server device, or may be a terminal device (for example, a mobile phone, a robot, a home appliance, etc.) owned by a user.

[0120] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0121] The data generation model 58 is a so-called generative AI. An example of the data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 receives a prompt containing an instruction, as well as inference data such as voice data representing speech, text data representing text, and image data representing an image. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The identification processing unit 290 performs the above-mentioned identification processing using the data generation model 58. The data generation model 58 may be a fine-tuned model so as to output an inference result from a prompt that does not include an instruction. In this case, the data generation model 58 can output an inference result from a prompt that does not include an instruction. The data processing device 12 and the like include multiple types of data generation models 58, and the data generation model 58 includes AI other than the generative AI. The AI ​​other than the generative AI may be, for example, linear regression, logistic regression, decision tree, random forest, support vector machine (SVM), k-means clustering, convolutional neural network (CNN), recurrent neural network (RNN), generative adversarial network (GAN), or naive Bayes, and can perform various processes, but is not limited to these examples. The AI ​​may also be an AI agent. When the processes of each of the above-mentioned parts are performed by AI, the processes may be performed in part or entirely by AI, but are not limited to these examples. The processes performed by AI, including the generative AI, may be replaced with rule-based processes.

[0122] The data processing system 210 according to the second embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 210 is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart glasses 214, but may also be executed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart glasses 214. Furthermore, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the smart glasses 214 or an external device, etc., and the smart glasses 214 acquires or collects information required for processing from the data processing device 12 or an external device, etc.

[0123] [Third embodiment] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0124] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0125] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN and / or a LAN.

[0126] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0127] The microphone 238 receives instructions and the like from the user by receiving voice uttered by the user. The microphone 238 captures the voice uttered by the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to instructions from the processor 46.

[0128] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the user's surroundings (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0129] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0130] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0131] The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0132] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate a user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotion, including estimation and prediction of the user's emotion, but is not limited to these examples. Furthermore, the estimation and prediction of emotion also includes, for example, emotion analysis.

[0133] In the headset type terminal 314, the specific processing is performed by the processor 46. A specific processing program 60 is stored in the storage 50. The processor 46 reads the specific processing program 60 from the storage 50 and executes the read specific processing program 60 on the RAM 48. The specific processing is realized by the processor 46 operating as the control unit 46A in accordance with the specific processing program 60 executed on the RAM 48. Note that the headset type terminal 314 has a data generation model and an emotion identification model similar to the data generation model 58 and the emotion identification model 59, and can also perform processing similar to that of the specific processing unit 290 using these models.

[0134] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 communicates with the server device having the data generation model 58 to obtain a processing result (such as a prediction result) using the data generation model 58. Furthermore, the data processing device 12 may be a server device, or may be a terminal device (for example, a mobile phone, a robot, a home appliance, etc.) owned by a user.

[0135] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0136] The data generation model 58 is a so-called generative AI. An example of the data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 receives a prompt containing an instruction, as well as inference data such as voice data representing speech, text data representing text, and image data representing an image. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The identification processing unit 290 performs the above-mentioned identification processing using the data generation model 58. The data generation model 58 may be a fine-tuned model so as to output an inference result from a prompt that does not include an instruction. In this case, the data generation model 58 can output an inference result from a prompt that does not include an instruction. The data processing device 12 and the like include multiple types of data generation models 58, and the data generation model 58 includes AIs other than the generative AI. The AI ​​other than the generative AI may be, for example, linear regression, logistic regression, decision tree, random forest, support vector machine (SVM), k-means clustering, convolutional neural network (CNN), recurrent neural network (RNN), generative adversarial network (GAN), or naive Bayes, and can perform various processes, but is not limited to these examples. The AI ​​may also be an AI agent. When the processes of each of the above-mentioned parts are performed by AI, the processes may be performed in part or entirely by AI, but are not limited to these examples. The processes performed by AI, including the generative AI, may be replaced with rule-based processes.

[0137] The data processing system 310 according to the third embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 310 is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the headset type terminal 314, but may also be executed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the headset type terminal 314. Furthermore, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the headset type terminal 314 or an external device, etc., and the headset type terminal 314 acquires or collects information required for processing from the data processing device 12 or an external device, etc.

[0138] [Fourth embodiment] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[0139] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0140] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN and / or a LAN.

[0141] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[0142] The microphone 238 receives instructions and the like from the user by receiving voice uttered by the user. The microphone 238 captures the voice uttered by the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to instructions from the processor 46.

[0143] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS image sensor or a CCD image sensor, and captures images of the user's surroundings (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0144] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0145] The control object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[0146] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0147] The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0148] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate a user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotion, including estimation and prediction of the user's emotion, but is not limited to these examples. Furthermore, the estimation and prediction of emotion also includes, for example, emotion analysis.

[0149] In the robot 414, the specific processing is performed by the processor 46. A specific processing program 60 is stored in the storage 50. The processor 46 reads the specific processing program 60 from the storage 50 and executes the read specific processing program 60 on the RAM 48. The specific processing is realized by the processor 46 operating as the control unit 46A in accordance with the specific processing program 60 executed on the RAM 48. The robot 414 also has a data generation model and an emotion identification model similar to the data generation model 58 and the emotion identification model 59, and can perform processing similar to that of the specific processing unit 290 using these models.

[0150] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 communicates with the server device having the data generation model 58 to obtain a processing result (such as a prediction result) using the data generation model 58. Furthermore, the data processing device 12 may be a server device, or may be a terminal device (for example, a mobile phone, a robot, a home appliance, etc.) owned by a user.

[0151] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0152] The data generation model 58 is a so-called generative AI. An example of the data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 receives a prompt containing an instruction, as well as inference data such as voice data representing speech, text data representing text, and image data representing an image. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The identification processing unit 290 performs the above-mentioned identification processing using the data generation model 58. The data generation model 58 may be a fine-tuned model so as to output an inference result from a prompt that does not include an instruction. In this case, the data generation model 58 can output an inference result from a prompt that does not include an instruction. The data processing device 12 and the like include multiple types of data generation models 58, and the data generation model 58 includes AI other than the generative AI. The AI ​​other than the generative AI may be, for example, linear regression, logistic regression, decision tree, random forest, support vector machine (SVM), k-means clustering, convolutional neural network (CNN), recurrent neural network (RNN), generative adversarial network (GAN), or naive Bayes, and can perform various processes, but is not limited to these examples. The AI ​​may also be an AI agent. When the processes of each of the above-mentioned parts are performed by AI, the processes may be performed in part or entirely by AI, but are not limited to these examples. The processes performed by AI, including the generative AI, may be replaced with rule-based processes.

[0153] The data processing system 410 according to the fourth embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 410 is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the robot 414, but may also be executed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the robot 414. Furthermore, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the robot 414 or an external device, etc., and the robot 414 acquires or collects information required for processing from the data processing device 12 or an external device, etc.

[0154] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0155] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion encompasses both emotions and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[0156] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[0157] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[0158] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is expressed, and when they approach the ideal, a state of pleasure is expressed. Emotions can also be created for robots, cars, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is expressed, and when they approach the ideal, a state of pleasure is expressed. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on speech emotion recognition and brain physiological signal analysis systems for emotions, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[0159] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[0160] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[0161] In the above embodiment, an example was given in which a specific process is performed by one computer 22, but the technology disclosed herein is not limited to this, and distributed processing of the specific process may be performed by multiple computers including computer 22.

[0162] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[0163] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0164] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[0165] The hardware resource for executing a specific process can be any of the following types of processors: A processor, for example, is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. A processor also includes a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[0166] The hardware resource that executes the specific process may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific process may be a single processor.

[0167] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[0168] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[0169] In the above example, the first to fourth embodiments have been described separately, but some or all of these embodiments may be combined. The smart device 14, smart glasses 214, headset terminal 314, and robot 414 are merely examples, and they may be combined, or other devices may be used. In the above example, the first and second embodiments have been described separately, but they may be combined.

[0170] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[0171] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference. [Explanation of symbols]

[0172] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot

Claims

1. A voice recognition unit that converts the user's voice into text using voice recognition AI; a summary generator for summarizing the text generated by the speech recognizer; a record management unit that stores the summaries generated by the summary generation unit as records; an action suggestion unit that suggests an action based on the records stored by the record management unit; A system characterized by:

2. The voice recognition unit Analyzes the tone and rate of a user's voice, estimates their emotions, and adjusts the nuances of the text based on those emotions.

2. The system of claim 1.

3. The voice recognition unit Identify multiple speakers and generate a separate text file for each speaker 2. The system of claim 1.

4. The voice recognition unit Automatically filters background noise to produce cleaner text 2. The system of claim 1.

5. The voice recognition unit Automatically translate between different languages ​​and generate text in multiple languages ​​simultaneously 2. The system of claim 1.

6. The voice recognition unit Converts audio data into text in real time and displays it as live subtitles 2. The system of claim 1.

Images