System

Abstract

A system is provided.SOLUTION: A system comprising: means for collecting speech data; means for converting the speech data to text; means for storing the subject matter of a current discussion; means for comparing the consistency of the converted text with the subject matter of the current discussion; means for detecting inconsistent speech; and means for displaying a warning to a speaker.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In today's world, remote meetings and text chats are common ways to advance projects, but consistent communication is essential for smooth discussions. However, discussions often go off track unintentionally, causing participants to lose sight of the main topic. In such cases, it can be difficult to point out the deviation, especially in discussions with other companies, resulting in the discussion stagnating. Furthermore, people who frequently derail discussions risk losing credibility. A solution to these issues is needed. [Means for solving the problem]

[0005] The present invention provides a system for preventing discussions from unintentionally going off topic in remote conferences or text chats. The system of the present invention includes the following means: means for collecting utterance data, means for converting the utterance data into text, means for storing the topic of the current discussion, means for comparing the consistency of the converted text with the topic of the current discussion, means for detecting inconsistent utterances, and means for displaying a warning to the speaker. This system allows users to receive a warning if their utterances stray from the topic of the discussion and to correct their utterances. Furthermore, by providing a means for converting voice data to text and a means for displaying a warning pop-up in real time, it is possible to more effectively prevent discussions from going off topic.

[0006] "Utterance data" refers to utterances made by a user in a remote conference or text chat, and exists in the form of voice data or text data.

[0007] "Means for converting to text" refers to the process or technology that analyzes and converts audio data into corresponding text data.

[0008] "Current topic of discussion" refers to the specific issue or discussion that participants are addressing in a remote meeting or text chat.

[0009] "Consistency" refers to a state in which statements are consistent with the topic of discussion and do not deviate from that flow.

[0010] "Consistency comparison methods" refer to techniques or algorithms that compare collected text data with the current topic of discussion to determine whether there is a match.

[0011] "Means for detecting inconsistent statements" refers to technologies or algorithms that automatically identify and detect statements that are inconsistent with the topic of discussion.

[0012] "Warning means" refers to a process or technology that provides a visual or audible warning to a speaker when there is a lack of coherence in the argument. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0014] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0017] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0018] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0019] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0021] [First embodiment]

[0022] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0023] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0026] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0029] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0033] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0034] The present invention relates to a discussion support system, and in particular to a system for preventing discussions from going off track and maintaining consistency in remote conferences and text chats. Specific embodiments for carrying out the present invention will be described below.

[0035] System Configuration

[0036] The system is broadly divided into the main functions of collecting speech data, analyzing it, detecting it, and generating and displaying warnings. Each function is realized through specific hardware and software components.

[0037] 1. Collection of speech data

[0038] User speaks

[0039] In a remote conference, users speak through their voices, and the voice data is collected through the microphones on their devices and transmitted to a server in real time.

[0040] In text chat, users send text by typing, and all this data is sent to the server in real time and temporarily stored.

[0041] 2. Speech-to-text

[0042] The server analyzes the voice

[0043] The server converts the received voice data into text using a natural language processing engine. For example, if you say, "Let's talk about the schedule for the next project," the content will be converted into text data.

[0044] 3. Check the consistency of the topic of discussion

[0045] Server refers to subject

[0046] The server accesses a database that stores meeting agendas and past meeting contents, and by reading the current topic of discussion, it can check for matches with the collected text data.

[0047] For example, if the current topic is "project schedule," the server checks whether the collected comments are on this topic.

[0048] 4. Derailment Detection

[0049] The server detects the derailment

[0050] The server uses an AI model to check whether the collected text data is straying from the topic of the discussion, and if it is determined to be a deviation, it will detect this in real time.

[0051] For example, if a comment about "the recent weather" is detected while a conversation about "the project schedule" is taking place, this will be considered a digression.

[0052] 5. Generating and Displaying Alerts

[0053] The server generated a warning

[0054] When the server detects a digression, it generates an appropriate warning message and sends it to the speaker's terminal, informing the speaker that his / her remarks are off-topic.

[0055] For example, a warning may be generated that reads, "This is off topic for the current discussion. Please talk about the schedule."

[0056] The device displays a warning

[0057] The device will display a warning message as a pop-up on the user's screen, allowing the user to realize that their comment has gone off-topic and correct it.

[0058] For example, a message may appear on the user's screen saying, "The discussion has gone off track. Please return to the correct topic."

[0059] Specific examples

[0060] Remote meeting example

[0061] User A's comment

[0062] User A says, "I'd like to talk about the schedule for the next project." The voice data is sent from the device to the server and converted into text.

[0063] User B's comment

[0064] After a short discussion, User B says, "I'd like to talk about the weather recently." This statement is also sent as voice data from the device to the server and converted into text.

[0065] Derailment detection and warning

[0066] The server detects that the comment "about the recent weather" does not match the topic of the discussion. Based on this information, the server generates a warning message and sends it to User B's device. User B's device then displays the warning message "The discussion has gone off track. Please return to the correct topic."

[0067] Correction statement

[0068] User B checks the warning message and corrects his statement by saying, "Got it. I'll get back to the topic of the schedule." The corrected statement is sent to the server again and reflected in the conference.

[0069] Text chat example

[0070] User A's comment

[0071] User A says in text chat, "Let's talk about the progress of the next project." The text data is sent from the device to the server and analyzed.

[0072] User B's comment

[0073] User B says in text chat, "I want to talk about a movie I saw recently." This text data is also sent to the server and analyzed.

[0074] Derailment detection and warning

[0075] The server detects that the comment "about movies" does not match the topic of the discussion. The server generates a warning message and sends it to User B's device. User B's device displays the message "The discussion has gone off track. Please return to the correct topic."

[0076] Correction statement

[0077] User B checks the warning message and corrects his comment by saying, "Got it. I'll get back to the topic of progress." The corrected comment is sent to the server again and reflected in the chat.

[0078] In this way, the system of the present invention is an effective tool for preventing discussions from going off track and supports smooth communication.

[0079] The processing flow will be explained below.

[0080] Step 1:

[0081] The user speaks. In the case of a remote conference, the user speaks aloud and the audio data is collected through the device's microphone. In the case of text chat, the user inputs text and the input contents are sent from the device to the server.

[0082] Step 2:

[0083] The device sends the collected voice data to the server in real time. In the case of text chat, the text data entered by the user is sent to the server as is.

[0084] Step 3:

[0085] The server converts the received voice data into text using a natural language processing engine. For example, the text "Let's talk about the schedule for the next project" is extracted from the voice data. In the case of text chat, this text conversion step is not necessary.

[0086] Step 4:

[0087] The server accesses the meeting topic database and retrieves the current discussion topic, for example, "the schedule for the next project" is recognized as the current discussion topic.

[0088] Step 5:

[0089] The server checks the converted or submitted text against the current topic of discussion. It compares the text with the topic to see if it is consistent.

[0090] Step 6:

[0091] The server uses an AI model to check whether the converted or submitted text strays from the topic of the discussion. For example, if a statement like "Let's talk about the weather these days" doesn't match the topic, it's considered a digression.

[0092] Step 7:

[0093] If the server detects a digression, it will generate an appropriate warning message, for example, "We are going off topic for the current discussion. Please talk about the schedule."

[0094] Step 8:

[0095] The server generates a warning message and sends it to the speaker's terminal, which then displays it as a pop-up on the user's screen in real time.

[0096] Step 9:

[0097] The user sees the warning message and realizes that their comment is off-topic. The user corrects their comment and re-comments on the correct topic.

[0098] Step 10:

[0099] The corrected statement is sent again from the terminal to the server, where it is checked for consistency again and reflected in the discussion.

[0100] Example 1

[0101] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0102] In remote meetings and text chats, discussions can easily stray from the topic, making it difficult to maintain consistency. If this problem is not resolved, meeting efficiency may decrease and important discussions may be overlooked. Since it is difficult to receive direct feedback like in face-to-face meetings, particularly in remote environments, effective methods for maintaining consistency in discussions are required.

[0103] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0104] In this invention, the server includes a means for a user to make a statement, a means for a terminal to send statement data, a means for the server to receive voice data, a means for the server to convert voice data to text, a means for the server to refer to agenda data, a means for the server to confirm that the speech data matches the agenda, a means for the server to analyze text data, a means for the server to detect deviations from the topic, a means for the server to generate a warning message, and a means for the terminal to display the warning message. This makes it possible to display a warning in real time even if a statement deviates from the topic, thereby maintaining the consistency of the discussion.

[0105] "User" refers to an individual participant who uses the system to participate in a remote conference or text chat and make a statement.

[0106] A "terminal" refers to an electronic device used by a user, such as a computer, smartphone, or tablet, that collects utterance data and transmits it to a server.

[0107] "Server" refers to a central processing unit that receives speech data, converts voice data to text, and checks the consistency of speech by referencing the agenda data.

[0108] "Utterance data" refers to data of voice utterances and text inputs made by a user during a remote conference or text chat.

[0109] "Audio data" refers to digital audio files of a user's spoken words collected through the device's microphone.

[0110] "Text" refers to voice data that has been analyzed by the server using a natural language processing engine and converted into text data.

[0111] "Agenda data" refers to the themes set as the subject of meetings and chats, and information stored in a database as the contents of past meetings.

[0112] "Conformance verification" refers to the process by which the server verifies that the converted text data is in line with the current agenda data.

[0113] "Derailment detection" refers to the process by which the server uses AI models to determine whether a comment is off-topic.

[0114] "Warning message" refers to a message generated by the server when a digression is detected, informing the user that the speech is off-topic.

[0115] "Real-time display" refers to the function of instantly displaying a warning message on the user's device immediately after a comment or text is entered.

[0116] A specific embodiment of the present invention for maintaining consistency in discussions in remote conferences and text chats will be described below.

[0117] First, a user uses a terminal as a means of making a statement. Terminals include computers, smartphones, tablets, etc., and these devices are equipped with input devices such as microphones and keyboards. The terminal collects the user's voice statements and text inputs and sends them to a server.

[0118] The server receives the voice data and converts it to text using a natural language processing engine. For example, the Google® Cloud Speech-to-Text API can be used to quickly convert the voice data to text. The converted text is then used to check whether it matches the agenda data.

[0119] The agenda data is stored in the server's database and includes the subject of the meeting and the contents of past meetings. The server accesses this database to obtain the current agenda and then verifies that it matches the user's speech data.

[0120] Furthermore, the server uses an AI model (e.g., GPT-4®) to analyze the collected text data to see if it deviates from the agenda. Specifically, if the content of a conversation is unrelated to the topic, it is considered a digression. For example, if someone mentions "the recent weather" while talking about "the project schedule," this is considered a digression.

[0121] When the server detects a deviation from the topic, it generates an appropriate warning message. This warning message is sent to the speaker's device. When the speaker's device receives the warning message, it displays it on the screen as a pop-up in real time. This allows the user to recognize that their comment has strayed from the topic of the discussion and to correct it.

[0122] For example, consider the following prompt:

[0123] Remote meeting prompts:

[0124] "As we conduct remote meetings to discuss the schedule for the next project, I want to create a system that will alert me if the discussion strays from the topic."

[0125] Text chat prompt:

[0126] "When we're text chatting about the next project, I want to create a system that will warn us if the conversation goes off topic."

[0127] In this way, the system of the present invention monitors user comments in real time and serves as an effective means of maintaining a consistent discussion, thereby improving the efficiency and productivity of remote meetings and text chats.

[0128] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0129] Step 1:

[0130] A user makes a statement. In a meeting or chat, the user makes a voice statement using the device's microphone, or inputs a text message using the keyboard. The input data can be audio data (e.g., a .wav file) or text data (e.g., a .txt file).

[0131] Step 2:

[0132] The device transmits speech data. The device transmits voice data and text data collected from the user to the server in real time. The device uploads the data to the server via the network.

[0133] Step 3:

[0134] The server receives the voice data. The server receives the voice data sent from the terminal. At this point, the input is voice data, and the output is saved as voice data in the server.

[0135] Step 4:

[0136] The server converts the voice data into text. The server uses a natural language processing engine (e.g., Google Cloud Speech-to-Text API) to convert the received voice data into text. For example, it takes "audio_data.wav" as input, converts it, and outputs the text "Let's talk about the schedule for the next project."

[0137] Step 5:

[0138] The server looks up the agenda data. The server queries the database to get the agenda information for the current meeting. The input is the query information, and the output is the agenda data, such as "Project Schedule."

[0139] Step 6:

[0140] The server checks whether the speech data matches the agenda. It takes text data as input and compares it with the agenda data. This check determines whether the speech is in line with the agenda. The output is a flag indicating whether it matches or does not match.

[0141] Step 7:

[0142] The server analyzes the text data. Using an AI model (e.g., GPT-4), the server analyzes the text data and checks whether the speech is on topic. The input is the text data, and the output is the analysis result.

[0143] Step 8:

[0144] The server detects digressions. Based on the analysis results, the server detects comments that stray from the topic. The input is the analysis results, and the output is a flag indicating that the comment has deviated from the topic.

[0145] Step 9:

[0146] The server generates a warning message. If a digression is detected, the server generates an appropriate warning message. The input is a digression flag, and the output is a warning message such as "We are going off topic of the current discussion. Please talk about the schedule."

[0147] Step 10:

[0148] The terminal displays the warning message. The terminal displays the received warning message in real time. The input is the warning message, and the output is a pop-up on the screen. The user can check it and correct the comment.

[0149] (Application example 1)

[0150] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0151] In factories, communication between workers and machines is an important factor that directly affects work efficiency and safety. However, inconsistent work instructions and reports can lead to work delays and mistakes, which can lead to reduced productivity and safety issues. Furthermore, when multiple workers are giving instructions to each other at the same time, discussions and instructions tend to go off track, and an effective method to prevent this is needed. Current systems have the challenge of making it difficult to issue warnings at the appropriate time or to correct the course of discussions.

[0152] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0153] In this invention, the server includes means for collecting utterance data, means for converting the utterance data into text, means for storing the topic of the current discussion, means for comparing the consistency of the converted text with the topic of the current discussion, means for detecting inconsistent utterances, means for displaying a warning to the speaker, means for checking whether instructions within the factory are consistent, means for analyzing the collected text and detecting content that deviates from the set work topic, and means for displaying a warning through the smart glasses. This makes it possible to monitor the consistency of instructions and reports in real time and quickly issue a warning if there is a deviation, thereby maintaining efficient and safe communication within the factory.

[0154] "Utterance data" refers to data that refers to the content of communication through voice or text.

[0155] "Text conversion means" refers to a system or algorithm used to convert audio data into written information.

[0156] "Means for storing the current discussion topic" refers to a system or storage that saves the topic of a meeting or instruction and retains it for future reference.

[0157] "Consistency comparison measures" refer to systems or algorithms that check whether the converted text deviates from the current subject matter.

[0158] "Incoherent speech detection" refers to systems or algorithms that automatically identify when discussions or instructions stray from the current topic.

[0159] "Means for displaying a warning" refers to a display or alert system that notifies the speaker when inconsistent speech is detected.

[0160] "Means to ensure consistency of instructions within the factory" refers to systems or algorithms that monitor whether instructions and reports in factory work deviate from the established work theme.

[0161] "Means for analyzing collected text" refers to systems or algorithms used to analyze text data and assess its content consistency and relevance.

[0162] "Means for detecting deviations from the scope of work" refers to systems or algorithms that detect when work instructions or reports deviate from the scope of the set work.

[0163] "Means for displaying a warning through smart glasses" refers to a system or application that uses smart glasses to display a warning to the speaker in real time.

[0164] Specific embodiments of the present invention will be described below.

[0165] System Configuration

[0166] The system of the present invention consists of the main functions of collecting, analyzing, and detecting speech data, and generating and displaying warnings. To achieve these functions, we use a server, smart glasses, speech recognition software, and a generative AI model.

[0167] Collection of speech data

[0168] The server collects voice data in real time from factory workers speaking through smart glasses. The smart glasses use a microphone to collect voice data and send it to the server, ensuring that the server always has the latest voice data.

[0169] Speech-to-text

[0170] The server converts the collected voice data into text data using "speech_recognition" software. For example, if a worker says, "Maintenance on this machine will be performed at 2 p.m.", the voice data is converted into text data, which is then used for further analysis.

[0171] Recollection of the current topic of discussion

[0172] The server uses a database to store the current task or topic of discussion. This database stores the set task topics and past records, allowing the collected text data to be compared with the current topic.

[0173] Checking consistency and detecting deviations

[0174] The server analyzes the converted text data using a generative AI model using the "transformers" library and checks for consistency with the current topic. This step checks whether the discussion or instructions deviate from the set task topic. If a deviation is detected, it is deemed to be a digression.

[0175] For example, if a worker says, "What time do we take our coffee break?", the system detects that this statement deviates from the current topic of machine maintenance procedures.

[0176] Generating and Displaying Alerts

[0177] If the server detects a deviation, it generates an appropriate warning message and sends it to the smart glasses, which then display the warning as a pop-up message to notify the worker, for example, "The discussion has deviated. Please return to the correct topic."

[0178] This allows the worker to review the warning and correct their statement, which is also sent to the server in real time, and the process continues until consistency is confirmed.

[0179] Prompt Sentence Examples

[0180] Below is an example of a prompt sentence to be input to the generative AI model.

[0181] Make sure instructions are consistent within your factory.

[0182] An example of text input is:

[0183] Maintenance on this machine will be done at 2 p.m. By the way, what time do you take your coffee break?

[0184] This system allows communication within the factory to be maintained consistently, efficiently and safely.

[0185] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0186] Step 1:

[0187] The user speaks through the smart glasses. The microphone in the smart glasses collects the voice data and transmits it to the server in real time. The user's speech arrives at the server as voice data.

[0188] input:

[0189] User voice data

[0190] Data processing:

[0191] Audio data collection

[0192] output:

[0193] Collected audio data

[0194] Specific behavior:

[0195] The user says, "Maintenance on this machine will be performed at 2 p.m." This speech is collected by the microphone in the smart glasses and sent to the server.

[0196] Step 2:

[0197] The server uses speech recognition software (speech_recognition) to convert the voice data into text data, which is then stored on the server.

[0198] input:

[0199] Collected audio data

[0200] Data Calculation:

[0201] Converting audio data to text

[0202] output:

[0203] Converted text data

[0204] Specific behavior:

[0205] The server uses voice recognition software to generate and store the text data: "Maintenance on this machine will be performed at 2:00 p.m."

[0206] Step 3:

[0207] The server retrieves the current discussion topic from the database, and this topic is compared with the converted text data.

[0208] input:

[0209] Current topic of discussion, converted text data

[0210] Data Calculation:

[0211] Thematic extraction and comparison with text data

[0212] output:

[0213] Data on the current subject

[0214] Specific behavior:

[0215] The server retrieves the current discussion topic "machine maintenance procedures" from the database and prepares it for analysis.

[0216] Step 4:

[0217] The server uses a generative AI model (e.g., the transformers library) to analyze the transformed text data for consistency with the current topic of discussion.

[0218] input:

[0219] Converted text data, current subject

[0220] Data Calculation:

[0221] Analyzing text data and checking consistency

[0222] output:

[0223] Consistency analysis results

[0224] Specific behavior:

[0225] The server uses a generative AI model to compare the input text "Maintenance on this machine will be performed at 2pm" with the subject "Machine maintenance procedures."

[0226] Step 5:

[0227] Based on the results of the consistency analysis, the server generates a warning message if the text data deviates from the topic of the current discussion.

[0228] input:

[0229] Consistency analysis results

[0230] Data processing:

[0231] Generate a warning message

[0232] output:

[0233] Warning message

[0234] Specific behavior:

[0235] The server detects that the utterance "What time do you take your coffee break?" is off-topic and generates a warning message saying "The discussion is going off topic. Please get back on topic."

[0236] Step 6:

[0237] The server sends the generated warning message to the smart glasses, which display it to the speaker in real time.

[0238] input:

[0239] Warning message

[0240] Data processing:

[0241] Sending, receiving, and displaying alert messages

[0242] output:

[0243] Displaying a warning message

[0244] Specific behavior:

[0245] A warning message is sent from the server to the smart glasses to notify the user that their speech is deviating. The smart glasses display shows a real-time message saying, "The discussion is going off topic. Please return to the correct topic."

[0246] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0247] The present invention combines a discussion support system with an emotion engine, and is a system that takes into account the emotions of users, particularly in remote conferences and text chats, prevents discussions from going off track, and maintains consistency in discussions. Specific embodiments for implementing the present invention are described below.

[0248] System Configuration

[0249] The system consists of the following main functions: speech data collection, analysis, emotion recognition and detection, and warning generation and display. Each function is realized through specific hardware and software components.

[0250] 1. Collection of speech data

[0251] User speaks

[0252] In a remote conference, users speak through their voices, and the voice data is collected through the microphones on their devices and transmitted to a server in real time.

[0253] In text chat, a user inputs text, which is then sent from the terminal to the server.

[0254] 2. Speech-to-text

[0255] The server analyzes the voice

[0256] The server converts the received voice data into text using a natural language processing engine. For example, if you say, "Let's talk about the schedule for the next project," the content will be converted into text data.

[0257] 3. Emotional Data Collection and Analysis

[0258] The server uses the emotion engine

[0259] The server uses an emotion engine to recognize user emotions from voice and text data, for example, identifying emotions such as "joy" or "anger" from voice tone and text keywords.

[0260] 4. Check the consistency of the topic of discussion

[0261] Server refers to subject

[0262] The server accesses a database that stores meeting agendas and past discussions, reads the current topic of the discussion, and compares the collected text data with the emotion data to check for consistency.

[0263] 5. Derailment Detection

[0264] The server detects the derailment

[0265] The server uses AI models to check whether the collected text and sentiment data deviates from the topic of the discussion. For example, if a conversation about the "project schedule" is interrupted by a mention of the "recent weather," it will be deemed a digression.

[0266] 6. Generating and Displaying Alerts

[0267] The server generated a warning

[0268] When the server detects a deviation, it generates an appropriate warning message, whose content and format are adjusted based on the user's emotional state. For example, if the user feels angry, it generates a more gentle warning.

[0269] The device displays a warning

[0270] The device will display a warning message as a pop-up on the user's screen, allowing the user to realize that their comment has gone off topic and correct it. For example, a message such as "The discussion has gone off topic. Please return to the correct topic."

[0271] Specific examples

[0272] Remote meeting example

[0273] User A's comment

[0274] User A says, "I'd like to talk about the schedule for the next project." The voice data is sent from the device to the server and converted into text.

[0275] User B's comment

[0276] After a short discussion, User B says, "I'd like to talk about the weather recently." This statement is also sent as voice data from the device to the server and converted into text.

[0277] Sentiment analysis and digression detection

[0278] The server detects that the comment "recent weather" does not match the topic of the discussion, and further recognizes the emotion of "joy" from User B's tone of voice. Based on this information, it determines that the discussion is derailing.

[0279] Generating and Displaying Alerts

[0280] The server generates a warning message saying, "This is off topic. Please talk about your schedule." and sends it to the terminal in a gentler tone to match the emotions of User B. User B's terminal displays this warning.

[0281] Correction statement

[0282] User B checks the warning message and corrects his statement by saying, "Got it. I'll get back to the topic of the schedule." The corrected statement is sent to the server again and reflected in the conference.

[0283] Text chat example

[0284] User A's comment

[0285] User A says in text chat, "Let's talk about the progress of the next project." The text data is sent from the device to the server and analyzed.

[0286] User B's comment

[0287] User B says in text chat, "I want to talk about a movie I saw recently." The server receives this text and analyzes it.

[0288] Sentiment analysis and digression detection

[0289] The server detects that the comment "about movies" does not match the topic of the discussion, and further recognizes the emotion of "enjoyment" from the text content of User B. Based on this, it is determined that the conversation has gone off topic.

[0290] Generating and Displaying Alerts

[0291] The server generates a warning message saying, "This is off topic. Please bring the discussion back to progress." and sends it to the terminal in a gentle manner that matches the emotions of User B. User B's terminal displays this warning.

[0292] Correction statement

[0293] User B checks the warning message and corrects his comment by saying, "Got it. I'll get back to the topic of progress." The corrected comment is sent to the server again and reflected in the chat.

[0294] In this way, the system of the present invention prevents discussions from going off track and supports smooth communication by providing appropriate feedback taking into account the user's emotions.

[0295] The processing flow will be explained below.

[0296] Step 1:

[0297] The user speaks. In the case of a remote conference, the user speaks aloud and the audio data is collected through the device's microphone. In the case of text chat, the user inputs text and the content is sent from the device to the server.

[0298] Step 2:

[0299] The device sends the collected voice data to the server in real time. In the case of text chat, the text data entered by the user is sent to the server as is.

[0300] Step 3:

[0301] The server converts the received voice data into text using a natural language processing engine. For example, the text "Let's talk about the schedule for the next project" is extracted from the voice data. In the case of text chat, this text conversion step is not necessary.

[0302] Step 4:

[0303] The server inputs voice and text data into an emotion engine to analyze the user's emotions. For example, emotions such as "joy" or "anger" are recognized from the tone and tempo of the voice and from keywords in the text.

[0304] Step 5:

[0305] The server accesses the agenda database and identifies the current topic of discussion, for example, recognizing that the current topic is "project schedule."

[0306] Step 6:

[0307] The server compares the converted text with the current discussion topic to ensure consistency. For example, does the statement "Let's talk about the next step" fit into the topic "Schedule for the next project"?

[0308] Step 7:

[0309] The server uses an AI model to check whether the converted text and analyzed sentiment data deviate from the topic of the discussion. For example, a statement like "Let's talk about the weather recently" is detected as inconsistent with the topic.

[0310] Step 8:

[0311] When the server detects a deviation, it generates a warning message that takes into account the user's emotions. For example, if the user is angry, it will choose a softer expression such as "Let's talk a little more calmly."

[0312] Step 9:

[0313] The server generates a warning message and sends it to the speaker's terminal, which then displays the warning message as a pop-up on the user's screen.

[0314] Step 10:

[0315] The user sees the warning message and realizes that their comment is off-topic. The user corrects their comment and re-comments on the correct topic.

[0316] Step 11:

[0317] The corrected statement is sent from the device to the server again, where it is checked for consistency and reflected in the discussion. For example, a corrected statement such as "I understand. I'll come back to the schedule for the next project" is made, allowing the discussion to proceed smoothly.

[0318] In this way, the system can prevent discussions from going off track and maintain consistency. Also, by taking into account the user's feelings, the system can provide appropriate feedback and support smooth communication.

[0319] Example 2

[0320] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0321] In remote meetings and text chats, it is difficult to maintain consistency while preventing discussions from going off track, and it is necessary to provide feedback that takes users' emotions into consideration. Furthermore, if a discussion goes off track, users may not notice it, which can hinder smooth communication.

[0322] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0323] In this invention, the server includes means for collecting utterance data, means for converting voice data into text, means for recognizing emotions from the text data and voice data, means for storing the topic of the current discussion, means for comparing the consistency of the converted text with the topic of the current discussion, means for detecting inconsistent utterances, means for generating a warning adjusted based on the emotional state of the speaker, and means for displaying the warning to the speaker. This allows for smooth communication by providing appropriate feedback that takes into account the user's emotions while maintaining the consistency of the discussion.

[0324] "Utterance data" refers to data in the form of voice and text uttered by a user.

[0325] "Audio data" refers to data that is a digital representation of the audio signal emitted by a user through a microphone.

[0326] "Text data" refers to data consisting of character information that has been converted from voice data using a natural language processing engine.

[0327] "Emotion Recognition" refers to techniques and systems for identifying a user's emotional state analyzed from speech and text data.

[0328] "Topic of discussion" refers to a specific topic or theme discussed in a meeting or text chat.

[0329] "Coherence" refers to the level of continuity and relevance that assesses whether the current statement matches the topic of discussion.

[0330] "Digression" refers to comments or actions that stray from the topic of the discussion.

[0331] A "warning message" refers to a notification to inform the user that the discussion has deviated from the topic.

[0332] "Pop-up display" refers to a type of notification format that is displayed on the user's screen in real time.

[0333] An "emotion engine" refers to software and algorithms that analyze and recognize user emotions from voice and text.

[0334] "Server" refers to a central processing unit that analyzes voice and text data, checks the consistency of discussions, recognizes emotions, generates alerts, and so on.

[0335] "Terminal" refers to a device that allows a user to input voice or text and displays alert messages.

[0336] System Configuration

[0337] This invention is a system that supports user speech in remote conferences and text chats. This system includes means for collecting user speech data, converting voice data into text, recognizing emotions, checking the consistency of discussions, and displaying warning messages for off-topic speech. Specific operations and examples are described in detail below.

[0338] Collection of speech data

[0339] User speaks

[0340] In a remote conference, users speak through a microphone. For example, user A might say, "I'd like to talk about the schedule for the next project." The speech is collected as audio data by the device's microphone and sent to the server in real time. In the case of text chat, the text entered by the user is sent directly from the device to the server.

[0341] Speech-to-text

[0342] The server analyzes the voice data

[0343] The server converts the voice data into text data using a natural language processing engine (e.g., Google Cloud Speech-to-Text). For example, a statement such as "I would like to talk about the schedule for the next project" is converted into text data.

[0344] Emotion data collection and analysis

[0345] The server uses the emotion engine

[0346] The server uses an emotion engine (e.g., IBM Watson (registered trademark) Tone Analyzer) to recognize the user's emotion from the voice and text data. For example, the server can identify the user's emotion as "joy" or "anger" from the tone of the voice and keywords in the text.

[0347] Checking the consistency of the topic of discussion

[0348] The server references the database

[0349] The server accesses a database that stores meeting agendas and past discussions, and reads the current topic of the discussion. For example, it extracts information about the "schedule for the next project." It then compares the collected text data with the emotion data to check whether the discussion is consistent.

[0350] Derailment detection

[0351] The server uses AI models to detect derailments

[0352] The server uses a generative AI model (e.g., GPT-4) to check whether the current discussion is deviating from the topic of the discussion. For example, if a comment about "recent weather" is detected while talking about "project schedule," it will consider this a digression.

[0353] Generating and Displaying Alerts

[0354] Server generates warning message

[0355] When the server detects a deviation, it generates an appropriate warning message, the content and format of which are adjusted based on the user's emotional state. For example, if the user feels angry, it generates a more gentle warning.

[0356] The device displays a warning message

[0357] The device will display a warning message as a pop-up on the user's screen, allowing the user to realize that their comment has gone off topic and correct it. For example, a message such as "The discussion has gone off topic. Please return to the correct topic" may be displayed.

[0358] Feedback and Corrections

[0359] The user confirms the warning message and corrects their comment.

[0360] The user confirms the warning message and corrects their comment to return to the topic of the discussion. For example, User B might say, "Got it. Let's get back to the topic of the schedule." This corrected comment data is sent back from the device to the server and reflected in the content of the meeting or chat. This maintains the consistency of the discussion.

[0361] Prompt Sentence Examples

[0362] "In your next discussion, please use an example of a meeting using this system. For example, explain in detail how the system generates a warning when User A speaks about the topic of the meeting and User B strays from the topic, and how User B responds to it."

[0363] In this way, the system can support smooth communication by maintaining consistency in the user's speech and providing appropriate feedback that takes emotions into consideration.

[0364] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0365] Step 1:

[0366] User speaks

[0367] How it works: Users speak through a microphone in a remote meeting, or, in the case of text chat, type text.

[0368] Input: Voice data (for remote meetings) or text data (for text chat)

[0369] Output: Audio data is collected from the microphone and sent to the device. Text data is sent directly to the device.

[0370] Step 2:

[0371] The device sends the voice data to the server

[0372] Operation: Collected voice data or entered text data is sent from the device to the server in real time.

[0373] Input: Audio or text data

[0374] Output: The server receives the audio or text data.

[0375] Step 3:

[0376] The server converts the voice data into text

[0377] How it works: The server converts the audio data into text using a natural language processing engine (e.g., Google Cloud Speech-to-Text).

[0378] Input: Audio data

[0379] Output: Text data generated by the natural language processing engine

[0380] Step 4:

[0381] The server collects and analyzes emotion data

[0382] How it works: The server uses an emotion engine (e.g., IBM Watson Tone Analyzer) to analyze the user's emotions from text and voice data.

[0383] Input: Text or audio data

[0384] Output: User's emotional state (e.g., happy, angry, surprised, etc.)

[0385] Step 5:

[0386] The server checks the consistency of the discussion topic

[0387] How it works: The server accesses a database that stores meeting agendas and past discussions, and retrieves the current topic of discussion.

[0388] Input: Parsed text and sentiment data

[0389] Output: Evaluation of the topical coherence of the discussion

[0390] Step 6:

[0391] The server detects the derailment

[0392] How it works: The server uses a generative AI model to ensure that the text and sentiment data stays on topic.

[0393] Input: Text data, sentiment data, and discussion topics

[0394] Output: Derailment status (e.g., derailed, not derailed)

[0395] Step 7:

[0396] Server generates warning message

[0397] BEHAVIOR: If the server detects a digression, it generates an appropriate warning message that is tailored based on the user's emotional state.

[0398] Input: Derailment detection results and user emotion data

[0399] Output: Adjusted warning message

[0400] Step 8:

[0401] The device displays a warning message

[0402] How it works: The device will pop up a warning message on the user's screen in real time, letting them know they've gone off topic.

[0403] Input: The warning message sent by the server

[0404] Output: A popup warning message is displayed on the user's screen.

[0405] Step 9:

[0406] User makes correction comment

[0407] Action: The user acknowledges the warning message and edits their comment to bring it back on topic.

[0408] Input: Correction statement by the user who confirmed the warning message

[0409] Output: The corrected utterance data is sent from the device to the server.

[0410] Step 10:

[0411] The server reflects the revised comments in the conference

[0412] How it works: The server receives the corrected speech data and reflects it in the discussions in meetings and chats to maintain consistency.

[0413] Input: Corrected utterance data

[0414] Output: Updated meeting and chat discussions

[0415] (Application example 2)

[0416] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0417] In remote meetings and text chats, there is a need for a system that can take into account the emotions of the speaker, prevent discussions from going off track, and maintain consistency. However, current systems lack sufficient emotional analysis, making it difficult to generate flexible warning messages that reflect the speaker's emotions. Furthermore, in workplaces where effective communication is required, such as factories, it is important to analyze emotions in real time and provide appropriate feedback.

[0418] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting utterance data, means for converting the utterance data into text, means for storing the topic of the current discussion, means for comparing the consistency of the converted text with the topic of the current discussion, means for detecting inconsistent utterances, means for displaying a warning to the speaker, means for analyzing emotions from the collected utterance data, and means for adjusting the content and format of the warning message based on the emotion analysis results. This makes it possible to prevent the discussion from going off track and provide appropriate feedback that takes the speaker's emotions into consideration.

[0419] The "means for collecting speech data" is a function that records speech or text spoken by users in real time and transmits it to a server.

[0420] The "means for converting speech data into text" is a function that converts collected speech data into text data using speech recognition technology.

[0421] "Means for storing the current discussion topic" is a function that stores the purpose and topic of a meeting or discussion in a database and makes it accessible as needed.

[0422] "Means for comparing the consistency of the converted text with the topic of the current discussion" is a function that checks whether the content of the speech converted into text is consistent with the topic of the current discussion.

[0423] "Means for detecting inconsistent statements" is a function that automatically identifies statements that deviate from the topic of discussion.

[0424] The "means for displaying a warning to the speaker" is a function that notifies the speaker visually or audibly of a warning when a deviation in the discussion is detected.

[0425] "Means for analyzing emotions from collected speech data" refers to a function that identifies the speaker's emotions from the tone of the voice and the vocabulary of the text.

[0426] "Means for adjusting the content and format of warning messages based on the results of emotion analysis" refers to a function that generates warning messages with appropriate tone and expression according to the analyzed emotions.

[0427] System program generation and explanation

[0428] In this invention, a system is implemented in which factory robots effectively support discussions. The specific configuration and operating principle of this system are described below.

[0429] Hardware used

[0430] Microphone: Used to collect what the workers say.

[0431] Robot Display: Used to display a warning message to the speaker.

[0432] Server: Performs various processes such as converting speech to text, checking the consistency of discussions, and sentiment analysis.

[0433] Software used

[0434] Python: Used as a programming language.

[0435] speech_recognition library: Used to convert speech to text.

[0436] text2emotion library: Used to analyze emotions from collected text.

[0437] The transformers library pipeline: used to provide sentiment analysis models.

[0438] Data processing and calculation

[0439] 1. Collecting speech data:

[0440] The server uses a microphone to collect voice data spoken by factory workers to the robot, and transmits the voice data to the server in real time.

[0441] 2. Speech to text transcription:

[0442] The server converts the speech data into text data using the speech_recognition library. For example, the speech "Proceed to install the next bolt" is converted into text.

[0443] 3. Emotion data collection and analysis:

[0444] The server uses the text2emotion and transformers libraries in a pipeline to analyze emotions from the collected speech data, for example, identifying emotions such as "urgency" or "excitement" from the tone of voice and text.

[0445] 4. Agenda consistency check:

[0446] The server accesses a database that stores meeting agendas and past instructions, and compares the current agenda with the collected text data to check for consistency and determine whether the discussion is on topic.

[0447] 5. Derailment detection:

[0448] The server uses an AI model to check whether comments stray from the topic of the discussion. For example, if someone starts discussing lunch while working, it will recognize this as a digression.

[0449] 6. Generating and displaying alerts:

[0450] When the server detects a deviation, it generates a warning message based on the emotion analysis results. For example, if the emotion analysis result is "enjoyment," it generates a gentle warning saying, "The discussion is derailing. Please return to the correct topic." The warning message is displayed on the robot display.

[0451] Specific examples

[0452] 1. Example 1:

[0453] Factory worker A says to the robot, "Let's proceed to installing the next bolt." The voice data is sent to the server and converted into text data.

[0454] After a while, worker B says, "I'd like to talk about lunch." This utterance is also collected as audio data and converted into text.

[0455] The server analyzes the combined data and determines that the "about lunch" statement does not match the agenda. Sentiment analysis also identifies the emotion of "enjoyment."

[0456] The server generates a warning message such as "The discussion is going off topic. Please return to the correct topic" and displays it on the robot display.

[0457] 2. Example prompt:

[0458] Say: "I want to talk about lunch."

[0459] Agenda: "Installing the bolts"

[0460] Current Emotion: "Excitement"

[0461] Warning message output:

[0462] "The discussion has gone off track. Please return to the correct topic."

[0463] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0464] Step 1:

[0465] Collection of speech data

[0466] When the user speaks, the robot's microphone collects the voice data, which is then sent to the server in real time. Specifically, the user might say, "Let's proceed to installing the next bolt."

[0467] Step 2:

[0468] Speech-to-text

[0469] The server converts the received voice data into text data using the speech_recognition library. The input here is voice data, and the output is the converted text data. For example, the voice data "Let's proceed to install the next bolt" is converted into text data.

[0470] Step 3:

[0471] Emotion data collection and analysis

[0472] The server uses a pipeline of the text2emotion and transformers libraries to analyze user emotions from the collected text data. The input of this step is text data, and the output is the emotion analysis result. Specifically, the emotion "hurry" is identified from the text data "Let's proceed to install the next bolt."

[0473] Step 4:

[0474] Checking the consistency of the agenda

[0475] The server accesses the database and compares the current agenda (e.g., "Install the bolt") with the collected text data (e.g., "Proceed to installing the next bolt"). The inputs in this step are the current agenda obtained from the database and the collected text data. The output is a consistency check result, specifically, "they match."

[0476] Step 5:

[0477] Derailment detection

[0478] The server uses an AI model to detect whether the content of the statement deviates from the topic of the discussion. The input for this step is the consistency check result and text data. The output is the digression detection result, which determines that the statement "I want to talk about lunch" does not match the topic of the discussion, for example.

[0479] Step 6:

[0480] Generating and Displaying Alerts

[0481] When the server detects a digression, it generates a warning message based on the emotion analysis results. The inputs to this step are the digression detection results and the emotion analysis results. The output is a warning message, specifically, a message that says, "The discussion is going off topic. Please return to the correct topic." The user is then notified via the robot display or voice.

[0482] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0483] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0484] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0485] [Second embodiment]

[0486] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0487] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0488] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0489] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0490] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0491] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0492] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0493] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0494] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0495] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0496] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0497] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0498] The present invention relates to a discussion support system, and in particular to a system for preventing discussions from going off track and maintaining consistency in remote conferences and text chats. Specific embodiments for carrying out the present invention will be described below.

[0499] System Configuration

[0500] The system is broadly divided into the main functions of collecting speech data, analyzing it, detecting it, and generating and displaying warnings. Each function is realized through specific hardware and software components.

[0501] 1. Collection of speech data

[0502] User speaks

[0503] In a remote conference, users speak through their voices, and the voice data is collected through the microphones on their devices and transmitted to a server in real time.

[0504] In text chat, users send text by typing, and all this data is sent to the server in real time and temporarily stored.

[0505] 2. Speech-to-text

[0506] The server analyzes the voice

[0507] The server converts the received voice data into text using a natural language processing engine. For example, if you say, "Let's talk about the schedule for the next project," the content will be converted into text data.

[0508] 3. Check the consistency of the topic of discussion

[0509] Server refers to subject

[0510] The server accesses a database that stores meeting agendas and past meeting contents, and by reading the current topic of discussion, it can check for matches with the collected text data.

[0511] For example, if the current topic is "project schedule," the server checks whether the collected comments are on this topic.

[0512] 4. Derailment Detection

[0513] The server detects the derailment

[0514] The server uses an AI model to check whether the collected text data is straying from the topic of the discussion, and if it is determined to be a deviation, it will detect this in real time.

[0515] For example, if a comment about "the recent weather" is detected while a conversation about "the project schedule" is taking place, this will be considered a digression.

[0516] 5. Generating and Displaying Alerts

[0517] The server generated a warning

[0518] When the server detects a digression, it generates an appropriate warning message and sends it to the speaker's terminal, informing the speaker that his / her remarks are off-topic.

[0519] For example, a warning may be generated that reads, "This is off topic for the current discussion. Please talk about the schedule."

[0520] The device displays a warning

[0521] The device will display a warning message as a pop-up on the user's screen, allowing the user to realize that their comment has gone off-topic and correct it.

[0522] For example, a message may appear on the user's screen saying, "The discussion has gone off track. Please return to the correct topic."

[0523] Specific examples

[0524] Remote meeting example

[0525] User A's comment

[0526] User A says, "I'd like to talk about the schedule for the next project." The voice data is sent from the device to the server and converted into text.

[0527] User B's comment

[0528] After a short discussion, User B says, "I'd like to talk about the weather recently." This statement is also sent as voice data from the device to the server and converted into text.

[0529] Derailment detection and warning

[0530] The server detects that the comment "about the recent weather" does not match the topic of the discussion. Based on this information, the server generates a warning message and sends it to User B's device. User B's device then displays the warning message "The discussion has gone off track. Please return to the correct topic."

[0531] Correction statement

[0532] User B checks the warning message and corrects his statement by saying, "Got it. I'll get back to the topic of the schedule." The corrected statement is sent to the server again and reflected in the conference.

[0533] Text chat example

[0534] User A's comment

[0535] User A says in text chat, "Let's talk about the progress of the next project." The text data is sent from the device to the server and analyzed.

[0536] User B's comment

[0537] User B says in text chat, "I want to talk about a movie I saw recently." This text data is also sent to the server and analyzed.

[0538] Derailment detection and warning

[0539] The server detects that the comment "about movies" does not match the topic of the discussion. The server generates a warning message and sends it to User B's device. User B's device displays the message "The discussion has gone off track. Please return to the correct topic."

[0540] Correction statement

[0541] User B checks the warning message and corrects his comment by saying, "Got it. I'll get back to the topic of progress." The corrected comment is sent to the server again and reflected in the chat.

[0542] In this way, the system of the present invention is an effective tool for preventing discussions from going off track and supports smooth communication.

[0543] The processing flow will be explained below.

[0544] Step 1:

[0545] The user speaks. In the case of a remote conference, the user speaks aloud and the audio data is collected through the device's microphone. In the case of text chat, the user inputs text and the input contents are sent from the device to the server.

[0546] Step 2:

[0547] The device sends the collected voice data to the server in real time. In the case of text chat, the text data entered by the user is sent to the server as is.

[0548] Step 3:

[0549] The server converts the received voice data into text using a natural language processing engine. For example, the text "Let's talk about the schedule for the next project" is extracted from the voice data. In the case of text chat, this text conversion step is not necessary.

[0550] Step 4:

[0551] The server accesses the meeting topic database and retrieves the current discussion topic, for example, "the schedule for the next project" is recognized as the current discussion topic.

[0552] Step 5:

[0553] The server checks the converted or submitted text against the current topic of discussion. It compares the text with the topic to see if it is consistent.

[0554] Step 6:

[0555] The server uses an AI model to check whether the converted or submitted text strays from the topic of the discussion. For example, if a statement like "Let's talk about the weather these days" doesn't match the topic, it's considered a digression.

[0556] Step 7:

[0557] If the server detects a digression, it will generate an appropriate warning message, for example, "We are going off topic for the current discussion. Please talk about the schedule."

[0558] Step 8:

[0559] The server generates a warning message and sends it to the speaker's terminal, which then displays it as a pop-up on the user's screen in real time.

[0560] Step 9:

[0561] The user sees the warning message and realizes that their comment is off-topic. The user corrects their comment and re-comments on the correct topic.

[0562] Step 10:

[0563] The corrected statement is sent again from the terminal to the server, where it is checked for consistency again and reflected in the discussion.

[0564] Example 1

[0565] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0566] In remote meetings and text chats, discussions can easily stray from the topic, making it difficult to maintain consistency. If this problem is not resolved, meeting efficiency may decrease and important discussions may be overlooked. Since it is difficult to receive direct feedback like in face-to-face meetings, particularly in remote environments, effective methods for maintaining consistency in discussions are required.

[0567] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0568] In this invention, the server includes a means for a user to make a statement, a means for a terminal to send statement data, a means for the server to receive voice data, a means for the server to convert voice data to text, a means for the server to refer to agenda data, a means for the server to confirm that the speech data matches the agenda, a means for the server to analyze text data, a means for the server to detect deviations from the topic, a means for the server to generate a warning message, and a means for the terminal to display the warning message. This makes it possible to display a warning in real time even if a statement deviates from the topic, thereby maintaining the consistency of the discussion.

[0569] "User" refers to an individual participant who uses the system to participate in a remote conference or text chat and make a statement.

[0570] A "terminal" refers to an electronic device used by a user, such as a computer, smartphone, or tablet, that collects utterance data and transmits it to a server.

[0571] "Server" refers to a central processing unit that receives speech data, converts voice data to text, and checks the consistency of speech by referencing the agenda data.

[0572] "Utterance data" refers to data of voice utterances and text inputs made by a user during a remote conference or text chat.

[0573] "Audio data" refers to digital audio files of a user's spoken words collected through the device's microphone.

[0574] "Text" refers to voice data that has been analyzed by the server using a natural language processing engine and converted into text data.

[0575] "Agenda data" refers to the themes set as the subject of meetings and chats, and information stored in a database as the contents of past meetings.

[0576] "Conformance verification" refers to the process by which the server verifies that the converted text data is in line with the current agenda data.

[0577] "Derailment detection" refers to the process by which the server uses AI models to determine whether a comment is off-topic.

[0578] "Warning message" refers to a message generated by the server when a digression is detected, informing the user that the speech is off-topic.

[0579] "Real-time display" refers to the function of instantly displaying a warning message on the user's device immediately after a comment or text is entered.

[0580] A specific embodiment of the present invention for maintaining consistency in discussions in remote conferences and text chats will be described below.

[0581] First, a user uses a terminal as a means of making a statement. Terminals include computers, smartphones, tablets, etc., and these devices are equipped with input devices such as microphones and keyboards. The terminal collects the user's voice statements and text inputs and sends them to a server.

[0582] The server receives the audio data and converts it to text using a natural language processing engine. For example, it can use the Google Cloud Speech-to-Text API to quickly convert the audio data to text. The converted text is then used to check whether it matches the agenda data.

[0583] The agenda data is stored in the server's database and includes the subject of the meeting and the contents of past meetings. The server accesses this database to obtain the current agenda and then verifies that it matches the user's speech data.

[0584] Additionally, the server uses AI models (e.g., GPT-4) to analyze the collected text data to determine whether it deviates from the agenda. Specifically, if a comment is off-topic, it is considered a digression. For example, if someone mentions the "recent weather" while talking about the "project schedule," this is considered a digression.

[0585] When the server detects a deviation from the topic, it generates an appropriate warning message. This warning message is sent to the speaker's device. When the speaker's device receives the warning message, it displays it on the screen as a pop-up in real time. This allows the user to recognize that their comment has strayed from the topic of the discussion and to correct it.

[0586] For example, consider the following prompt:

[0587] Remote meeting prompts:

[0588] "As we conduct remote meetings to discuss the schedule for the next project, I want to create a system that will alert me if the discussion strays from the topic."

[0589] Text chat prompt:

[0590] "When we're text chatting about the next project, I want to create a system that will warn us if the conversation goes off topic."

[0591] In this way, the system of the present invention monitors user comments in real time and serves as an effective means of maintaining a consistent discussion, thereby improving the efficiency and productivity of remote meetings and text chats.

[0592] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0593] Step 1:

[0594] A user makes a statement. In a meeting or chat, the user makes a voice statement using the device's microphone, or inputs a text message using the keyboard. The input data can be audio data (e.g., a .wav file) or text data (e.g., a .txt file).

[0595] Step 2:

[0596] The device transmits speech data. The device transmits voice data and text data collected from the user to the server in real time. The device uploads the data to the server via the network.

[0597] Step 3:

[0598] The server receives the voice data. The server receives the voice data sent from the terminal. At this point, the input is voice data, and the output is saved as voice data in the server.

[0599] Step 4:

[0600] The server converts the voice data into text. The server uses a natural language processing engine (e.g., Google Cloud Speech-to-Text API) to convert the received voice data into text. For example, it takes "audio_data.wav" as input, converts it, and outputs the text "Let's talk about the schedule for the next project."

[0601] Step 5:

[0602] The server looks up the agenda data. The server queries the database to get the agenda information for the current meeting. The input is the query information, and the output is the agenda data, such as "Project Schedule."

[0603] Step 6:

[0604] The server checks whether the speech data matches the agenda. It takes text data as input and compares it with the agenda data. This check determines whether the speech is in line with the agenda. The output is a flag indicating whether it matches or does not match.

[0605] Step 7:

[0606] The server analyzes the text data. Using an AI model (e.g., GPT-4), the server analyzes the text data and checks whether the speech is on topic. The input is the text data, and the output is the analysis result.

[0607] Step 8:

[0608] The server detects digressions. Based on the analysis results, the server detects comments that stray from the topic. The input is the analysis results, and the output is a flag indicating that the comment has deviated from the topic.

[0609] Step 9:

[0610] The server generates a warning message. If a digression is detected, the server generates an appropriate warning message. The input is a digression flag, and the output is a warning message such as "We are going off topic of the current discussion. Please talk about the schedule."

[0611] Step 10:

[0612] The terminal displays the warning message. The terminal displays the received warning message in real time. The input is the warning message, and the output is a pop-up on the screen. The user can check it and correct the comment.

[0613] (Application example 1)

[0614] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0615] In factories, communication between workers and machines is an important factor that directly affects work efficiency and safety. However, inconsistent work instructions and reports can lead to work delays and mistakes, which can lead to reduced productivity and safety issues. Furthermore, when multiple workers are giving instructions to each other at the same time, discussions and instructions tend to go off track, and an effective method to prevent this is needed. Current systems have the challenge of making it difficult to issue warnings at the appropriate time or to correct the course of discussions.

[0616] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0617] In this invention, the server includes means for collecting utterance data, means for converting the utterance data into text, means for storing the topic of the current discussion, means for comparing the consistency of the converted text with the topic of the current discussion, means for detecting inconsistent utterances, means for displaying a warning to the speaker, means for checking whether instructions within the factory are consistent, means for analyzing the collected text and detecting content that deviates from the set work topic, and means for displaying a warning through the smart glasses. This makes it possible to monitor the consistency of instructions and reports in real time and quickly issue a warning if there is a deviation, thereby maintaining efficient and safe communication within the factory.

[0618] "Utterance data" refers to data that refers to the content of communication through voice or text.

[0619] "Text conversion means" refers to a system or algorithm used to convert audio data into written information.

[0620] "Means for storing the current discussion topic" refers to a system or storage that saves the topic of a meeting or instruction and retains it for future reference.

[0621] "Consistency comparison measures" refer to systems or algorithms that check whether the converted text deviates from the current subject matter.

[0622] "Incoherent speech detection" refers to systems or algorithms that automatically identify when discussions or instructions stray from the current topic.

[0623] "Means for displaying a warning" refers to a display or alert system that notifies the speaker when inconsistent speech is detected.

[0624] "Means to ensure consistency of instructions within the factory" refers to systems or algorithms that monitor whether instructions and reports in factory work deviate from the established work theme.

[0625] "Means for analyzing collected text" refers to systems or algorithms used to analyze text data and assess its content consistency and relevance.

[0626] "Means for detecting deviations from the scope of work" refers to systems or algorithms that detect when work instructions or reports deviate from the scope of the set work.

[0627] "Means for displaying a warning through smart glasses" refers to a system or application that uses smart glasses to display a warning to the speaker in real time.

[0628] Specific embodiments of the present invention will be described below.

[0629] System Configuration

[0630] The system of the present invention consists of the main functions of collecting, analyzing, and detecting speech data, and generating and displaying warnings. To achieve these functions, we use a server, smart glasses, speech recognition software, and a generative AI model.

[0631] Collection of speech data

[0632] The server collects voice data in real time from factory workers speaking through smart glasses. The smart glasses use a microphone to collect voice data and send it to the server, ensuring that the server always has the latest voice data.

[0633] Speech-to-text

[0634] The server converts the collected voice data into text data using "speech_recognition" software. For example, if a worker says, "Maintenance on this machine will be performed at 2 p.m.", the voice data is converted into text data, which is then used for further analysis.

[0635] Recollection of the current topic of discussion

[0636] The server uses a database to store the current task or topic of discussion. This database stores the set task topics and past records, allowing the collected text data to be compared with the current topic.

[0637] Checking consistency and detecting deviations

[0638] The server analyzes the converted text data using a generative AI model using the "transformers" library and checks for consistency with the current topic. This step checks whether the discussion or instructions deviate from the set task topic. If a deviation is detected, it is deemed to be a digression.

[0639] For example, if a worker says, "What time do we take our coffee break?", the system detects that this statement deviates from the current topic of machine maintenance procedures.

[0640] Generating and Displaying Alerts

[0641] If the server detects a deviation, it generates an appropriate warning message and sends it to the smart glasses, which then display the warning as a pop-up message to notify the worker, for example, "The discussion has deviated. Please return to the correct topic."

[0642] This allows the worker to review the warning and correct their statement, which is also sent to the server in real time, and the process continues until consistency is confirmed.

[0643] Prompt Sentence Examples

[0644] Below is an example of a prompt sentence to be input to the generative AI model.

[0645] Make sure instructions are consistent within your factory.

[0646] An example of text input is:

[0647] Maintenance on this machine will be done at 2 p.m. By the way, what time do you take your coffee break?

[0648] This system allows communication within the factory to be maintained consistently, efficiently and safely.

[0649] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0650] Step 1:

[0651] The user speaks through the smart glasses. The microphone in the smart glasses collects the voice data and transmits it to the server in real time. The user's speech arrives at the server as voice data.

[0652] input:

[0653] User voice data

[0654] Data processing:

[0655] Audio data collection

[0656] output:

[0657] Collected audio data

[0658] Specific behavior:

[0659] The user says, "Maintenance on this machine will be performed at 2 p.m." This speech is collected by the microphone in the smart glasses and sent to the server.

[0660] Step 2:

[0661] The server uses speech recognition software (speech_recognition) to convert the voice data into text data, which is then stored on the server.

[0662] input:

[0663] Collected audio data

[0664] Data Calculation:

[0665] Converting audio data to text

[0666] output:

[0667] Converted text data

[0668] Specific behavior:

[0669] The server uses voice recognition software to generate and store the text data: "Maintenance on this machine will be performed at 2:00 p.m."

[0670] Step 3:

[0671] The server retrieves the current discussion topic from the database, and this topic is compared with the converted text data.

[0672] input:

[0673] Current topic of discussion, converted text data

[0674] Data Calculation:

[0675] Thematic extraction and comparison with text data

[0676] output:

[0677] Data on the current subject

[0678] Specific behavior:

[0679] The server retrieves the current discussion topic "machine maintenance procedures" from the database and prepares it for analysis.

[0680] Step 4:

[0681] The server uses a generative AI model (e.g., the transformers library) to analyze the transformed text data for consistency with the current topic of discussion.

[0682] input:

[0683] Converted text data, current subject

[0684] Data Calculation:

[0685] Analyzing text data and checking consistency

[0686] output:

[0687] Consistency analysis results

[0688] Specific behavior:

[0689] The server uses a generative AI model to compare the input text "Maintenance on this machine will be performed at 2pm" with the subject "Machine maintenance procedures."

[0690] Step 5:

[0691] Based on the results of the consistency analysis, the server generates a warning message if the text data deviates from the topic of the current discussion.

[0692] input:

[0693] Consistency analysis results

[0694] Data processing:

[0695] Generate a warning message

[0696] output:

[0697] Warning message

[0698] Specific behavior:

[0699] The server detects that the utterance "What time do you take your coffee break?" is off-topic and generates a warning message saying "The discussion is going off topic. Please get back on topic."

[0700] Step 6:

[0701] The server sends the generated warning message to the smart glasses, which display it to the speaker in real time.

[0702] input:

[0703] Warning message

[0704] Data processing:

[0705] Sending, receiving, and displaying alert messages

[0706] output:

[0707] Displaying a warning message

[0708] Specific behavior:

[0709] A warning message is sent from the server to the smart glasses to notify the user that their speech is deviating. The smart glasses display shows a real-time message saying, "The discussion is going off topic. Please return to the correct topic."

[0710] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0711] The present invention combines a discussion support system with an emotion engine, and is a system that takes into account the emotions of users, particularly in remote conferences and text chats, prevents discussions from going off track, and maintains consistency in discussions. Specific embodiments for implementing the present invention are described below.

[0712] System Configuration

[0713] The system consists of the following main functions: speech data collection, analysis, emotion recognition and detection, and warning generation and display. Each function is realized through specific hardware and software components.

[0714] 1. Collection of speech data

[0715] User speaks

[0716] In a remote conference, users speak through their voices, and the voice data is collected through the microphones on their devices and transmitted to a server in real time.

[0717] In text chat, a user inputs text, which is then sent from the terminal to the server.

[0718] 2. Speech-to-text

[0719] The server analyzes the voice

[0720] The server converts the received voice data into text using a natural language processing engine. For example, if you say, "Let's talk about the schedule for the next project," the content will be converted into text data.

[0721] 3. Emotional Data Collection and Analysis

[0722] The server uses the emotion engine

[0723] The server uses an emotion engine to recognize user emotions from voice and text data, for example, identifying emotions such as "joy" or "anger" from voice tone and text keywords.

[0724] 4. Check the consistency of the topic of discussion

[0725] Server refers to subject

[0726] The server accesses a database that stores meeting agendas and past discussions, reads the current topic of the discussion, and compares the collected text data with the emotion data to check for consistency.

[0727] 5. Derailment Detection

[0728] The server detects the derailment

[0729] The server uses AI models to check whether the collected text and sentiment data deviates from the topic of the discussion. For example, if a conversation about the "project schedule" is interrupted by a mention of the "recent weather," it will be deemed a digression.

[0730] 6. Generating and Displaying Alerts

[0731] The server generated a warning

[0732] When the server detects a deviation, it generates an appropriate warning message, whose content and format are adjusted based on the user's emotional state. For example, if the user feels angry, it generates a more gentle warning.

[0733] The device displays a warning

[0734] The device will display a warning message as a pop-up on the user's screen, allowing the user to realize that their comment has gone off topic and correct it. For example, a message such as "The discussion has gone off topic. Please return to the correct topic."

[0735] Specific examples

[0736] Remote meeting example

[0737] User A's comment

[0738] User A says, "I'd like to talk about the schedule for the next project." The voice data is sent from the device to the server and converted into text.

[0739] User B's comment

[0740] After a short discussion, User B says, "I'd like to talk about the weather recently." This statement is also sent as voice data from the device to the server and converted into text.

[0741] Sentiment analysis and digression detection

[0742] The server detects that the comment "recent weather" does not match the topic of the discussion, and further recognizes the emotion of "joy" from User B's tone of voice. Based on this information, it determines that the discussion is derailing.

[0743] Generating and Displaying Alerts

[0744] The server generates a warning message saying, "This is off topic. Please talk about your schedule." and sends it to the terminal in a gentler tone to match the emotions of User B. User B's terminal displays this warning.

[0745] Correction statement

[0746] User B checks the warning message and corrects his statement by saying, "Got it. I'll get back to the topic of the schedule." The corrected statement is sent to the server again and reflected in the conference.

[0747] Text chat example

[0748] User A's comment

[0749] User A says in text chat, "Let's talk about the progress of the next project." The text data is sent from the device to the server and analyzed.

[0750] User B's comment

[0751] User B says in text chat, "I want to talk about a movie I saw recently." The server receives this text and analyzes it.

[0752] Sentiment analysis and digression detection

[0753] The server detects that the comment "about movies" does not match the topic of the discussion, and further recognizes the emotion of "enjoyment" from the text content of User B. Based on this, it is determined that the conversation has gone off topic.

[0754] Generating and Displaying Alerts

[0755] The server generates a warning message saying, "This is off topic. Please bring the discussion back to progress." and sends it to the terminal in a gentle manner that matches the emotions of User B. User B's terminal displays this warning.

[0756] Correction statement

[0757] User B checks the warning message and corrects his comment by saying, "Got it. I'll get back to the topic of progress." The corrected comment is sent to the server again and reflected in the chat.

[0758] In this way, the system of the present invention prevents discussions from going off track and supports smooth communication by providing appropriate feedback taking into account the user's emotions.

[0759] The processing flow will be explained below.

[0760] Step 1:

[0761] The user speaks. In the case of a remote conference, the user speaks aloud and the audio data is collected through the device's microphone. In the case of text chat, the user inputs text and the content is sent from the device to the server.

[0762] Step 2:

[0763] The device sends the collected voice data to the server in real time. In the case of text chat, the text data entered by the user is sent to the server as is.

[0764] Step 3:

[0765] The server converts the received voice data into text using a natural language processing engine. For example, the text "Let's talk about the schedule for the next project" is extracted from the voice data. In the case of text chat, this text conversion step is not necessary.

[0766] Step 4:

[0767] The server inputs voice and text data into an emotion engine to analyze the user's emotions. For example, emotions such as "joy" or "anger" are recognized from the tone and tempo of the voice and from keywords in the text.

[0768] Step 5:

[0769] The server accesses the agenda database and identifies the current topic of discussion, for example, recognizing that the current topic is "project schedule."

[0770] Step 6:

[0771] The server compares the converted text with the current discussion topic to ensure consistency. For example, does the statement "Let's talk about the next step" fit into the topic "Schedule for the next project"?

[0772] Step 7:

[0773] The server uses an AI model to check whether the converted text and analyzed sentiment data deviate from the topic of the discussion. For example, a statement like "Let's talk about the weather recently" is detected as inconsistent with the topic.

[0774] Step 8:

[0775] When the server detects a deviation, it generates a warning message that takes into account the user's emotions. For example, if the user is angry, it will choose a softer expression such as "Let's talk a little more calmly."

[0776] Step 9:

[0777] The server generates a warning message and sends it to the speaker's terminal, which then displays the warning message as a pop-up on the user's screen.

[0778] Step 10:

[0779] The user sees the warning message and realizes that their comment is off-topic. The user corrects their comment and re-comments on the correct topic.

[0780] Step 11:

[0781] The corrected statement is sent from the device to the server again, where it is checked for consistency and reflected in the discussion. For example, a corrected statement such as "I understand. I'll come back to the schedule for the next project" is made, allowing the discussion to proceed smoothly.

[0782] In this way, the system can prevent discussions from going off track and maintain consistency. Also, by taking into account the user's feelings, the system can provide appropriate feedback and support smooth communication.

[0783] Example 2

[0784] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0785] In remote meetings and text chats, it is difficult to maintain consistency while preventing discussions from going off track, and it is necessary to provide feedback that takes users' emotions into consideration. Furthermore, if a discussion goes off track, users may not notice it, which can hinder smooth communication.

[0786] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0787] In this invention, the server includes means for collecting utterance data, means for converting voice data into text, means for recognizing emotions from the text data and voice data, means for storing the topic of the current discussion, means for comparing the consistency of the converted text with the topic of the current discussion, means for detecting inconsistent utterances, means for generating a warning adjusted based on the emotional state of the speaker, and means for displaying the warning to the speaker. This allows for smooth communication by providing appropriate feedback that takes into account the user's emotions while maintaining the consistency of the discussion.

[0788] "Utterance data" refers to data in the form of voice and text uttered by a user.

[0789] "Audio data" refers to data that is a digital representation of the audio signal emitted by a user through a microphone.

[0790] "Text data" refers to data consisting of character information that has been converted from voice data using a natural language processing engine.

[0791] "Emotion Recognition" refers to techniques and systems for identifying a user's emotional state analyzed from speech and text data.

[0792] "Topic of discussion" refers to a specific topic or theme discussed in a meeting or text chat.

[0793] "Coherence" refers to the level of continuity and relevance that assesses whether the current statement matches the topic of discussion.

[0794] "Digression" refers to comments or actions that stray from the topic of the discussion.

[0795] A "warning message" refers to a notification to inform the user that the discussion has deviated from the topic.

[0796] "Pop-up display" refers to a type of notification format that is displayed on the user's screen in real time.

[0797] An "emotion engine" refers to software and algorithms that analyze and recognize user emotions from voice and text.

[0798] "Server" refers to a central processing unit that analyzes voice and text data, checks the consistency of discussions, recognizes emotions, generates alerts, and so on.

[0799] "Terminal" refers to a device that allows a user to input voice or text and displays alert messages.

[0800] System Configuration

[0801] This invention is a system that supports user speech in remote conferences and text chats. This system includes means for collecting user speech data, converting voice data into text, recognizing emotions, checking the consistency of discussions, and displaying warning messages for off-topic speech. Specific operations and examples are described in detail below.

[0802] Collection of speech data

[0803] User speaks

[0804] In a remote conference, users speak through a microphone. For example, user A might say, "I'd like to talk about the schedule for the next project." The speech is collected as audio data by the device's microphone and sent to the server in real time. In the case of text chat, the text entered by the user is sent directly from the device to the server.

[0805] Speech-to-text

[0806] The server analyzes the voice data

[0807] The server converts the voice data into text data using a natural language processing engine (e.g., Google Cloud Speech-to-Text). For example, a statement such as "I would like to talk about the schedule for the next project" is converted into text data.

[0808] Emotion data collection and analysis

[0809] The server uses the emotion engine

[0810] The server uses an emotion engine (e.g., IBM Watson Tone Analyzer) to recognize the user's emotion from voice and text data. For example, it can identify the user's emotion as "joy" or "anger" from the tone of the voice or keywords in the text.

[0811] Checking the consistency of the topic of discussion

[0812] The server references the database

[0813] The server accesses a database that stores meeting agendas and past discussions, and reads the current topic of the discussion. For example, it extracts information about the "schedule for the next project." It then compares the collected text data with the emotion data to check whether the discussion is consistent.

[0814] Derailment detection

[0815] The server uses AI models to detect derailments

[0816] The server uses a generative AI model (e.g., GPT-4) to check whether the current discussion is deviating from the topic of the discussion. For example, if a comment about "recent weather" is detected while talking about "project schedule," it will consider this a digression.

[0817] Generating and Displaying Alerts

[0818] Server generates warning message

[0819] When the server detects a deviation, it generates an appropriate warning message, the content and format of which are adjusted based on the user's emotional state. For example, if the user feels angry, it generates a more gentle warning.

[0820] The device displays a warning message

[0821] The device will display a warning message as a pop-up on the user's screen, allowing the user to realize that their comment has gone off topic and correct it. For example, a message such as "The discussion has gone off topic. Please return to the correct topic" may be displayed.

[0822] Feedback and Corrections

[0823] The user confirms the warning message and corrects their comment.

[0824] The user confirms the warning message and corrects their comment to return to the topic of the discussion. For example, User B might say, "Got it. Let's get back to the topic of the schedule." This corrected comment data is sent back from the device to the server and reflected in the content of the meeting or chat. This maintains the consistency of the discussion.

[0825] Prompt Sentence Examples

[0826] "In your next discussion, please use an example of a meeting using this system. For example, explain in detail how the system generates a warning when User A speaks about the topic of the meeting and User B strays from the topic, and how User B responds to it."

[0827] In this way, the system can support smooth communication by maintaining consistency in the user's speech and providing appropriate feedback that takes emotions into consideration.

[0828] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0829] Step 1:

[0830] User speaks

[0831] How it works: Users speak through a microphone in a remote meeting, or, in the case of text chat, type text.

[0832] Input: Voice data (for remote meetings) or text data (for text chat)

[0833] Output: Audio data is collected from the microphone and sent to the device. Text data is sent directly to the device.

[0834] Step 2:

[0835] The device sends the voice data to the server

[0836] Operation: Collected voice data or entered text data is sent from the device to the server in real time.

[0837] Input: Audio or text data

[0838] Output: The server receives the audio or text data.

[0839] Step 3:

[0840] The server converts the voice data into text

[0841] How it works: The server converts the audio data into text using a natural language processing engine (e.g., Google Cloud Speech-to-Text).

[0842] Input: Audio data

[0843] Output: Text data generated by the natural language processing engine

[0844] Step 4:

[0845] The server collects and analyzes emotion data

[0846] How it works: The server uses an emotion engine (e.g., IBM Watson Tone Analyzer) to analyze the user's emotions from text and voice data.

[0847] Input: Text or audio data

[0848] Output: User's emotional state (e.g., happy, angry, surprised, etc.)

[0849] Step 5:

[0850] The server checks the consistency of the discussion topic

[0851] How it works: The server accesses a database that stores meeting agendas and past discussions, and retrieves the current topic of discussion.

[0852] Input: Parsed text and sentiment data

[0853] Output: Evaluation of the topical coherence of the discussion

[0854] Step 6:

[0855] The server detects the derailment

[0856] How it works: The server uses a generative AI model to ensure that the text and sentiment data stays on topic.

[0857] Input: Text data, sentiment data, and discussion topics

[0858] Output: Derailment status (e.g., derailed, not derailed)

[0859] Step 7:

[0860] Server generates warning message

[0861] BEHAVIOR: If the server detects a digression, it generates an appropriate warning message that is tailored based on the user's emotional state.

[0862] Input: Derailment detection results and user emotion data

[0863] Output: Adjusted warning message

[0864] Step 8:

[0865] The device displays a warning message

[0866] How it works: The device will pop up a warning message on the user's screen in real time, letting them know they've gone off topic.

[0867] Input: The warning message sent by the server

[0868] Output: A popup warning message is displayed on the user's screen.

[0869] Step 9:

[0870] User makes correction comment

[0871] Action: The user acknowledges the warning message and edits their comment to bring it back on topic.

[0872] Input: Correction statement by the user who confirmed the warning message

[0873] Output: The corrected utterance data is sent from the device to the server.

[0874] Step 10:

[0875] The server reflects the revised comments in the conference

[0876] How it works: The server receives the corrected speech data and reflects it in the discussions in meetings and chats to maintain consistency.

[0877] Input: Corrected utterance data

[0878] Output: Updated meeting and chat discussions

[0879] (Application example 2)

[0880] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0881] In remote meetings and text chats, there is a need for a system that can take into account the emotions of the speaker, prevent discussions from going off track, and maintain consistency. However, current systems lack sufficient emotional analysis, making it difficult to generate flexible warning messages that reflect the speaker's emotions. Furthermore, in workplaces where effective communication is required, such as factories, it is important to analyze emotions in real time and provide appropriate feedback.

[0882] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting utterance data, means for converting the utterance data into text, means for storing the topic of the current discussion, means for comparing the consistency of the converted text with the topic of the current discussion, means for detecting inconsistent utterances, means for displaying a warning to the speaker, means for analyzing emotions from the collected utterance data, and means for adjusting the content and format of the warning message based on the emotion analysis results. This makes it possible to prevent the discussion from going off track and provide appropriate feedback that takes the speaker's emotions into consideration.

[0883] The "means for collecting speech data" is a function that records speech or text spoken by users in real time and transmits it to a server.

[0884] The "means for converting speech data into text" is a function that converts collected speech data into text data using speech recognition technology.

[0885] "Means for storing the current discussion topic" is a function that stores the purpose and topic of a meeting or discussion in a database and makes it accessible as needed.

[0886] "Means for comparing the consistency of the converted text with the topic of the current discussion" is a function that checks whether the content of the speech converted into text is consistent with the topic of the current discussion.

[0887] "Means for detecting inconsistent statements" is a function that automatically identifies statements that deviate from the topic of discussion.

[0888] The "means for displaying a warning to the speaker" is a function that notifies the speaker visually or audibly of a warning when a deviation in the discussion is detected.

[0889] "Means for analyzing emotions from collected speech data" refers to a function that identifies the speaker's emotions from the tone of the voice and the vocabulary of the text.

[0890] "Means for adjusting the content and format of warning messages based on the results of emotion analysis" refers to a function that generates warning messages with appropriate tone and expression according to the analyzed emotions.

[0891] System program generation and explanation

[0892] In this invention, a system is implemented in which factory robots effectively support discussions. The specific configuration and operating principle of this system are described below.

[0893] Hardware used

[0894] Microphone: Used to collect what the workers say.

[0895] Robot Display: Used to display a warning message to the speaker.

[0896] Server: Performs various processes such as converting speech to text, checking the consistency of discussions, and sentiment analysis.

[0897] Software used

[0898] Python: Used as a programming language.

[0899] speech_recognition library: Used to convert speech to text.

[0900] text2emotion library: Used to analyze emotions from collected text.

[0901] The transformers library pipeline: used to provide sentiment analysis models.

[0902] Data processing and calculation

[0903] 1. Collecting speech data:

[0904] The server uses a microphone to collect voice data spoken by factory workers to the robot, and transmits the voice data to the server in real time.

[0905] 2. Speech to text transcription:

[0906] The server converts the speech data into text data using the speech_recognition library. For example, the speech "Proceed to install the next bolt" is converted into text.

[0907] 3. Emotion data collection and analysis:

[0908] The server uses the text2emotion and transformers libraries in a pipeline to analyze emotions from the collected speech data, for example, identifying emotions such as "urgency" or "excitement" from the tone of voice and text.

[0909] 4. Agenda consistency check:

[0910] The server accesses a database that stores meeting agendas and past instructions, and compares the current agenda with the collected text data to check for consistency and determine whether the discussion is on topic.

[0911] 5. Derailment detection:

[0912] The server uses an AI model to check whether comments stray from the topic of the discussion. For example, if someone starts discussing lunch while working, it will recognize this as a digression.

[0913] 6. Generating and displaying alerts:

[0914] When the server detects a deviation, it generates a warning message based on the emotion analysis results. For example, if the emotion analysis result is "enjoyment," it generates a gentle warning saying, "The discussion is derailing. Please return to the correct topic." The warning message is displayed on the robot display.

[0915] Specific examples

[0916] 1. Example 1:

[0917] Factory worker A says to the robot, "Let's proceed to installing the next bolt." The voice data is sent to the server and converted into text data.

[0918] After a while, worker B says, "I'd like to talk about lunch." This utterance is also collected as audio data and converted into text.

[0919] The server analyzes the combined data and determines that the "about lunch" statement does not match the agenda. Sentiment analysis also identifies the emotion of "enjoyment."

[0920] The server generates a warning message such as "The discussion is going off topic. Please return to the correct topic" and displays it on the robot display.

[0921] 2. Example prompt:

[0922] Say: "I want to talk about lunch."

[0923] Agenda: "Installing the bolts"

[0924] Current Emotion: "Excitement"

[0925] Warning message output:

[0926] "The discussion has gone off track. Please return to the correct topic."

[0927] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0928] Step 1:

[0929] Collection of speech data

[0930] When the user speaks, the robot's microphone collects the voice data, which is then sent to the server in real time. Specifically, the user might say, "Let's proceed to installing the next bolt."

[0931] Step 2:

[0932] Speech-to-text

[0933] The server converts the received voice data into text data using the speech_recognition library. The input here is voice data, and the output is the converted text data. For example, the voice data "Let's proceed to install the next bolt" is converted into text data.

[0934] Step 3:

[0935] Emotion data collection and analysis

[0936] The server uses a pipeline of the text2emotion and transformers libraries to analyze user emotions from the collected text data. The input of this step is text data, and the output is the emotion analysis result. Specifically, the emotion "hurry" is identified from the text data "Let's proceed to install the next bolt."

[0937] Step 4:

[0938] Checking the consistency of the agenda

[0939] The server accesses the database and compares the current agenda (e.g., "Install the bolt") with the collected text data (e.g., "Proceed to installing the next bolt"). The inputs in this step are the current agenda obtained from the database and the collected text data. The output is a consistency check result, specifically, "they match."

[0940] Step 5:

[0941] Derailment detection

[0942] The server uses an AI model to detect whether the content of the statement deviates from the topic of the discussion. The input for this step is the consistency check result and text data. The output is the digression detection result, which determines that the statement "I want to talk about lunch" does not match the topic of the discussion, for example.

[0943] Step 6:

[0944] Generating and Displaying Alerts

[0945] When the server detects a digression, it generates a warning message based on the emotion analysis results. The inputs to this step are the digression detection results and the emotion analysis results. The output is a warning message, specifically, a message that says, "The discussion is going off topic. Please return to the correct topic." The user is then notified via the robot display or voice.

[0946] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0947] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0948] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0949] [Third embodiment]

[0950] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0951] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0952] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0953] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0954] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0955] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0956] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0957] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0958] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0959] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0960] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0961] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0962] The present invention relates to a discussion support system, and in particular to a system for preventing discussions from going off track and maintaining consistency in remote conferences and text chats. Specific embodiments for carrying out the present invention will be described below.

[0963] System Configuration

[0964] The system is broadly divided into the main functions of collecting speech data, analyzing it, detecting it, and generating and displaying warnings. Each function is realized through specific hardware and software components.

[0965] 1. Collection of speech data

[0966] User speaks

[0967] In a remote conference, users speak through their voices, and the voice data is collected through the microphones on their devices and transmitted to a server in real time.

[0968] In text chat, users send text by typing, and all this data is sent to the server in real time and temporarily stored.

[0969] 2. Speech-to-text

[0970] The server analyzes the voice

[0971] The server converts the received voice data into text using a natural language processing engine. For example, if you say, "Let's talk about the schedule for the next project," the content will be converted into text data.

[0972] 3. Check the consistency of the topic of discussion

[0973] Server refers to subject

[0974] The server accesses a database that stores meeting agendas and past meeting contents, and by reading the current topic of discussion, it can check for matches with the collected text data.

[0975] For example, if the current topic is "project schedule," the server checks whether the collected comments are on this topic.

[0976] 4. Derailment Detection

[0977] The server detects the derailment

[0978] The server uses an AI model to check whether the collected text data is straying from the topic of the discussion, and if it is determined to be a deviation, it will detect this in real time.

[0979] For example, if a comment about "the recent weather" is detected while a conversation about "the project schedule" is taking place, this will be considered a digression.

[0980] 5. Generating and Displaying Alerts

[0981] The server generated a warning

[0982] When the server detects a digression, it generates an appropriate warning message and sends it to the speaker's terminal, informing the speaker that his / her remarks are off-topic.

[0983] For example, a warning may be generated that reads, "This is off topic for the current discussion. Please talk about the schedule."

[0984] The device displays a warning

[0985] The device will display a warning message as a pop-up on the user's screen, allowing the user to realize that their comment has gone off-topic and correct it.

[0986] For example, a message may appear on the user's screen saying, "The discussion has gone off track. Please return to the correct topic."

[0987] Specific examples

[0988] Remote meeting example

[0989] User A's comment

[0990] User A says, "I'd like to talk about the schedule for the next project." The voice data is sent from the device to the server and converted into text.

[0991] User B's comment

[0992] After a short discussion, User B says, "I'd like to talk about the weather recently." This statement is also sent as voice data from the device to the server and converted into text.

[0993] Derailment detection and warning

[0994] The server detects that the comment "about the recent weather" does not match the topic of the discussion. Based on this information, the server generates a warning message and sends it to User B's device. User B's device then displays the warning message "The discussion has gone off track. Please return to the correct topic."

[0995] Correction statement

[0996] User B checks the warning message and corrects his statement by saying, "Got it. I'll get back to the topic of the schedule." The corrected statement is sent to the server again and reflected in the conference.

[0997] Text chat example

[0998] User A's comment

[0999] User A says in text chat, "Let's talk about the progress of the next project." The text data is sent from the device to the server and analyzed.

[1000] User B's comment

[1001] User B says in text chat, "I want to talk about a movie I saw recently." This text data is also sent to the server and analyzed.

[1002] Derailment detection and warning

[1003] The server detects that the comment "about movies" does not match the topic of the discussion. The server generates a warning message and sends it to User B's device. User B's device displays the message "The discussion has gone off track. Please return to the correct topic."

[1004] Correction statement

[1005] User B checks the warning message and corrects his comment by saying, "Got it. I'll get back to the topic of progress." The corrected comment is sent to the server again and reflected in the chat.

[1006] In this way, the system of the present invention is an effective tool for preventing discussions from going off track and supports smooth communication.

[1007] The processing flow will be explained below.

[1008] Step 1:

[1009] The user speaks. In the case of a remote conference, the user speaks aloud and the audio data is collected through the device's microphone. In the case of text chat, the user inputs text and the input contents are sent from the device to the server.

[1010] Step 2:

[1011] The device sends the collected voice data to the server in real time. In the case of text chat, the text data entered by the user is sent to the server as is.

[1012] Step 3:

[1013] The server converts the received voice data into text using a natural language processing engine. For example, the text "Let's talk about the schedule for the next project" is extracted from the voice data. In the case of text chat, this text conversion step is not necessary.

[1014] Step 4:

[1015] The server accesses the meeting topic database and retrieves the current discussion topic, for example, "the schedule for the next project" is recognized as the current discussion topic.

[1016] Step 5:

[1017] The server checks the converted or submitted text against the current topic of discussion. It compares the text with the topic to see if it is consistent.

[1018] Step 6:

[1019] The server uses an AI model to check whether the converted or submitted text strays from the topic of the discussion. For example, if a statement like "Let's talk about the weather these days" doesn't match the topic, it's considered a digression.

[1020] Step 7:

[1021] If the server detects a digression, it will generate an appropriate warning message, for example, "We are going off topic for the current discussion. Please talk about the schedule."

[1022] Step 8:

[1023] The server generates a warning message and sends it to the speaker's terminal, which then displays it as a pop-up on the user's screen in real time.

[1024] Step 9:

[1025] The user sees the warning message and realizes that their comment is off-topic. The user corrects their comment and re-comments on the correct topic.

[1026] Step 10:

[1027] The corrected statement is sent again from the terminal to the server, where it is checked for consistency again and reflected in the discussion.

[1028] Example 1

[1029] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1030] In remote meetings and text chats, discussions can easily stray from the topic, making it difficult to maintain consistency. If this problem is not resolved, meeting efficiency may decrease and important discussions may be overlooked. Since it is difficult to receive direct feedback like in face-to-face meetings, particularly in remote environments, effective methods for maintaining consistency in discussions are required.

[1031] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1032] In this invention, the server includes a means for a user to make a statement, a means for a terminal to send statement data, a means for the server to receive voice data, a means for the server to convert voice data to text, a means for the server to refer to agenda data, a means for the server to confirm that the speech data matches the agenda, a means for the server to analyze text data, a means for the server to detect deviations from the topic, a means for the server to generate a warning message, and a means for the terminal to display the warning message. This makes it possible to display a warning in real time even if a statement deviates from the topic, thereby maintaining the consistency of the discussion.

[1033] "User" refers to an individual participant who uses the system to participate in a remote conference or text chat and make a statement.

[1034] A "terminal" refers to an electronic device used by a user, such as a computer, smartphone, or tablet, that collects utterance data and transmits it to a server.

[1035] "Server" refers to a central processing unit that receives speech data, converts voice data to text, and checks the consistency of speech by referencing the agenda data.

[1036] "Utterance data" refers to data of voice utterances and text inputs made by a user during a remote conference or text chat.

[1037] "Audio data" refers to digital audio files of a user's spoken words collected through the device's microphone.

[1038] "Text" refers to voice data that has been analyzed by the server using a natural language processing engine and converted into text data.

[1039] "Agenda data" refers to the themes set as the subject of meetings and chats, and information stored in a database as the contents of past meetings.

[1040] "Conformance verification" refers to the process by which the server verifies that the converted text data is in line with the current agenda data.

[1041] "Derailment detection" refers to the process by which the server uses AI models to determine whether a comment is off-topic.

[1042] "Warning message" refers to a message generated by the server when a digression is detected, informing the user that the speech is off-topic.

[1043] "Real-time display" refers to the function of instantly displaying a warning message on the user's device immediately after a comment or text is entered.

[1044] A specific embodiment of the present invention for maintaining consistency in discussions in remote conferences and text chats will be described below.

[1045] First, a user uses a terminal as a means of making a statement. Terminals include computers, smartphones, tablets, etc., and these devices are equipped with input devices such as microphones and keyboards. The terminal collects the user's voice statements and text inputs and sends them to a server.

[1046] The server receives the audio data and converts it to text using a natural language processing engine. For example, it can use the Google Cloud Speech-to-Text API to quickly convert the audio data to text. The converted text is then used to check whether it matches the agenda data.

[1047] The agenda data is stored in the server's database and includes the subject of the meeting and the contents of past meetings. The server accesses this database to obtain the current agenda and then verifies that it matches the user's speech data.

[1048] Additionally, the server uses AI models (e.g., GPT-4) to analyze the collected text data to determine whether it deviates from the agenda. Specifically, if a comment is off-topic, it is considered a digression. For example, if someone mentions the "recent weather" while talking about the "project schedule," this is considered a digression.

[1049] When the server detects a deviation from the topic, it generates an appropriate warning message. This warning message is sent to the speaker's device. When the speaker's device receives the warning message, it displays it on the screen as a pop-up in real time. This allows the user to recognize that their comment has strayed from the topic of the discussion and to correct it.

[1050] For example, consider the following prompt:

[1051] Remote meeting prompts:

[1052] "As we conduct remote meetings to discuss the schedule for the next project, I want to create a system that will alert me if the discussion strays from the topic."

[1053] Text chat prompt:

[1054] "When we're text chatting about the next project, I want to create a system that will warn us if the conversation goes off topic."

[1055] In this way, the system of the present invention monitors user comments in real time and serves as an effective means of maintaining a consistent discussion, thereby improving the efficiency and productivity of remote meetings and text chats.

[1056] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1057] Step 1:

[1058] A user makes a statement. In a meeting or chat, the user makes a voice statement using the device's microphone, or inputs a text message using the keyboard. The input data can be audio data (e.g., a .wav file) or text data (e.g., a .txt file).

[1059] Step 2:

[1060] The device transmits speech data. The device transmits voice data and text data collected from the user to the server in real time. The device uploads the data to the server via the network.

[1061] Step 3:

[1062] The server receives the voice data. The server receives the voice data sent from the terminal. At this point, the input is voice data, and the output is saved as voice data in the server.

[1063] Step 4:

[1064] The server converts the voice data into text. The server uses a natural language processing engine (e.g., Google Cloud Speech-to-Text API) to convert the received voice data into text. For example, it takes "audio_data.wav" as input, converts it, and outputs the text "Let's talk about the schedule for the next project."

[1065] Step 5:

[1066] The server looks up the agenda data. The server queries the database to get the agenda information for the current meeting. The input is the query information, and the output is the agenda data, such as "Project Schedule."

[1067] Step 6:

[1068] The server checks whether the speech data matches the agenda. It takes text data as input and compares it with the agenda data. This check determines whether the speech is in line with the agenda. The output is a flag indicating whether it matches or does not match.

[1069] Step 7:

[1070] The server analyzes the text data. Using an AI model (e.g., GPT-4), the server analyzes the text data and checks whether the speech is on topic. The input is the text data, and the output is the analysis result.

[1071] Step 8:

[1072] The server detects digressions. Based on the analysis results, the server detects comments that stray from the topic. The input is the analysis results, and the output is a flag indicating that the comment has deviated from the topic.

[1073] Step 9:

[1074] The server generates a warning message. If a digression is detected, the server generates an appropriate warning message. The input is a digression flag, and the output is a warning message such as "We are going off topic of the current discussion. Please talk about the schedule."

[1075] Step 10:

[1076] The terminal displays the warning message. The terminal displays the received warning message in real time. The input is the warning message, and the output is a pop-up on the screen. The user can check it and correct the comment.

[1077] (Application example 1)

[1078] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1079] In factories, communication between workers and machines is an important factor that directly affects work efficiency and safety. However, inconsistent work instructions and reports can lead to work delays and mistakes, which can lead to reduced productivity and safety issues. Furthermore, when multiple workers are giving instructions to each other at the same time, discussions and instructions tend to go off track, and an effective method to prevent this is needed. Current systems have the challenge of making it difficult to issue warnings at the appropriate time or to correct the course of discussions.

[1080] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1081] In this invention, the server includes means for collecting utterance data, means for converting the utterance data into text, means for storing the topic of the current discussion, means for comparing the consistency of the converted text with the topic of the current discussion, means for detecting inconsistent utterances, means for displaying a warning to the speaker, means for checking whether instructions within the factory are consistent, means for analyzing the collected text and detecting content that deviates from the set work topic, and means for displaying a warning through the smart glasses. This makes it possible to monitor the consistency of instructions and reports in real time and quickly issue a warning if there is a deviation, thereby maintaining efficient and safe communication within the factory.

[1082] "Utterance data" refers to data that refers to the content of communication through voice or text.

[1083] "Text conversion means" refers to a system or algorithm used to convert audio data into written information.

[1084] "Means for storing the current discussion topic" refers to a system or storage that saves the topic of a meeting or instruction and retains it for future reference.

[1085] "Consistency comparison measures" refer to systems or algorithms that check whether the converted text deviates from the current subject matter.

[1086] "Incoherent speech detection" refers to systems or algorithms that automatically identify when discussions or instructions stray from the current topic.

[1087] "Means for displaying a warning" refers to a display or alert system that notifies the speaker when inconsistent speech is detected.

[1088] "Means to ensure consistency of instructions within the factory" refers to systems or algorithms that monitor whether instructions and reports in factory work deviate from the established work theme.

[1089] "Means for analyzing collected text" refers to systems or algorithms used to analyze text data and assess its content consistency and relevance.

[1090] "Means for detecting deviations from the scope of work" refers to systems or algorithms that detect when work instructions or reports deviate from the scope of the set work.

[1091] "Means for displaying a warning through smart glasses" refers to a system or application that uses smart glasses to display a warning to the speaker in real time.

[1092] Specific embodiments of the present invention will be described below.

[1093] System Configuration

[1094] The system of the present invention consists of the main functions of collecting, analyzing, and detecting speech data, and generating and displaying warnings. To achieve these functions, we use a server, smart glasses, speech recognition software, and a generative AI model.

[1095] Collection of speech data

[1096] The server collects voice data in real time from factory workers speaking through smart glasses. The smart glasses use a microphone to collect voice data and send it to the server, ensuring that the server always has the latest voice data.

[1097] Speech-to-text

[1098] The server converts the collected voice data into text data using "speech_recognition" software. For example, if a worker says, "Maintenance on this machine will be performed at 2 p.m.", the voice data is converted into text data, which is then used for further analysis.

[1099] Recollection of the current topic of discussion

[1100] The server uses a database to store the current task or topic of discussion. This database stores the set task topics and past records, allowing the collected text data to be compared with the current topic.

[1101] Checking consistency and detecting deviations

[1102] The server analyzes the converted text data using a generative AI model using the "transformers" library and checks for consistency with the current topic. This step checks whether the discussion or instructions deviate from the set task topic. If a deviation is detected, it is deemed to be a digression.

[1103] For example, if a worker says, "What time do we take our coffee break?", the system detects that this statement deviates from the current topic of machine maintenance procedures.

[1104] Generating and Displaying Alerts

[1105] If the server detects a deviation, it generates an appropriate warning message and sends it to the smart glasses, which then display the warning as a pop-up message to notify the worker, for example, "The discussion has deviated. Please return to the correct topic."

[1106] This allows the worker to review the warning and correct their statement, which is also sent to the server in real time, and the process continues until consistency is confirmed.

[1107] Prompt Sentence Examples

[1108] Below is an example of a prompt sentence to be input to the generative AI model.

[1109] Make sure instructions are consistent within your factory.

[1110] An example of text input is:

[1111] Maintenance on this machine will be done at 2 p.m. By the way, what time do you take your coffee break?

[1112] This system allows communication within the factory to be maintained consistently, efficiently and safely.

[1113] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1114] Step 1:

[1115] The user speaks through the smart glasses. The microphone in the smart glasses collects the voice data and transmits it to the server in real time. The user's speech arrives at the server as voice data.

[1116] input:

[1117] User voice data

[1118] Data processing:

[1119] Audio data collection

[1120] output:

[1121] Collected audio data

[1122] Specific behavior:

[1123] The user says, "Maintenance on this machine will be performed at 2 p.m." This speech is collected by the microphone in the smart glasses and sent to the server.

[1124] Step 2:

[1125] The server uses speech recognition software (speech_recognition) to convert the voice data into text data, which is then stored on the server.

[1126] input:

[1127] Collected audio data

[1128] Data Calculation:

[1129] Converting audio data to text

[1130] output:

[1131] Converted text data

[1132] Specific behavior:

[1133] The server uses voice recognition software to generate and store the text data: "Maintenance on this machine will be performed at 2:00 p.m."

[1134] Step 3:

[1135] The server retrieves the current discussion topic from the database, and this topic is compared with the converted text data.

[1136] input:

[1137] Current topic of discussion, converted text data

[1138] Data Calculation:

[1139] Thematic extraction and comparison with text data

[1140] output:

[1141] Data on the current subject

[1142] Specific behavior:

[1143] The server retrieves the current discussion topic "machine maintenance procedures" from the database and prepares it for analysis.

[1144] Step 4:

[1145] The server uses a generative AI model (e.g., the transformers library) to analyze the transformed text data for consistency with the current topic of discussion.

[1146] input:

[1147] Converted text data, current subject

[1148] Data Calculation:

[1149] Analyzing text data and checking consistency

[1150] output:

[1151] Consistency analysis results

[1152] Specific behavior:

[1153] The server uses a generative AI model to compare the input text "Maintenance on this machine will be performed at 2pm" with the subject "Machine maintenance procedures."

[1154] Step 5:

[1155] Based on the results of the consistency analysis, the server generates a warning message if the text data deviates from the topic of the current discussion.

[1156] input:

[1157] Consistency analysis results

[1158] Data processing:

[1159] Generate a warning message

[1160] output:

[1161] Warning message

[1162] Specific behavior:

[1163] The server detects that the utterance "What time do you take your coffee break?" is off-topic and generates a warning message saying "The discussion is going off topic. Please get back on topic."

[1164] Step 6:

[1165] The server sends the generated warning message to the smart glasses, which display it to the speaker in real time.

[1166] input:

[1167] Warning message

[1168] Data processing:

[1169] Sending, receiving, and displaying alert messages

[1170] output:

[1171] Displaying a warning message

[1172] Specific behavior:

[1173] A warning message is sent from the server to the smart glasses to notify the user that their speech is deviating. The smart glasses display shows a real-time message saying, "The discussion is going off topic. Please return to the correct topic."

[1174] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1175] The present invention combines a discussion support system with an emotion engine, and is a system that takes into account the emotions of users, particularly in remote conferences and text chats, prevents discussions from going off track, and maintains consistency in discussions. Specific embodiments for implementing the present invention are described below.

[1176] System Configuration

[1177] The system consists of the following main functions: speech data collection, analysis, emotion recognition and detection, and warning generation and display. Each function is realized through specific hardware and software components.

[1178] 1. Collection of speech data

[1179] User speaks

[1180] In a remote conference, users speak through their voices, and the voice data is collected through the microphones on their devices and transmitted to a server in real time.

[1181] In text chat, a user inputs text, which is then sent from the terminal to the server.

[1182] 2. Speech-to-text

[1183] The server analyzes the voice

[1184] The server converts the received voice data into text using a natural language processing engine. For example, if you say, "Let's talk about the schedule for the next project," the content will be converted into text data.

[1185] 3. Emotional Data Collection and Analysis

[1186] The server uses the emotion engine

[1187] The server uses an emotion engine to recognize user emotions from voice and text data, for example, identifying emotions such as "joy" or "anger" from voice tone and text keywords.

[1188] 4. Check the consistency of the topic of discussion

[1189] Server refers to subject

[1190] The server accesses a database that stores meeting agendas and past discussions, reads the current topic of the discussion, and compares the collected text data with the emotion data to check for consistency.

[1191] 5. Derailment Detection

[1192] The server detects the derailment

[1193] The server uses AI models to check whether the collected text and sentiment data deviates from the topic of the discussion. For example, if a conversation about the "project schedule" is interrupted by a mention of the "recent weather," it will be deemed a digression.

[1194] 6. Generating and Displaying Alerts

[1195] The server generated a warning

[1196] When the server detects a deviation, it generates an appropriate warning message, whose content and format are adjusted based on the user's emotional state. For example, if the user feels angry, it generates a more gentle warning.

[1197] The device displays a warning

[1198] The device will display a warning message as a pop-up on the user's screen, allowing the user to realize that their comment has gone off topic and correct it. For example, a message such as "The discussion has gone off topic. Please return to the correct topic."

[1199] Specific examples

[1200] Remote meeting example

[1201] User A's comment

[1202] User A says, "I'd like to talk about the schedule for the next project." The voice data is sent from the device to the server and converted into text.

[1203] User B's comment

[1204] After a short discussion, User B says, "I'd like to talk about the weather recently." This statement is also sent as voice data from the device to the server and converted into text.

[1205] Sentiment analysis and digression detection

[1206] The server detects that the comment "recent weather" does not match the topic of the discussion, and further recognizes the emotion of "joy" from User B's tone of voice. Based on this information, it determines that the discussion is derailing.

[1207] Generating and Displaying Alerts

[1208] The server generates a warning message saying, "This is off topic. Please talk about your schedule." and sends it to the terminal in a gentler tone to match the emotions of User B. User B's terminal displays this warning.

[1209] Correction statement

[1210] User B checks the warning message and corrects his statement by saying, "Got it. I'll get back to the topic of the schedule." The corrected statement is sent to the server again and reflected in the conference.

[1211] Text chat example

[1212] User A's comment

[1213] User A says in text chat, "Let's talk about the progress of the next project." The text data is sent from the device to the server and analyzed.

[1214] User B's comment

[1215] User B says in text chat, "I want to talk about a movie I saw recently." The server receives this text and analyzes it.

[1216] Sentiment analysis and digression detection

[1217] The server detects that the comment "about movies" does not match the topic of the discussion, and further recognizes the emotion of "enjoyment" from the text content of User B. Based on this, it is determined that the conversation has gone off topic.

[1218] Generating and Displaying Alerts

[1219] The server generates a warning message saying, "This is off topic. Please bring the discussion back to progress." and sends it to the terminal in a gentle manner that matches the emotions of User B. User B's terminal displays this warning.

[1220] Correction statement

[1221] User B checks the warning message and corrects his comment by saying, "Got it. I'll get back to the topic of progress." The corrected comment is sent to the server again and reflected in the chat.

[1222] In this way, the system of the present invention prevents discussions from going off track and supports smooth communication by providing appropriate feedback taking into account the user's emotions.

[1223] The processing flow will be explained below.

[1224] Step 1:

[1225] The user speaks. In the case of a remote conference, the user speaks aloud and the audio data is collected through the device's microphone. In the case of text chat, the user inputs text and the content is sent from the device to the server.

[1226] Step 2:

[1227] The device sends the collected voice data to the server in real time. In the case of text chat, the text data entered by the user is sent to the server as is.

[1228] Step 3:

[1229] The server converts the received voice data into text using a natural language processing engine. For example, the text "Let's talk about the schedule for the next project" is extracted from the voice data. In the case of text chat, this text conversion step is not necessary.

[1230] Step 4:

[1231] The server inputs voice and text data into an emotion engine to analyze the user's emotions. For example, emotions such as "joy" or "anger" are recognized from the tone and tempo of the voice and from keywords in the text.

[1232] Step 5:

[1233] The server accesses the agenda database and identifies the current topic of discussion, for example, recognizing that the current topic is "project schedule."

[1234] Step 6:

[1235] The server compares the converted text with the current discussion topic to ensure consistency. For example, does the statement "Let's talk about the next step" fit into the topic "Schedule for the next project"?

[1236] Step 7:

[1237] The server uses an AI model to check whether the converted text and analyzed sentiment data deviate from the topic of the discussion. For example, a statement like "Let's talk about the weather recently" is detected as inconsistent with the topic.

[1238] Step 8:

[1239] When the server detects a deviation, it generates a warning message that takes into account the user's emotions. For example, if the user is angry, it will choose a softer expression such as "Let's talk a little more calmly."

[1240] Step 9:

[1241] The server generates a warning message and sends it to the speaker's terminal, which then displays the warning message as a pop-up on the user's screen.

[1242] Step 10:

[1243] The user sees the warning message and realizes that their comment is off-topic. The user corrects their comment and re-comments on the correct topic.

[1244] Step 11:

[1245] The corrected statement is sent from the device to the server again, where it is checked for consistency and reflected in the discussion. For example, a corrected statement such as "I understand. I'll come back to the schedule for the next project" is made, allowing the discussion to proceed smoothly.

[1246] In this way, the system can prevent discussions from going off track and maintain consistency. Also, by taking into account the user's feelings, the system can provide appropriate feedback and support smooth communication.

[1247] Example 2

[1248] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1249] In remote meetings and text chats, it is difficult to maintain consistency while preventing discussions from going off track, and it is necessary to provide feedback that takes users' emotions into consideration. Furthermore, if a discussion goes off track, users may not notice it, which can hinder smooth communication.

[1250] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1251] In this invention, the server includes means for collecting utterance data, means for converting voice data into text, means for recognizing emotions from the text data and voice data, means for storing the topic of the current discussion, means for comparing the consistency of the converted text with the topic of the current discussion, means for detecting inconsistent utterances, means for generating a warning adjusted based on the emotional state of the speaker, and means for displaying the warning to the speaker. This allows for smooth communication by providing appropriate feedback that takes into account the user's emotions while maintaining the consistency of the discussion.

[1252] "Utterance data" refers to data in the form of voice and text uttered by a user.

[1253] "Audio data" refers to data that is a digital representation of the audio signal emitted by a user through a microphone.

[1254] "Text data" refers to data consisting of character information that has been converted from voice data using a natural language processing engine.

[1255] "Emotion Recognition" refers to techniques and systems for identifying a user's emotional state analyzed from speech and text data.

[1256] "Topic of discussion" refers to a specific topic or theme discussed in a meeting or text chat.

[1257] "Coherence" refers to the level of continuity and relevance that assesses whether the current statement matches the topic of discussion.

[1258] "Digression" refers to comments or actions that stray from the topic of the discussion.

[1259] A "warning message" refers to a notification to inform the user that the discussion has deviated from the topic.

[1260] "Pop-up display" refers to a type of notification format that is displayed on the user's screen in real time.

[1261] An "emotion engine" refers to software and algorithms that analyze and recognize user emotions from voice and text.

[1262] "Server" refers to a central processing unit that analyzes voice and text data, checks the consistency of discussions, recognizes emotions, generates alerts, and so on.

[1263] "Terminal" refers to a device that allows a user to input voice or text and displays alert messages.

[1264] System Configuration

[1265] This invention is a system that supports user speech in remote conferences and text chats. This system includes means for collecting user speech data, converting voice data into text, recognizing emotions, checking the consistency of discussions, and displaying warning messages for off-topic speech. Specific operations and examples are described in detail below.

[1266] Collection of speech data

[1267] User speaks

[1268] In a remote conference, users speak through a microphone. For example, user A might say, "I'd like to talk about the schedule for the next project." The speech is collected as audio data by the device's microphone and sent to the server in real time. In the case of text chat, the text entered by the user is sent directly from the device to the server.

[1269] Speech-to-text

[1270] The server analyzes the voice data

[1271] The server converts the voice data into text data using a natural language processing engine (e.g., Google Cloud Speech-to-Text). For example, a statement such as "I would like to talk about the schedule for the next project" is converted into text data.

[1272] Emotion data collection and analysis

[1273] The server uses the emotion engine

[1274] The server uses an emotion engine (e.g., IBM Watson Tone Analyzer) to recognize the user's emotion from voice and text data. For example, it can identify the user's emotion as "joy" or "anger" from the tone of the voice or keywords in the text.

[1275] Checking the consistency of the topic of discussion

[1276] The server references the database

[1277] The server accesses a database that stores meeting agendas and past discussions, and reads the current topic of the discussion. For example, it extracts information about the "schedule for the next project." It then compares the collected text data with the emotion data to check whether the discussion is consistent.

[1278] Derailment detection

[1279] The server uses AI models to detect derailments

[1280] The server uses a generative AI model (e.g., GPT-4) to check whether the current discussion is deviating from the topic of the discussion. For example, if a comment about "recent weather" is detected while talking about "project schedule," it will consider this a digression.

[1281] Generating and Displaying Alerts

[1282] Server generates warning message

[1283] When the server detects a deviation, it generates an appropriate warning message, the content and format of which are adjusted based on the user's emotional state. For example, if the user feels angry, it generates a more gentle warning.

[1284] The device displays a warning message

[1285] The device will display a warning message as a pop-up on the user's screen, allowing the user to realize that their comment has gone off topic and correct it. For example, a message such as "The discussion has gone off topic. Please return to the correct topic" may be displayed.

[1286] Feedback and Corrections

[1287] The user confirms the warning message and corrects their comment.

[1288] The user confirms the warning message and corrects their comment to return to the topic of the discussion. For example, User B might say, "Got it. Let's get back to the topic of the schedule." This corrected comment data is sent back from the device to the server and reflected in the content of the meeting or chat. This maintains the consistency of the discussion.

[1289] Prompt Sentence Examples

[1290] "In your next discussion, please use an example of a meeting using this system. For example, explain in detail how the system generates a warning when User A speaks about the topic of the meeting and User B strays from the topic, and how User B responds to it."

[1291] In this way, the system can support smooth communication by maintaining consistency in the user's speech and providing appropriate feedback that takes emotions into consideration.

[1292] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1293] Step 1:

[1294] User speaks

[1295] How it works: Users speak through a microphone in a remote meeting, or, in the case of text chat, type text.

[1296] Input: Voice data (for remote meetings) or text data (for text chat)

[1297] Output: Audio data is collected from the microphone and sent to the device. Text data is sent directly to the device.

[1298] Step 2:

[1299] The device sends the voice data to the server

[1300] Operation: Collected voice data or entered text data is sent from the device to the server in real time.

[1301] Input: Audio or text data

[1302] Output: The server receives the audio or text data.

[1303] Step 3:

[1304] The server converts the voice data into text

[1305] How it works: The server converts the audio data into text using a natural language processing engine (e.g., Google Cloud Speech-to-Text).

[1306] Input: Audio data

[1307] Output: Text data generated by the natural language processing engine

[1308] Step 4:

[1309] The server collects and analyzes emotion data

[1310] How it works: The server uses an emotion engine (e.g., IBM Watson Tone Analyzer) to analyze the user's emotions from text and voice data.

[1311] Input: Text or audio data

[1312] Output: User's emotional state (e.g., happy, angry, surprised, etc.)

[1313] Step 5:

[1314] The server checks the consistency of the discussion topic

[1315] How it works: The server accesses a database that stores meeting agendas and past discussions, and retrieves the current topic of discussion.

[1316] Input: Parsed text and sentiment data

[1317] Output: Evaluation of the topical coherence of the discussion

[1318] Step 6:

[1319] The server detects the derailment

[1320] How it works: The server uses a generative AI model to ensure that the text and sentiment data stays on topic.

[1321] Input: Text data, sentiment data, and discussion topics

[1322] Output: Derailment status (e.g., derailed, not derailed)

[1323] Step 7:

[1324] Server generates warning message

[1325] BEHAVIOR: If the server detects a digression, it generates an appropriate warning message that is tailored based on the user's emotional state.

[1326] Input: Derailment detection results and user emotion data

[1327] Output: Adjusted warning message

[1328] Step 8:

[1329] The device displays a warning message

[1330] How it works: The device will pop up a warning message on the user's screen in real time, letting them know they've gone off topic.

[1331] Input: The warning message sent by the server

[1332] Output: A popup warning message is displayed on the user's screen.

[1333] Step 9:

[1334] User makes correction comment

[1335] Action: The user acknowledges the warning message and edits their comment to bring it back on topic.

[1336] Input: Correction statement by the user who confirmed the warning message

[1337] Output: The corrected utterance data is sent from the device to the server.

[1338] Step 10:

[1339] The server reflects the revised comments in the conference

[1340] How it works: The server receives the corrected speech data and reflects it in the discussions in meetings and chats to maintain consistency.

[1341] Input: Corrected utterance data

[1342] Output: Updated meeting and chat discussions

[1343] (Application example 2)

[1344] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1345] In remote meetings and text chats, there is a need for a system that can take into account the emotions of the speaker, prevent discussions from going off track, and maintain consistency. However, current systems lack sufficient emotional analysis, making it difficult to generate flexible warning messages that reflect the speaker's emotions. Furthermore, in workplaces where effective communication is required, such as factories, it is important to analyze emotions in real time and provide appropriate feedback.

[1346] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting utterance data, means for converting the utterance data into text, means for storing the topic of the current discussion, means for comparing the consistency of the converted text with the topic of the current discussion, means for detecting inconsistent utterances, means for displaying a warning to the speaker, means for analyzing emotions from the collected utterance data, and means for adjusting the content and format of the warning message based on the emotion analysis results. This makes it possible to prevent the discussion from going off track and provide appropriate feedback that takes the speaker's emotions into consideration.

[1347] The "means for collecting speech data" is a function that records speech or text spoken by users in real time and transmits it to a server.

[1348] The "means for converting speech data into text" is a function that converts collected speech data into text data using speech recognition technology.

[1349] "Means for storing the current discussion topic" is a function that stores the purpose and topic of a meeting or discussion in a database and makes it accessible as needed.

[1350] "Means for comparing the consistency of the converted text with the topic of the current discussion" is a function that checks whether the content of the speech converted into text is consistent with the topic of the current discussion.

[1351] "Means for detecting inconsistent statements" is a function that automatically identifies statements that deviate from the topic of discussion.

[1352] The "means for displaying a warning to the speaker" is a function that notifies the speaker visually or audibly of a warning when a deviation in the discussion is detected.

[1353] "Means for analyzing emotions from collected speech data" refers to a function that identifies the speaker's emotions from the tone of the voice and the vocabulary of the text.

[1354] "Means for adjusting the content and format of warning messages based on the results of emotion analysis" refers to a function that generates warning messages with appropriate tone and expression according to the analyzed emotions.

[1355] System program generation and explanation

[1356] In this invention, a system is implemented in which factory robots effectively support discussions. The specific configuration and operating principle of this system are described below.

[1357] Hardware used

[1358] Microphone: Used to collect what the workers say.

[1359] Robot Display: Used to display a warning message to the speaker.

[1360] Server: Performs various processes such as converting speech to text, checking the consistency of discussions, and sentiment analysis.

[1361] Software used

[1362] Python: Used as a programming language.

[1363] speech_recognition library: Used to convert speech to text.

[1364] text2emotion library: Used to analyze emotions from collected text.

[1365] The transformers library pipeline: used to provide sentiment analysis models.

[1366] Data processing and calculation

[1367] 1. Collecting speech data:

[1368] The server uses a microphone to collect voice data spoken by factory workers to the robot, and transmits the voice data to the server in real time.

[1369] 2. Speech to text transcription:

[1370] The server converts the speech data into text data using the speech_recognition library. For example, the speech "Proceed to install the next bolt" is converted into text.

[1371] 3. Emotion data collection and analysis:

[1372] The server uses the text2emotion and transformers libraries in a pipeline to analyze emotions from the collected speech data, for example, identifying emotions such as "urgency" or "excitement" from the tone of voice and text.

[1373] 4. Agenda consistency check:

[1374] The server accesses a database that stores meeting agendas and past instructions, and compares the current agenda with the collected text data to check for consistency and determine whether the discussion is on topic.

[1375] 5. Derailment detection:

[1376] The server uses an AI model to check whether comments stray from the topic of the discussion. For example, if someone starts discussing lunch while working, it will recognize this as a digression.

[1377] 6. Generating and displaying alerts:

[1378] When the server detects a deviation, it generates a warning message based on the emotion analysis results. For example, if the emotion analysis result is "enjoyment," it generates a gentle warning saying, "The discussion is derailing. Please return to the correct topic." The warning message is displayed on the robot display.

[1379] Specific examples

[1380] 1. Example 1:

[1381] Factory worker A says to the robot, "Let's proceed to installing the next bolt." The voice data is sent to the server and converted into text data.

[1382] After a while, worker B says, "I'd like to talk about lunch." This utterance is also collected as audio data and converted into text.

[1383] The server analyzes the combined data and determines that the "about lunch" statement does not match the agenda. Sentiment analysis also identifies the emotion of "enjoyment."

[1384] The server generates a warning message such as "The discussion is going off topic. Please return to the correct topic" and displays it on the robot display.

[1385] 2. Example prompt:

[1386] Say: "I want to talk about lunch."

[1387] Agenda: "Installing the bolts"

[1388] Current Emotion: "Excitement"

[1389] Warning message output:

[1390] "The discussion has gone off track. Please return to the correct topic."

[1391] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1392] Step 1:

[1393] Collection of speech data

[1394] When the user speaks, the robot's microphone collects the voice data, which is then sent to the server in real time. Specifically, the user might say, "Let's proceed to installing the next bolt."

[1395] Step 2:

[1396] Speech-to-text

[1397] The server converts the received voice data into text data using the speech_recognition library. The input here is voice data, and the output is the converted text data. For example, the voice data "Let's proceed to install the next bolt" is converted into text data.

[1398] Step 3:

[1399] Emotion data collection and analysis

[1400] The server uses a pipeline of the text2emotion and transformers libraries to analyze user emotions from the collected text data. The input of this step is text data, and the output is the emotion analysis result. Specifically, the emotion "hurry" is identified from the text data "Let's proceed to install the next bolt."

[1401] Step 4:

[1402] Checking the consistency of the agenda

[1403] The server accesses the database and compares the current agenda (e.g., "Install the bolt") with the collected text data (e.g., "Proceed to installing the next bolt"). The inputs in this step are the current agenda obtained from the database and the collected text data. The output is a consistency check result, specifically, "they match."

[1404] Step 5:

[1405] Derailment detection

[1406] The server uses an AI model to detect whether the content of the statement deviates from the topic of the discussion. The input for this step is the consistency check result and text data. The output is the digression detection result, which determines that the statement "I want to talk about lunch" does not match the topic of the discussion, for example.

[1407] Step 6:

[1408] Generating and Displaying Alerts

[1409] When the server detects a digression, it generates a warning message based on the emotion analysis results. The inputs to this step are the digression detection results and the emotion analysis results. The output is a warning message, specifically, a message that says, "The discussion is going off topic. Please return to the correct topic." The user is then notified via the robot display or voice.

[1410] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1411] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1412] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1413] [Fourth embodiment]

[1414] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1415] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1416] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1417] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1418] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1419] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1420] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1421] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1422] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1423] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1424] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1425] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1426] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1427] The present invention relates to a discussion support system, and in particular to a system for preventing discussions from going off track and maintaining consistency in remote conferences and text chats. Specific embodiments for carrying out the present invention will be described below.

[1428] System Configuration

[1429] The system is broadly divided into the main functions of collecting speech data, analyzing it, detecting it, and generating and displaying warnings. Each function is realized through specific hardware and software components.

[1430] 1. Collection of speech data

[1431] User speaks

[1432] In a remote conference, users speak through their voices, and the voice data is collected through the microphones on their devices and transmitted to a server in real time.

[1433] In text chat, users send text by typing, and all this data is sent to the server in real time and temporarily stored.

[1434] 2. Speech-to-text

[1435] The server analyzes the voice

[1436] The server converts the received voice data into text using a natural language processing engine. For example, if you say, "Let's talk about the schedule for the next project," the content will be converted into text data.

[1437] 3. Check the consistency of the topic of discussion

[1438] Server refers to subject

[1439] The server accesses a database that stores meeting agendas and past meeting contents, and by reading the current topic of discussion, it can check for matches with the collected text data.

[1440] For example, if the current topic is "project schedule," the server checks whether the collected comments are on this topic.

[1441] 4. Derailment Detection

[1442] The server detects the derailment

[1443] The server uses an AI model to check whether the collected text data is straying from the topic of the discussion, and if it is determined to be a deviation, it will detect this in real time.

[1444] For example, if a comment about "the recent weather" is detected while a conversation about "the project schedule" is taking place, this will be considered a digression.

[1445] 5. Generating and Displaying Alerts

[1446] The server generated a warning

[1447] When the server detects a digression, it generates an appropriate warning message and sends it to the speaker's terminal, informing the speaker that his / her remarks are off-topic.

[1448] For example, a warning may be generated that reads, "This is off topic for the current discussion. Please talk about the schedule."

[1449] The device displays a warning

[1450] The device will display a warning message as a pop-up on the user's screen, allowing the user to realize that their comment has gone off-topic and correct it.

[1451] For example, a message may appear on the user's screen saying, "The discussion has gone off track. Please return to the correct topic."

[1452] Specific examples

[1453] Remote meeting example

[1454] User A's comment

[1455] User A says, "I'd like to talk about the schedule for the next project." The voice data is sent from the device to the server and converted into text.

[1456] User B's comment

[1457] After a short discussion, User B says, "I'd like to talk about the weather recently." This statement is also sent as voice data from the device to the server and converted into text.

[1458] Derailment detection and warning

[1459] The server detects that the comment "about the recent weather" does not match the topic of the discussion. Based on this information, the server generates a warning message and sends it to User B's device. User B's device then displays the warning message "The discussion has gone off track. Please return to the correct topic."

[1460] Correction statement

[1461] User B checks the warning message and corrects his statement by saying, "Got it. I'll get back to the topic of the schedule." The corrected statement is sent to the server again and reflected in the conference.

[1462] Text chat example

[1463] User A's comment

[1464] User A says in text chat, "Let's talk about the progress of the next project." The text data is sent from the device to the server and analyzed.

[1465] User B's comment

[1466] User B says in text chat, "I want to talk about a movie I saw recently." This text data is also sent to the server and analyzed.

[1467] Derailment detection and warning

[1468] The server detects that the comment "about movies" does not match the topic of the discussion. The server generates a warning message and sends it to User B's device. User B's device displays the message "The discussion has gone off track. Please return to the correct topic."

[1469] Correction statement

[1470] User B checks the warning message and corrects his comment by saying, "Got it. I'll get back to the topic of progress." The corrected comment is sent to the server again and reflected in the chat.

[1471] In this way, the system of the present invention is an effective tool for preventing discussions from going off track and supports smooth communication.

[1472] The processing flow will be explained below.

[1473] Step 1:

[1474] The user speaks. In the case of a remote conference, the user speaks aloud and the audio data is collected through the device's microphone. In the case of text chat, the user inputs text and the input contents are sent from the device to the server.

[1475] Step 2:

[1476] The device sends the collected voice data to the server in real time. In the case of text chat, the text data entered by the user is sent to the server as is.

[1477] Step 3:

[1478] The server converts the received voice data into text using a natural language processing engine. For example, the text "Let's talk about the schedule for the next project" is extracted from the voice data. In the case of text chat, this text conversion step is not necessary.

[1479] Step 4:

[1480] The server accesses the meeting topic database and retrieves the current discussion topic, for example, "the schedule for the next project" is recognized as the current discussion topic.

[1481] Step 5:

[1482] The server checks the converted or submitted text against the current topic of discussion. It compares the text with the topic to see if it is consistent.

[1483] Step 6:

[1484] The server uses an AI model to check whether the converted or submitted text strays from the topic of the discussion. For example, if a statement like "Let's talk about the weather these days" doesn't match the topic, it's considered a digression.

[1485] Step 7:

[1486] If the server detects a digression, it will generate an appropriate warning message, for example, "We are going off topic for the current discussion. Please talk about the schedule."

[1487] Step 8:

[1488] The server generates a warning message and sends it to the speaker's terminal, which then displays it as a pop-up on the user's screen in real time.

[1489] Step 9:

[1490] The user sees the warning message and realizes that their comment is off-topic. The user corrects their comment and re-comments on the correct topic.

[1491] Step 10:

[1492] The corrected statement is sent again from the terminal to the server, where it is checked for consistency again and reflected in the discussion.

[1493] Example 1

[1494] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1495] In remote meetings and text chats, discussions can easily stray from the topic, making it difficult to maintain consistency. If this problem is not resolved, meeting efficiency may decrease and important discussions may be overlooked. Since it is difficult to receive direct feedback like in face-to-face meetings, particularly in remote environments, effective methods for maintaining consistency in discussions are required.

[1496] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1497] In this invention, the server includes a means for a user to make a statement, a means for a terminal to send statement data, a means for the server to receive voice data, a means for the server to convert voice data to text, a means for the server to refer to agenda data, a means for the server to confirm that the speech data matches the agenda, a means for the server to analyze text data, a means for the server to detect deviations from the topic, a means for the server to generate a warning message, and a means for the terminal to display the warning message. This makes it possible to display a warning in real time even if a statement deviates from the topic, thereby maintaining the consistency of the discussion.

[1498] "User" refers to an individual participant who uses the system to participate in a remote conference or text chat and make a statement.

[1499] A "terminal" refers to an electronic device used by a user, such as a computer, smartphone, or tablet, that collects utterance data and transmits it to a server.

[1500] "Server" refers to a central processing unit that receives speech data, converts voice data to text, and checks the consistency of speech by referencing the agenda data.

[1501] "Utterance data" refers to data of voice utterances and text inputs made by a user during a remote conference or text chat.

[1502] "Audio data" refers to digital audio files of a user's spoken words collected through the device's microphone.

[1503] "Text" refers to voice data that has been analyzed by the server using a natural language processing engine and converted into text data.

[1504] "Agenda data" refers to the themes set as the subject of meetings and chats, and information stored in a database as the contents of past meetings.

[1505] "Conformance verification" refers to the process by which the server verifies that the converted text data is in line with the current agenda data.

[1506] "Derailment detection" refers to the process by which the server uses AI models to determine whether a comment is off-topic.

[1507] "Warning message" refers to a message generated by the server when a digression is detected, informing the user that the speech is off-topic.

[1508] "Real-time display" refers to the function of instantly displaying a warning message on the user's device immediately after a comment or text is entered.

[1509] A specific embodiment of the present invention for maintaining consistency in discussions in remote conferences and text chats will be described below.

[1510] First, a user uses a terminal as a means of making a statement. Terminals include computers, smartphones, tablets, etc., and these devices are equipped with input devices such as microphones and keyboards. The terminal collects the user's voice statements and text inputs and sends them to a server.

[1511] The server receives the audio data and converts it to text using a natural language processing engine. For example, it can use the Google Cloud Speech-to-Text API to quickly convert the audio data to text. The converted text is then used to check whether it matches the agenda data.

[1512] The agenda data is stored in the server's database and includes the subject of the meeting and the contents of past meetings. The server accesses this database to obtain the current agenda and then verifies that it matches the user's speech data.

[1513] Additionally, the server uses AI models (e.g., GPT-4) to analyze the collected text data to determine whether it deviates from the agenda. Specifically, if a comment is off-topic, it is considered a digression. For example, if someone mentions the "recent weather" while talking about the "project schedule," this is considered a digression.

[1514] When the server detects a deviation from the topic, it generates an appropriate warning message. This warning message is sent to the speaker's device. When the speaker's device receives the warning message, it displays it on the screen as a pop-up in real time. This allows the user to recognize that their comment has strayed from the topic of the discussion and to correct it.

[1515] For example, consider the following prompt:

[1516] Remote meeting prompts:

[1517] "As we conduct remote meetings to discuss the schedule for the next project, I want to create a system that will alert me if the discussion strays from the topic."

[1518] Text chat prompt:

[1519] "When we're text chatting about the next project, I want to create a system that will warn us if the conversation goes off topic."

[1520] In this way, the system of the present invention monitors user comments in real time and serves as an effective means of maintaining a consistent discussion, thereby improving the efficiency and productivity of remote meetings and text chats.

[1521] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1522] Step 1:

[1523] A user makes a statement. In a meeting or chat, the user makes a voice statement using the device's microphone, or inputs a text message using the keyboard. The input data can be audio data (e.g., a .wav file) or text data (e.g., a .txt file).

[1524] Step 2:

[1525] The device transmits speech data. The device transmits voice data and text data collected from the user to the server in real time. The device uploads the data to the server via the network.

[1526] Step 3:

[1527] The server receives the voice data. The server receives the voice data sent from the terminal. At this point, the input is voice data, and the output is saved as voice data in the server.

[1528] Step 4:

[1529] The server converts the voice data into text. The server uses a natural language processing engine (e.g., Google Cloud Speech-to-Text API) to convert the received voice data into text. For example, it takes "audio_data.wav" as input, converts it, and outputs the text "Let's talk about the schedule for the next project."

[1530] Step 5:

[1531] The server looks up the agenda data. The server queries the database to get the agenda information for the current meeting. The input is the query information, and the output is the agenda data, such as "Project Schedule."

[1532] Step 6:

[1533] The server checks whether the speech data matches the agenda. It takes text data as input and compares it with the agenda data. This check determines whether the speech is in line with the agenda. The output is a flag indicating whether it matches or does not match.

[1534] Step 7:

[1535] The server analyzes the text data. Using an AI model (e.g., GPT-4), the server analyzes the text data and checks whether the speech is on topic. The input is the text data, and the output is the analysis result.

[1536] Step 8:

[1537] The server detects digressions. Based on the analysis results, the server detects comments that stray from the topic. The input is the analysis results, and the output is a flag indicating that the comment has deviated from the topic.

[1538] Step 9:

[1539] The server generates a warning message. If a digression is detected, the server generates an appropriate warning message. The input is a digression flag, and the output is a warning message such as "We are going off topic of the current discussion. Please talk about the schedule."

[1540] Step 10:

[1541] The terminal displays the warning message. The terminal displays the received warning message in real time. The input is the warning message, and the output is a pop-up on the screen. The user can check it and correct the comment.

[1542] (Application example 1)

[1543] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1544] In factories, communication between workers and machines is an important factor that directly affects work efficiency and safety. However, inconsistent work instructions and reports can lead to work delays and mistakes, which can lead to reduced productivity and safety issues. Furthermore, when multiple workers are giving instructions to each other at the same time, discussions and instructions tend to go off track, and an effective method to prevent this is needed. Current systems have the challenge of making it difficult to issue warnings at the appropriate time or to correct the course of discussions.

[1545] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1546] In this invention, the server includes means for collecting utterance data, means for converting the utterance data into text, means for storing the topic of the current discussion, means for comparing the consistency of the converted text with the topic of the current discussion, means for detecting inconsistent utterances, means for displaying a warning to the speaker, means for checking whether instructions within the factory are consistent, means for analyzing the collected text and detecting content that deviates from the set work topic, and means for displaying a warning through the smart glasses. This makes it possible to monitor the consistency of instructions and reports in real time and quickly issue a warning if there is a deviation, thereby maintaining efficient and safe communication within the factory.

[1547] "Utterance data" refers to data that refers to the content of communication through voice or text.

[1548] "Text conversion means" refers to a system or algorithm used to convert audio data into written information.

[1549] "Means for storing the current discussion topic" refers to a system or storage that saves the topic of a meeting or instruction and retains it for future reference.

[1550] "Consistency comparison measures" refer to systems or algorithms that check whether the converted text deviates from the current subject matter.

[1551] "Incoherent speech detection" refers to systems or algorithms that automatically identify when discussions or instructions stray from the current topic.

[1552] "Means for displaying a warning" refers to a display or alert system that notifies the speaker when inconsistent speech is detected.

[1553] "Means to ensure consistency of instructions within the factory" refers to systems or algorithms that monitor whether instructions and reports in factory work deviate from the established work theme.

[1554] "Means for analyzing collected text" refers to systems or algorithms used to analyze text data and assess its content consistency and relevance.

[1555] "Means for detecting deviations from the scope of work" refers to systems or algorithms that detect when work instructions or reports deviate from the scope of the set work.

[1556] "Means for displaying a warning through smart glasses" refers to a system or application that uses smart glasses to display a warning to the speaker in real time.

[1557] Specific embodiments of the present invention will be described below.

[1558] System Configuration

[1559] The system of the present invention consists of the main functions of collecting, analyzing, and detecting speech data, and generating and displaying warnings. To achieve these functions, we use a server, smart glasses, speech recognition software, and a generative AI model.

[1560] Collection of speech data

[1561] The server collects voice data in real time from factory workers speaking through smart glasses. The smart glasses use a microphone to collect voice data and send it to the server, ensuring that the server always has the latest voice data.

[1562] Speech-to-text

[1563] The server converts the collected voice data into text data using "speech_recognition" software. For example, if a worker says, "Maintenance on this machine will be performed at 2 p.m.", the voice data is converted into text data, which is then used for further analysis.

[1564] Recollection of the current topic of discussion

[1565] The server uses a database to store the current task or topic of discussion. This database stores the set task topics and past records, allowing the collected text data to be compared with the current topic.

[1566] Checking consistency and detecting deviations

[1567] The server analyzes the converted text data using a generative AI model using the "transformers" library and checks for consistency with the current topic. This step checks whether the discussion or instructions deviate from the set task topic. If a deviation is detected, it is deemed to be a digression.

[1568] For example, if a worker says, "What time do we take our coffee break?", the system detects that this statement deviates from the current topic of machine maintenance procedures.

[1569] Generating and Displaying Alerts

[1570] If the server detects a deviation, it generates an appropriate warning message and sends it to the smart glasses, which then display the warning as a pop-up message to notify the worker, for example, "The discussion has deviated. Please return to the correct topic."

[1571] This allows the worker to review the warning and correct their statement, which is also sent to the server in real time, and the process continues until consistency is confirmed.

[1572] Prompt Sentence Examples

[1573] Below is an example of a prompt sentence to be input to the generative AI model.

[1574] Make sure instructions are consistent within your factory.

[1575] An example of text input is:

[1576] Maintenance on this machine will be done at 2 p.m. By the way, what time do you take your coffee break?

[1577] This system allows communication within the factory to be maintained consistently, efficiently and safely.

[1578] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1579] Step 1:

[1580] The user speaks through the smart glasses. The microphone in the smart glasses collects the voice data and transmits it to the server in real time. The user's speech arrives at the server as voice data.

[1581] input:

[1582] User voice data

[1583] Data processing:

[1584] Audio data collection

[1585] output:

[1586] Collected audio data

[1587] Specific behavior:

[1588] The user says, "Maintenance on this machine will be performed at 2 p.m." This speech is collected by the microphone in the smart glasses and sent to the server.

[1589] Step 2:

[1590] The server uses speech recognition software (speech_recognition) to convert the voice data into text data, which is then stored on the server.

[1591] input:

[1592] Collected audio data

[1593] Data Calculation:

[1594] Converting audio data to text

[1595] output:

[1596] Converted text data

[1597] Specific behavior:

[1598] The server uses voice recognition software to generate and store the text data: "Maintenance on this machine will be performed at 2:00 p.m."

[1599] Step 3:

[1600] The server retrieves the current discussion topic from the database, and this topic is compared with the converted text data.

[1601] input:

[1602] Current topic of discussion, converted text data

[1603] Data Calculation:

[1604] Thematic extraction and comparison with text data

[1605] output:

[1606] Data on the current subject

[1607] Specific behavior:

[1608] The server retrieves the current discussion topic "machine maintenance procedures" from the database and prepares it for analysis.

[1609] Step 4:

[1610] The server uses a generative AI model (e.g., the transformers library) to analyze the transformed text data for consistency with the current topic of discussion.

[1611] input:

[1612] Converted text data, current subject

[1613] Data Calculation:

[1614] Analyzing text data and checking consistency

[1615] output:

[1616] Consistency analysis results

[1617] Specific behavior:

[1618] The server uses a generative AI model to compare the input text "Maintenance on this machine will be performed at 2pm" with the subject "Machine maintenance procedures."

[1619] Step 5:

[1620] Based on the results of the consistency analysis, the server generates a warning message if the text data deviates from the topic of the current discussion.

[1621] input:

[1622] Consistency analysis results

[1623] Data processing:

[1624] Generate a warning message

[1625] output:

[1626] Warning message

[1627] Specific behavior:

[1628] The server detects that the utterance "What time do you take your coffee break?" is off-topic and generates a warning message saying "The discussion is going off topic. Please get back on topic."

[1629] Step 6:

[1630] The server sends the generated warning message to the smart glasses, which display it to the speaker in real time.

[1631] input:

[1632] Warning message

[1633] Data processing:

[1634] Sending, receiving, and displaying alert messages

[1635] output:

[1636] Displaying a warning message

[1637] Specific behavior:

[1638] A warning message is sent from the server to the smart glasses to notify the user that their speech is deviating. The smart glasses display shows a real-time message saying, "The discussion is going off topic. Please return to the correct topic."

[1639] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1640] The present invention combines a discussion support system with an emotion engine, and is a system that takes into account the emotions of users, particularly in remote conferences and text chats, prevents discussions from going off track, and maintains consistency in discussions. Specific embodiments for implementing the present invention are described below.

[1641] System Configuration

[1642] The system consists of the following main functions: speech data collection, analysis, emotion recognition and detection, and warning generation and display. Each function is realized through specific hardware and software components.

[1643] 1. Collection of speech data

[1644] User speaks

[1645] In a remote conference, users speak through their voices, and the voice data is collected through the microphones on their devices and transmitted to a server in real time.

[1646] In text chat, a user inputs text, which is then sent from the terminal to the server.

[1647] 2. Speech-to-text

[1648] The server analyzes the voice

[1649] The server converts the received voice data into text using a natural language processing engine. For example, if you say, "Let's talk about the schedule for the next project," the content will be converted into text data.

[1650] 3. Emotional Data Collection and Analysis

[1651] The server uses the emotion engine

[1652] The server uses an emotion engine to recognize user emotions from voice and text data, for example, identifying emotions such as "joy" or "anger" from voice tone and text keywords.

[1653] 4. Check the consistency of the topic of discussion

[1654] Server refers to subject

[1655] The server accesses a database that stores meeting agendas and past discussions, reads the current topic of the discussion, and compares the collected text data with the emotion data to check for consistency.

[1656] 5. Derailment Detection

[1657] The server detects the derailment

[1658] The server uses AI models to check whether the collected text and sentiment data deviates from the topic of the discussion. For example, if a conversation about the "project schedule" is interrupted by a mention of the "recent weather," it will be deemed a digression.

[1659] 6. Generating and Displaying Alerts

[1660] The server generated a warning

[1661] When the server detects a deviation, it generates an appropriate warning message, whose content and format are adjusted based on the user's emotional state. For example, if the user feels angry, it generates a more gentle warning.

[1662] The device displays a warning

[1663] The device will display a warning message as a pop-up on the user's screen, allowing the user to realize that their comment has gone off topic and correct it. For example, a message such as "The discussion has gone off topic. Please return to the correct topic."

[1664] Specific examples

[1665] Remote meeting example

[1666] User A's comment

[1667] User A says, "I'd like to talk about the schedule for the next project." The voice data is sent from the device to the server and converted into text.

[1668] User B's comment

[1669] After a short discussion, User B says, "I'd like to talk about the weather recently." This statement is also sent as voice data from the device to the server and converted into text.

[1670] Sentiment analysis and digression detection

[1671] The server detects that the comment "recent weather" does not match the topic of the discussion, and further recognizes the emotion of "joy" from User B's tone of voice. Based on this information, it determines that the discussion is derailing.

[1672] Generating and Displaying Alerts

[1673] The server generates a warning message saying, "This is off topic. Please talk about your schedule." and sends it to the terminal in a gentler tone to match the emotions of User B. User B's terminal displays this warning.

[1674] Correction statement

[1675] User B checks the warning message and corrects his statement by saying, "Got it. I'll get back to the topic of the schedule." The corrected statement is sent to the server again and reflected in the conference.

[1676] Text chat example

[1677] User A's comment

[1678] User A says in text chat, "Let's talk about the progress of the next project." The text data is sent from the device to the server and analyzed.

[1679] User B's comment

[1680] User B says in text chat, "I want to talk about a movie I saw recently." The server receives this text and analyzes it.

[1681] Sentiment analysis and digression detection

[1682] The server detects that the comment "about movies" does not match the topic of the discussion, and further recognizes the emotion of "enjoyment" from the text content of User B. Based on this, it is determined that the conversation has gone off topic.

[1683] Generating and Displaying Alerts

[1684] The server generates a warning message saying, "This is off topic. Please bring the discussion back to progress." and sends it to the terminal in a gentle manner that matches the emotions of User B. User B's terminal displays this warning.

[1685] Correction statement

[1686] User B checks the warning message and corrects his comment by saying, "Got it. I'll get back to the topic of progress." The corrected comment is sent to the server again and reflected in the chat.

[1687] In this way, the system of the present invention prevents discussions from going off track and supports smooth communication by providing appropriate feedback taking into account the user's emotions.

[1688] The processing flow will be explained below.

[1689] Step 1:

[1690] The user speaks. In the case of a remote conference, the user speaks aloud and the audio data is collected through the device's microphone. In the case of text chat, the user inputs text and the content is sent from the device to the server.

[1691] Step 2:

[1692] The device sends the collected voice data to the server in real time. In the case of text chat, the text data entered by the user is sent to the server as is.

[1693] Step 3:

[1694] The server converts the received voice data into text using a natural language processing engine. For example, the text "Let's talk about the schedule for the next project" is extracted from the voice data. In the case of text chat, this text conversion step is not necessary.

[1695] Step 4:

[1696] The server inputs voice and text data into an emotion engine to analyze the user's emotions. For example, emotions such as "joy" or "anger" are recognized from the tone and tempo of the voice and from keywords in the text.

[1697] Step 5:

[1698] The server accesses the agenda database and identifies the current topic of discussion, for example, recognizing that the current topic is "project schedule."

[1699] Step 6:

[1700] The server compares the converted text with the current discussion topic to ensure consistency. For example, does the statement "Let's talk about the next step" fit into the topic "Schedule for the next project"?

[1701] Step 7:

[1702] The server uses an AI model to check whether the converted text and analyzed sentiment data deviate from the topic of the discussion. For example, a statement like "Let's talk about the weather recently" is detected as inconsistent with the topic.

[1703] Step 8:

[1704] When the server detects a deviation, it generates a warning message that takes into account the user's emotions. For example, if the user is angry, it will choose a softer expression such as "Let's talk a little more calmly."

[1705] Step 9:

[1706] The server generates a warning message and sends it to the speaker's terminal, which then displays the warning message as a pop-up on the user's screen.

[1707] Step 10:

[1708] The user sees the warning message and realizes that their comment is off-topic. The user corrects their comment and re-comments on the correct topic.

[1709] Step 11:

[1710] The corrected statement is sent from the device to the server again, where it is checked for consistency and reflected in the discussion. For example, a corrected statement such as "I understand. I'll come back to the schedule for the next project" is made, allowing the discussion to proceed smoothly.

[1711] In this way, the system can prevent discussions from going off track and maintain consistency. Also, by taking into account the user's feelings, the system can provide appropriate feedback and support smooth communication.

[1712] Example 2

[1713] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1714] In remote meetings and text chats, it is difficult to maintain consistency while preventing discussions from going off track, and it is necessary to provide feedback that takes users' emotions into consideration. Furthermore, if a discussion goes off track, users may not notice it, which can hinder smooth communication.

[1715] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1716] In this invention, the server includes means for collecting utterance data, means for converting voice data into text, means for recognizing emotions from the text data and voice data, means for storing the topic of the current discussion, means for comparing the consistency of the converted text with the topic of the current discussion, means for detecting inconsistent utterances, means for generating a warning adjusted based on the emotional state of the speaker, and means for displaying the warning to the speaker. This allows for smooth communication by providing appropriate feedback that takes into account the user's emotions while maintaining the consistency of the discussion.

[1717] "Utterance data" refers to data in the form of voice and text uttered by a user.

[1718] "Audio data" refers to data that is a digital representation of the audio signal emitted by a user through a microphone.

[1719] "Text data" refers to data consisting of character information that has been converted from voice data using a natural language processing engine.

[1720] "Emotion Recognition" refers to techniques and systems for identifying a user's emotional state analyzed from speech and text data.

[1721] "Topic of discussion" refers to a specific topic or theme discussed in a meeting or text chat.

[1722] "Coherence" refers to the level of continuity and relevance that assesses whether the current statement matches the topic of discussion.

[1723] "Digression" refers to comments or actions that stray from the topic of the discussion.

[1724] A "warning message" refers to a notification to inform the user that the discussion has deviated from the topic.

[1725] "Pop-up display" refers to a type of notification format that is displayed on the user's screen in real time.

[1726] An "emotion engine" refers to software and algorithms that analyze and recognize user emotions from voice and text.

[1727] "Server" refers to a central processing unit that analyzes voice and text data, checks the consistency of discussions, recognizes emotions, generates alerts, and so on.

[1728] "Terminal" refers to a device that allows a user to input voice or text and displays alert messages.

[1729] System Configuration

[1730] This invention is a system that supports user speech in remote conferences and text chats. This system includes means for collecting user speech data, converting voice data into text, recognizing emotions, checking the consistency of discussions, and displaying warning messages for off-topic speech. Specific operations and examples are described in detail below.

[1731] Collection of speech data

[1732] User speaks

[1733] In a remote conference, users speak through a microphone. For example, user A might say, "I'd like to talk about the schedule for the next project." The speech is collected as audio data by the device's microphone and sent to the server in real time. In the case of text chat, the text entered by the user is sent directly from the device to the server.

[1734] Speech-to-text

[1735] The server analyzes the voice data

[1736] The server converts the voice data into text data using a natural language processing engine (e.g., Google Cloud Speech-to-Text). For example, a statement such as "I would like to talk about the schedule for the next project" is converted into text data.

[1737] Emotion data collection and analysis

[1738] The server uses the emotion engine

[1739] The server uses an emotion engine (e.g., IBM Watson Tone Analyzer) to recognize the user's emotion from voice and text data. For example, it can identify the user's emotion as "joy" or "anger" from the tone of the voice or keywords in the text.

[1740] Checking the consistency of the topic of discussion

[1741] The server references the database

[1742] The server accesses a database that stores meeting agendas and past discussions, and reads the current topic of the discussion. For example, it extracts information about the "schedule for the next project." It then compares the collected text data with the emotion data to check whether the discussion is consistent.

[1743] Derailment detection

[1744] The server uses AI models to detect derailments

[1745] The server uses a generative AI model (e.g., GPT-4) to check whether the current discussion is deviating from the topic of the discussion. For example, if a comment about "recent weather" is detected while talking about "project schedule," it will consider this a digression.

[1746] Generating and Displaying Alerts

[1747] Server generates warning message

[1748] When the server detects a deviation, it generates an appropriate warning message, the content and format of which are adjusted based on the user's emotional state. For example, if the user feels angry, it generates a more gentle warning.

[1749] The device displays a warning message

[1750] The device will display a warning message as a pop-up on the user's screen, allowing the user to realize that their comment has gone off topic and correct it. For example, a message such as "The discussion has gone off topic. Please return to the correct topic" may be displayed.

[1751] Feedback and Corrections

[1752] The user confirms the warning message and corrects their comment.

[1753] The user confirms the warning message and corrects their comment to return to the topic of the discussion. For example, User B might say, "Got it. Let's get back to the topic of the schedule." This corrected comment data is sent back from the device to the server and reflected in the content of the meeting or chat. This maintains the consistency of the discussion.

[1754] Prompt Sentence Examples

[1755] "In your next discussion, please use an example of a meeting using this system. For example, explain in detail how the system generates a warning when User A speaks about the topic of the meeting and User B strays from the topic, and how User B responds to it."

[1756] In this way, the system can support smooth communication by maintaining consistency in the user's speech and providing appropriate feedback that takes emotions into consideration.

[1757] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1758] Step 1:

[1759] User speaks

[1760] How it works: Users speak through a microphone in a remote meeting, or, in the case of text chat, type text.

[1761] Input: Voice data (for remote meetings) or text data (for text chat)

[1762] Output: Audio data is collected from the microphone and sent to the device. Text data is sent directly to the device.

[1763] Step 2:

[1764] The device sends the voice data to the server

[1765] Operation: Collected voice data or entered text data is sent from the device to the server in real time.

[1766] Input: Audio or text data

[1767] Output: The server receives the audio or text data.

[1768] Step 3:

[1769] The server converts the voice data into text

[1770] How it works: The server converts the audio data into text using a natural language processing engine (e.g., Google Cloud Speech-to-Text).

[1771] Input: Audio data

[1772] Output: Text data generated by the natural language processing engine

[1773] Step 4:

[1774] The server collects and analyzes emotion data

[1775] How it works: The server uses an emotion engine (e.g., IBM Watson Tone Analyzer) to analyze the user's emotions from text and voice data.

[1776] Input: Text or audio data

[1777] Output: User's emotional state (e.g., happy, angry, surprised, etc.)

[1778] Step 5:

[1779] The server checks the consistency of the discussion topic

[1780] How it works: The server accesses a database that stores meeting agendas and past discussions, and retrieves the current topic of discussion.

[1781] Input: Parsed text and sentiment data

[1782] Output: Evaluation of the topical coherence of the discussion

[1783] Step 6:

[1784] The server detects the derailment

[1785] How it works: The server uses a generative AI model to ensure that the text and sentiment data stays on topic.

[1786] Input: Text data, sentiment data, and discussion topics

[1787] Output: Derailment status (e.g., derailed, not derailed)

[1788] Step 7:

[1789] Server generates warning message

[1790] BEHAVIOR: If the server detects a digression, it generates an appropriate warning message that is tailored based on the user's emotional state.

[1791] Input: Derailment detection results and user emotion data

[1792] Output: Adjusted warning message

[1793] Step 8:

[1794] The device displays a warning message

[1795] How it works: The device will pop up a warning message on the user's screen in real time, letting them know they've gone off topic.

[1796] Input: The warning message sent by the server

[1797] Output: A popup warning message is displayed on the user's screen.

[1798] Step 9:

[1799] User makes correction comment

[1800] Action: The user acknowledges the warning message and edits their comment to bring it back on topic.

[1801] Input: Correction statement by the user who confirmed the warning message

[1802] Output: The corrected utterance data is sent from the device to the server.

[1803] Step 10:

[1804] The server reflects the revised comments in the conference

[1805] How it works: The server receives the corrected speech data and reflects it in the discussions in meetings and chats to maintain consistency.

[1806] Input: Corrected utterance data

[1807] Output: Updated meeting and chat discussions

[1808] (Application example 2)

[1809] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1810] In remote meetings and text chats, there is a need for a system that can take into account the emotions of the speaker, prevent discussions from going off track, and maintain consistency. However, current systems lack sufficient emotional analysis, making it difficult to generate flexible warning messages that reflect the speaker's emotions. Furthermore, in workplaces where effective communication is required, such as factories, it is important to analyze emotions in real time and provide appropriate feedback.

[1811] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting utterance data, means for converting the utterance data into text, means for storing the topic of the current discussion, means for comparing the consistency of the converted text with the topic of the current discussion, means for detecting inconsistent utterances, means for displaying a warning to the speaker, means for analyzing emotions from the collected utterance data, and means for adjusting the content and format of the warning message based on the emotion analysis results. This makes it possible to prevent the discussion from going off track and provide appropriate feedback that takes the speaker's emotions into consideration.

[1812] The "means for collecting speech data" is a function that records speech or text spoken by users in real time and transmits it to a server.

[1813] The "means for converting speech data into text" is a function that converts collected speech data into text data using speech recognition technology.

[1814] "Means for storing the current discussion topic" is a function that stores the purpose and topic of a meeting or discussion in a database and makes it accessible as needed.

[1815] "Means for comparing the consistency of the converted text with the topic of the current discussion" is a function that checks whether the content of the speech converted into text is consistent with the topic of the current discussion.

[1816] "Means for detecting inconsistent statements" is a function that automatically identifies statements that deviate from the topic of discussion.

[1817] The "means for displaying a warning to the speaker" is a function that notifies the speaker visually or audibly of a warning when a deviation in the discussion is detected.

[1818] "Means for analyzing emotions from collected speech data" refers to a function that identifies the speaker's emotions from the tone of the voice and the vocabulary of the text.

[1819] "Means for adjusting the content and format of warning messages based on the results of emotion analysis" refers to a function that generates warning messages with appropriate tone and expression according to the analyzed emotions.

[1820] System program generation and explanation

[1821] In this invention, a system is implemented in which factory robots effectively support discussions. The specific configuration and operating principle of this system are described below.

[1822] Hardware used

[1823] Microphone: Used to collect what the workers say.

[1824] Robot Display: Used to display a warning message to the speaker.

[1825] Server: Performs various processes such as converting speech to text, checking the consistency of discussions, and sentiment analysis.

[1826] Software used

[1827] Python: Used as a programming language.

[1828] speech_recognition library: Used to convert speech to text.

[1829] text2emotion library: Used to analyze emotions from collected text.

[1830] The transformers library pipeline: used to provide sentiment analysis models.

[1831] Data processing and calculation

[1832] 1. Collecting speech data:

[1833] The server uses a microphone to collect voice data spoken by factory workers to the robot, and transmits the voice data to the server in real time.

[1834] 2. Speech to text transcription:

[1835] The server converts the speech data into text data using the speech_recognition library. For example, the speech "Proceed to install the next bolt" is converted into text.

[1836] 3. Emotion data collection and analysis:

[1837] The server uses the text2emotion and transformers libraries in a pipeline to analyze emotions from the collected speech data, for example, identifying emotions such as "urgency" or "excitement" from the tone of voice and text.

[1838] 4. Agenda consistency check:

[1839] The server accesses a database that stores meeting agendas and past instructions, and compares the current agenda with the collected text data to check for consistency and determine whether the discussion is on topic.

[1840] 5. Derailment detection:

[1841] The server uses an AI model to check whether comments stray from the topic of the discussion. For example, if someone starts discussing lunch while working, it will recognize this as a digression.

[1842] 6. Generating and displaying alerts:

[1843] When the server detects a deviation, it generates a warning message based on the emotion analysis results. For example, if the emotion analysis result is "enjoyment," it generates a gentle warning saying, "The discussion is derailing. Please return to the correct topic." The warning message is displayed on the robot display.

[1844] Specific examples

[1845] 1. Example 1:

[1846] Factory worker A says to the robot, "Let's proceed to installing the next bolt." The voice data is sent to the server and converted into text data.

[1847] After a while, worker B says, "I'd like to talk about lunch." This utterance is also collected as audio data and converted into text.

[1848] The server analyzes the combined data and determines that the "about lunch" statement does not match the agenda. Sentiment analysis also identifies the emotion of "enjoyment."

[1849] The server generates a warning message such as "The discussion is going off topic. Please return to the correct topic" and displays it on the robot display.

[1850] 2. Example prompt:

[1851] Say: "I want to talk about lunch."

[1852] Agenda: "Installing the bolts"

[1853] Current Emotion: "Excitement"

[1854] Warning message output:

[1855] "The discussion has gone off track. Please return to the correct topic."

[1856] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1857] Step 1:

[1858] Collection of speech data

[1859] When the user speaks, the robot's microphone collects the voice data, which is then sent to the server in real time. Specifically, the user might say, "Let's proceed to installing the next bolt."

[1860] Step 2:

[1861] Speech-to-text

[1862] The server converts the received voice data into text data using the speech_recognition library. The input here is voice data, and the output is the converted text data. For example, the voice data "Let's proceed to install the next bolt" is converted into text data.

[1863] Step 3:

[1864] Emotion data collection and analysis

[1865] The server uses a pipeline of the text2emotion and transformers libraries to analyze user emotions from the collected text data. The input of this step is text data, and the output is the emotion analysis result. Specifically, the emotion "hurry" is identified from the text data "Let's proceed to install the next bolt."

[1866] Step 4:

[1867] Checking the consistency of the agenda

[1868] The server accesses the database and compares the current agenda (e.g., "Install the bolt") with the collected text data (e.g., "Proceed to installing the next bolt"). The inputs in this step are the current agenda obtained from the database and the collected text data. The output is a consistency check result, specifically, "they match."

[1869] Step 5:

[1870] Derailment detection

[1871] The server uses an AI model to detect whether the content of the statement deviates from the topic of the discussion. The input for this step is the consistency check result and text data. The output is the digression detection result, which determines that the statement "I want to talk about lunch" does not match the topic of the discussion, for example.

[1872] Step 6:

[1873] Generating and Displaying Alerts

[1874] When the server detects a digression, it generates a warning message based on the emotion analysis results. The inputs to this step are the digression detection results and the emotion analysis results. The output is a warning message, specifically, a message that says, "The discussion is going off topic. Please return to the correct topic." The user is then notified via the robot display or voice.

[1875] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1876] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1877] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1878] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1879] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1880] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1881] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1882] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1883] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1884] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1885] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1886] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1887] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1888] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1889] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1890] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1891] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1892] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1893] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1894] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1895] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1896] The following is further disclosed regarding the above embodiment.

[1897] (Claim 1)

[1898] a means for collecting speech data;

[1899] a means for converting the speech data into text;

[1900] a means of remembering the subject of current discussion;

[1901] a means of comparing the consistency of the transformed text with the current topic of discussion;

[1902] a means of detecting inconsistent speech;

[1903] means for displaying a warning to the speaker;

[1904] A system including:

[1905] (Claim 2)

[1906] 2. The system according to claim 1, further comprising means for collecting the utterance data as voice data and converting the voice data into text.

[1907] (Claim 3)

[1908] 2. The system according to claim 1, further comprising means for displaying the warning as a pop-up on the speaker's screen in real time.

[1909] (Claim 4)

[1910] 10. The system of claim 1, further comprising means for displaying said warning to the sender as a pre-sent text message.

[1911] (Claim 5)

[1912] 2. The system according to claim 1, further comprising means for referring to multiple discussion topics and discussion histories when checking the consistency of the converted text.

[1913] "Example 1"

[1914] (Claim 1)

[1915] a means for users to make statements;

[1916] A means for transmitting utterance data by the terminal;

[1917] A means for the server to receive the audio data;

[1918] a means for the server to convert the voice data into text;

[1919] a means for the server to reference the agenda data;

[1920] A means for the server to confirm that the speech data matches the agenda;

[1921] A means for the server to analyze the text data;

[1922] a means for the server to detect derailment;

[1923] a means for the server to generate a warning message;

[1924] means for the terminal to display a warning message;

[1925] A system including:

[1926] (Claim 2)

[1927] 2. The system according to claim 1, further comprising means for collecting utterance data as voice data and converting the voice data into text.

[1928] (Claim 3)

[1929] 2. The system according to claim 1, further comprising means for displaying a warning as a pop-up on the speaker's screen in real time.

[1930] "Application Example 1"

[1931] (Claim 1)

[1932] a means for collecting speech data;

[1933] a means for converting the speech data into text;

[1934] a means of remembering the subject of current discussion;

[1935] a means of comparing the consistency of the transformed text with the current topic of discussion;

[1936] a means of detecting inconsistent speech;

[1937] means for displaying a warning to the speaker;

[1938] A means to ensure consistency of instructions within the factory;

[1939] A means for analyzing the collected text and detecting content that deviates from the established business topic;

[1940] a means for displaying a warning through the smart glasses;

[1941] A system including:

[1942] (Claim 2)

[1943] 2. The system according to claim 1, further comprising means for collecting the utterance data as voice data and converting the voice data into text.

[1944] (Claim 3)

[1945] 2. The system according to claim 1, further comprising means for displaying the warning as a pop-up on the speaker's screen in real time.

[1946] "Example 2: Combining Emotion Engines"

[1947] (Claim 1)

[1948] a means for collecting speech data;

[1949] a means for converting the audio data into text;

[1950] means for recognizing emotions from text data and voice data;

[1951] a means of remembering the subject of current discussion;

[1952] a means of comparing the consistency of the transformed text with the current topic of discussion;

[1953] a means of detecting inconsistent speech;

[1954] means for generating a tailored warning based on the emotional state of the speaker;

[1955] means for displaying a warning to the speaker;

[1956] A system including:

[1957] (Claim 2)

[1958] 2. The system according to claim 1, further comprising means for collecting the utterance data as voice data and converting the voice data into text.

[1959] (Claim 3)

[1960] 2. The system according to claim 1, further comprising means for displaying the warning as a pop-up on the speaker's screen in real time.

[1961] "Application example 2 when combining emotion engines"

[1962] (Claim 1)

[1963] a means for collecting speech data;

[1964] a means for converting the speech data into text;

[1965] a means of remembering the subject of current discussion;

[1966] a means of comparing the consistency of the transformed text with the current topic of discussion;

[1967] a means of detecting inconsistent speech;

[1968] means for displaying a warning to the speaker;

[1969] A means of analyzing emotions from collected speech data,

[1970] a means for adjusting the content and format of the warning message based on the sentiment analysis results;

[1971] A system including:

[1972] (Claim 2)

[1973] 2. The system according to claim 1, further comprising means for collecting the utterance data as voice data and converting the voice data into text.

[1974] (Claim 3)

[1975] 2. The system according to claim 1, further comprising means for notifying the speaker of the warning in real time by voice or display. [Explanation of symbols]

[1976] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for collecting speech data; a means for converting the speech data into text; a means of remembering the subject of current discussion; a means of comparing the consistency of the transformed text with the current topic of discussion; a means of detecting inconsistent speech; means for displaying a warning to the speaker; A system including:

2. The system according to claim 1 , further comprising means for collecting the utterance data as voice data and converting the voice data into text.

3. The system according to claim 1, further comprising means for displaying the warning as a pop-up on the screen of the speaker in real time.

4. 2. The system of claim 1, further comprising means for displaying said warning to the sender as a pre-sent text message.

5. 2. The system according to claim 1, further comprising means for referring to a plurality of discussion topics and discussion histories when checking the consistency of the converted text.

Images