System

Abstract

A system is provided.SOLUTION: A system comprising: means for using a generative model for understanding the purpose and goal of a conference and appropriately managing the progress of the conference; means for correcting speech bias based on the number of speeches and scoring results of participants; means for recording the contents of the conference in real time and automatically generating minutes; means for managing action points and adjusting the schedule of the next conference; and means for providing a simultaneous translation function in a multilingual conference.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In general, meetings can be difficult to conduct efficiently due to discussions that go off track or bias in the comments of some participants. Another problem is that creating minutes and managing action points requires a lot of time and effort. Furthermore, in international conferences that support multiple languages, language barriers can become an obstacle to communication. The purpose of this invention is to solve these issues and achieve efficient and neutral meeting conduct and appropriate management of minutes and action points. [Means for solving the problem]

[0005] The present invention solves the above problem by providing a system that includes a means for using a generative model to understand the purpose and goals of a meeting and appropriately manage the progress of the meeting, a means for correcting bias in speech based on the number of times participants speak and scoring results, a means for recording the content of the meeting in real time and automatically generating minutes, a means for managing action points and scheduling the next meeting, and a means for providing a simultaneous translation function in multilingual meetings.

[0006] A "generative model" is an artificial intelligence model that appropriately promotes speech and discussion based on the purpose and goals of the meeting.

[0007] "Measures to correct bias in comments" is a function that balances comments based on the number of times each participant speaks and the content of their comments, preventing comments from being concentrated on specific participants.

[0008] "Method for automatically generating minutes" is a function that records statements and discussions made during meetings in real time and automatically documents them.

[0009] "Action point management tool" is a function that lists the decisions made at meetings and tasks until the next meeting, and manages them by setting responsible persons and deadlines.

[0010] The "simultaneous translation function" is a function that translates between different languages ​​in real time in a multilingual conference and provides the translation to participants simultaneously. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0012] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0013] First, the terms used in the following description will be explained.

[0014] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0015] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0016] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0017] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0018] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0019] [First embodiment]

[0020] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0021] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0022] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0023] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0024] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0025] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0026] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0027] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0028] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0029] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0030] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0031] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0032] The present invention relates to a system for realizing efficient and neutral meetings. Specific embodiments for carrying out the present invention are described below.

[0033] 1. Discussion Management

[0034] The server first sets the purpose and goals of the meeting. For example, the purpose could be "formulating a market strategy for a new product" or "determining a specific action plan for the next quarter." When the meeting begins, users make comments. The server receives these comments and records them as a discussion history. Then, based on the discussion history and the purpose of the meeting, the server uses a generative model to encourage the next most appropriate comment. This suggestion is displayed on the user's device, ensuring that the meeting discussion progresses in a direction consistent with the purpose.

[0035] 2. Measures to correct bias in speech

[0036] The server scores each user's comment records. This is to count the number of comments each user makes and distinguish between users who speak frequently and those who speak infrequently. To make it easier for users who speak infrequently to express their opinions, the server encourages them to speak next time. For example, by encouraging the user who has spoken the least to speak by asking, "Would you like to give your opinion next?", it corrects bias in the discussion.

[0037] 3. Automatic generation of minutes

[0038] The server records the meeting contents in real time and automatically generates minutes. As the discussion progresses, users' comments are recorded sequentially and documented in one place. After the meeting ends, the server sends the automatically generated minutes to the user's device, where they can be viewed and saved.

[0039] 4. Action point management methods

[0040] The server records and manages action points (APs) decided during meetings in real time. Each AP is assigned a person in charge and a deadline, and is managed as a task to be completed by the next meeting. For example, when an AP such as "detailed analysis of market research" is decided, the server records the content, person in charge, and deadline, and manages it accordingly. The server periodically notifies the person in charge of this information and follows up on the progress.

[0041] 5. Simultaneous translation function

[0042] The server has the ability to translate between different languages ​​in real time in multilingual conferences. For example, it can translate minutes spoken in Japanese into English and display them on the user's device. This allows smooth communication between users who speak different languages.

[0043] By combining the above functions, the present invention provides a system that enables efficient and neutral meeting conduct and automates the management of minutes and action points, allowing all participants to engage in constructive discussions.

[0044] The processing flow will be explained below.

[0045] 1. Discussion Management

[0046] server

[0047] Step 1:

[0048] The server sets the purpose and goals of the meeting and stores them in its internal memory, such as "Draw up a market strategy for a new product" or "Decide on a concrete action plan for the next quarter."

[0049] Step 2:

[0050] The server receives a signal to start a conference and initializes the discussion history.

[0051] User

[0052] Step 3:

[0053] A user speaks during a conference.

[0054] Terminal

[0055] Step 4:

[0056] The user terminal inputs the content of the message and sends it to the server.

[0057] server

[0058] Step 5:

[0059] The server adds the user's comments to the discussion history and uses a generative model to generate the next most appropriate comment.

[0060] Step 6:

[0061] The server sends the generated message to the user terminal.

[0062] Terminal

[0063] Step 7:

[0064] The user terminal displays the next recommended comment sent from the server.

[0065] 2. Measures to correct bias in speech

[0066] server

[0067] Step 1:

[0068] The server scores each user's speech record. For example, suppose user 1 spoke three times, user 2 spoke once, and user 3 spoke twice.

[0069] Step 2:

[0070] The server identifies the user with the least number of comments, which in this example is User 2.

[0071] Step 3:

[0072] The server generates a notification to prompt a specific user to speak next, such as "User 2, would you like to give your next comment?"

[0073] Step 4:

[0074] The server sends this notification to the user terminal.

[0075] Terminal

[0076] Step 5:

[0077] The user terminal displays this notification and prompts the user to make the next statement.

[0078] 3. Automatic generation of minutes

[0079] server

[0080] Step 1:

[0081] The server records the discussion history in real time and saves the content of comments sequentially.

[0082] User

[0083] Step 2:

[0084] Each user makes a statement.

[0085] server

[0086] Step 3:

[0087] The server keeps a real-time log of everything said during the meeting.

[0088] Step 4:

[0089] After the meeting ends, the server generates minutes based on the log.

[0090] Step 5:

[0091] The server transmits the generated minutes to the user terminal.

[0092] Terminal

[0093] Step 6:

[0094] The user terminal displays the minutes, allowing the user to review and save them.

[0095] 4. Action point management methods

[0096] server

[0097] Step 1:

[0098] The server initializes the Action Point (AP) list.

[0099] User

[0100] Step 2:

[0101] A user proposes a new action point (AP) during a meeting.

[0102] Terminal

[0103] Step 3:

[0104] The user device sends the contents of the new AP to the server, including specific details such as "Detailed analysis of market research," the person in charge, and the deadline.

[0105] server

[0106] Step 4:

[0107] The server adds the new AP to the AP list and records the assignee and deadline.

[0108] Step 5:

[0109] The server will notify the appropriate person of this information.

[0110] Terminal

[0111] Step 6:

[0112] The user terminal displays the AP information sent from the server, and the information is checked by the person in charge.

[0113] 5. Simultaneous translation function

[0114] server

[0115] Step 1:

[0116] The server initializes the multilingual translation API.

[0117] User

[0118] Step 2:

[0119] Users speak in different languages ​​during a conference.

[0120] Terminal

[0121] Step 3:

[0122] A user terminal inputs a message and sends it to the server.

[0123] server

[0124] Step 4:

[0125] The server uses a translation API to translate the speech into different languages.

[0126] Step 5:

[0127] The server sends the translation results to the user terminal.

[0128] Terminal

[0129] Step 6:

[0130] The user's device displays the translation results, supporting communication between different languages.

[0131] As described above, each processing step has been specifically described, which will enable efficient and neutral meeting progress, real-time minutes creation, and AP management.

[0132] Example 1

[0133] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0134] Conventional conference systems tend to lead to biased discussions, with discussions going in directions that are not aligned with the objectives of the meeting or with certain participants speaking more than others. Furthermore, the manual process of recording meeting content and managing action points reduces efficiency. Furthermore, communication between multiple languages ​​is difficult, making it difficult to share opinions in international conferences.

[0135] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0136] In this invention, the server includes a means for setting the purpose and goals of the meeting and promoting discussion based on them, a means for recording participants' comments in real time and automatically generating minutes, a means for counting the number of comments and correcting bias in comments, a means for managing action points decided during the meeting and setting responsible parties and deadlines, and a means for real-time translation in multilingual meetings. This allows the meeting to proceed efficiently and neutrally, enabling all participants to engage in constructive discussion. Furthermore, the recording of meeting content and management of action points are automated, realizing smooth communication between multiple languages.

[0137] The "purpose and goals of the meeting" refer to the specific objectives and destinations that the meeting aims to achieve, and are the factors that determine the direction of the discussion.

[0138] "Means to promote discussion" refers to methods and mechanisms for effectively advancing discussion among participants by suggesting appropriate comments and next steps based on the purpose and goals of the meeting.

[0139] "Means for recording speech content in real time" refers to technology or devices that record participants' speech on the spot and save it as digital data.

[0140] "Means for automatically generating minutes" refers to technology or methods for automatically creating official minutes after a meeting based on recorded remarks and meeting progress information.

[0141] "Means for counting speech counts" refers to a mechanism for automatically recording the number of times each participant speaks during a meeting and aggregating that data.

[0142] "Measures to correct bias in speaking" refers to methods and mechanisms to prevent certain participants from speaking too much and to provide all participants with an opportunity to express their opinions fairly.

[0143] "Action point management means" refers to a system or method for recording specific work items decided during meetings, along with the person responsible and deadline, and tracking their progress and completion status.

[0144] "Means for real-time translation in multilingual meetings" refers to techniques and methods for instantly translating speech into other languages ​​to facilitate communication between participants who speak different languages.

[0145] The present invention relates to a system for conducting and managing a multifunctional and automated conference in which a server, terminals, and users cooperate with each other to realize efficient and neutral conferences.

[0146] 1. Setting the purpose and goals of the meeting

[0147] The server first provides a form for the meeting organizer to input the purpose and goals of the meeting. For example, the purpose of the meeting might be to "formulate a market strategy for a new product," and the goal might be to "determine a specific action plan for the next quarter." This data is saved and used to guide the meeting.

[0148] 2. Real-time recording of speech content and automatic generation of meeting minutes

[0149] When a meeting begins, the server recognizes speech in real time and transcribes it. Specific software used is Google Cloud Speech-to-Text, for example. The recognized speech is stored in a database and automatically generated as minutes after the meeting ends. Generative AI models such as OpenAI's GPT-4 are used to generate the minutes.

[0150] 3. Scoring to correct bias in statements

[0151] The server records the number of comments made by each user and performs scoring based on this. When a user who has made few comments is identified, a message is generated to prompt that user to make the next comment. For example, a prompt message such as "Would you like to give your next comment?" is generated and displayed on the user's terminal.

[0152] 4. Action Point Management

[0153] The server records the action points (APs) decided during the meeting and assigns each one a responsible person and deadline. For example, if an AP such as "detailed analysis of market research" is decided, the content, responsible person, and deadline are saved. The server periodically notifies the responsible person of this information and follows up on the progress.

[0154] 5. Multilingual simultaneous translation function

[0155] The server translates between different languages ​​in real time during a multilingual conference. For example, if a user says in Japanese, "Let's discuss the market strategy for the next quarter," the server translates it into English and displays the translated result, "Let's discuss the market strategy for the next quarter," on the user's device.

[0156] Specific examples

[0157] For example, if the purpose of a meeting is set as "Draft a market strategy for a new product" and someone says, "Let's think about what the target market for the new product should be," the server will use speech recognition software to transcribe this and store it in a database.Then, using a generative AI model, it will generate a prompt such as, "Please give us some specific ideas for determining the target market," and display it on the user's device.

[0158] As a result, the present invention provides a system that enables efficient and neutral meeting proceedings and automates the management of minutes and action points, allowing all participants to engage in constructive discussions.

[0159] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0160] Step 1:

[0161] The server provides a form for the meeting organizer to input the purpose and goals of the meeting. In the form, users input the purpose, "to formulate a market strategy for a new product," and the goal, "to determine a specific action plan for the next quarter." The input data is sent to the server and stored in a database. This helps ensure that the progress of discussions during the meeting is in line with the purpose and goals.

[0162] Step 2:

[0163] When a user speaks during a meeting, the speech is sent as audio data via the microphone to the server. The server then uses speech recognition software (e.g., Google Cloud Speech-to-Text) to convert the audio data into text data. This text data is then stored in a database. This allows the content of speech during the meeting to be recorded in real time.

[0164] Step 3:

[0165] The server generates prompts to encourage the next appropriate utterance based on the utterances recorded in real time. The server inputs the recorded utterances and the purpose of the meeting into a generative AI model (e.g., OpenAI's GPT-4) to generate recommendations for the next utterance. The generated prompts (e.g., "Next, let's list specific regions that could be considered as target markets") are displayed on the user's device, thereby encouraging the discussion to proceed in a direction that is in line with the purpose.

[0166] Step 4:

[0167] The server measures the number of comments made by each user in real time and stores it in a database for counting the number of comments. The server scores the data on the number of comments and identifies users who have made few comments. When a user who has made few comments is identified, the server generates a message saying, "Would you like to give your opinion next?" and displays it on the user's device. This corrects any bias in comments.

[0168] Step 5:

[0169] The server records the action points (APs) proposed during the meeting. The server receives the APs proposed by the users (e.g., "Detailed analysis of market research") as input and stores them in a database along with the person in charge and the deadline. After the meeting ends, the server periodically sends notifications to the person in charge to follow up on the progress. This allows for efficient management and execution of action points.

[0170] Step 6:

[0171] The server translates between different languages ​​in real time during a multilingual conference. When a user speaks in Japanese, the server uses translation software to translate this voice data into another language (e.g., English). The translated text is then displayed on the user's device. For example, "Let's discuss the market strategy for the next quarter" is translated as "Let's discuss the market strategy for the next quarter." This facilitates communication between users who speak different languages.

[0172] Through these steps, the system makes meetings more efficient and neutral, providing an environment in which all participants can engage in constructive discussions.

[0173] (Application example 1)

[0174] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0175] Conventional conference systems have problems such as difficulty in managing the progress of discussions, correcting bias in speech, creating minutes in real time, supporting multiple languages, and properly managing action items. In particular, communication breakdowns due to bias in speech and inadequate multilingual support, and insufficient follow-up on decisions are prominent, making it difficult to achieve efficient and neutral conference proceedings.

[0176] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0177] In this invention, the server includes means for using a generative AI model that understands the purpose and goals of a meeting and appropriately manages the progress of the meeting, means for correcting bias in comments based on the number of times participants speak and scoring results, means for recording meeting content in real time and automatically generating minutes, means for managing action items and scheduling the next meeting, means for providing a simultaneous translation function for multilingual meetings, means for a generative AI model to suggest appropriate next comments based on comments made during the meeting, means for generating minutes based on the comments made and distributing them to participants, and means for recording action items decided in the meeting, setting responsible parties and deadlines, and following up on progress. This makes the progress of meetings more efficient and enables neutral and constructive discussions.

[0178] A "meeting" is a gathering of multiple participants to discuss and make decisions in order to achieve a specific purpose or goal.

[0179] A "generative AI model" is an artificial intelligence model that supports speech content and decision-making based on input data.

[0180] A "server" is a computer system that processes data and provides information.

[0181] "Number of speeches" refers to the number of times each participant spoke during the conference.

[0182] The "scoring result" is the result of evaluating the number of statements and their content based on specific criteria and converting them into a score.

[0183] "Bias" refers to a situation in which the fairness of a discussion is compromised when comments and opinions are concentrated among certain participants.

[0184] "Real-time" refers to information processing at the exact moment the meeting is taking place.

[0185] "Minutes" are a written record of what was said and the results of discussions during a meeting.

[0186] "Action items" are specific actions or tasks decided upon during the meeting.

[0187] "Scheduling" refers to adjusting the schedules of participants and deciding the date and time of the next meeting.

[0188] "Multilingual support" is a function that enables participants who speak multiple languages ​​to communicate smoothly.

[0189] The "simultaneous translation function" is a function that translates between different languages ​​in real time.

[0190] "Participants" refers to people who attend meetings and are involved in discussions and decision-making.

[0191] "Appropriate content" means useful and accurate content that is in line with the purpose and goals of the meeting.

[0192] "Progress" indicates the progress and achievement of action items.

[0193] "Follow-up" refers to monitoring the progress of an action item and responding or adjusting accordingly.

[0194] The present invention relates to a system for realizing efficiency and neutrality in a meeting. Specific embodiments for carrying out the present invention are described below.

[0195] 1. System Configuration

[0196] The server uses hardware and software equipped with a generative AI model, database, and multilingual translation function. The server operates in a cloud computing environment, including the OpenAI API. Client devices include smartphones, tablets, and PCs, and users use these to participate in the conference.

[0197] 2. Meeting progress management

[0198] The server receives initial data to set the purpose and goals of the meeting and uses a generative AI model to appropriately manage the progress of the meeting. For example, if the purpose is set as "formulating a market strategy for a new product," the server analyzes the user's comments in real time and suggests the next appropriate comment. This allows the meeting to proceed in a direction that is consistent with the purpose.

[0199] 3. Correcting bias in statements

[0200] The server scores the number of comments made by each user and prompts users who have made fewer comments to make the next comment. For example, by notifying the user who has made the fewest comments, "Would you like to give your opinion next?", it is possible to correct bias in comments and promote neutral discussions.

[0201] 4. Automatic generation of meeting minutes

[0202] The server records what is said during the meeting in real time and automatically generates minutes. These minutes are automatically generated after the meeting ends and distributed to the participants' client devices. For example, if a user says, "I will talk about the results of our market research," this will be automatically recorded in the minutes.

[0203] 5. Managing Action Items

[0204] The server records the action items decided in the meeting and assigns the person in charge and deadline. These are saved in a database and notifications are sent to the person in charge to follow up on the progress as appropriate. For example, if an action item such as "Detailed analysis of market research" is set, the person in charge and deadline are recorded based on that and reminders are sent as appropriate.

[0205] 6. Multilingual simultaneous translation

[0206] In multilingual conferences, the server translates between different languages ​​in real time. For example, it translates what is said in Japanese into English and displays it on the device of an English-speaking user. This allows smooth communication between users who speak different languages.

[0207] 7. Suggested speech using generative AI models

[0208] The server uses a generative AI model to suggest the next most appropriate utterance based on the user's utterance. For example, if the user says, "I'll talk about the results of our market research," the generative AI model will suggest, "Let's consider the next action while looking at specific data."

[0209] Prompt Sentence Examples

[0210] "Suggest appropriate discussion points to follow the statement 'I'll talk about the results of my market research.'"

[0211] As described above, the present invention provides a system that realizes efficient and neutral meetings, corrects bias in discussions, and facilitates multilingual meetings.

[0212] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0213] Step 1:

[0214] The server receives initial data for setting the purpose and goals of the meeting. As input, information about the purpose and goals of the meeting is given and recorded in the database. As output, the set purpose and goals are saved on the server.

[0215] Step 2:

[0216] The server uses a generative AI model to analyze what is said during the meeting in real time. As input, the voice data spoken by the user is sent to the server, which converts it into text data. The generative AI model generates the next appropriate utterance based on this text data. As output, the generated next utterance is displayed on the user's device.

[0217] Step 3:

[0218] The server records the number of times each user has spoken and performs scoring. Information about the content of the speech and the speaker is sent to the server as input. The server records this in a database and counts the number of times each user has spoken. The server saves the scoring results as output.

[0219] Step 4:

[0220] The server generates a message to encourage users who have not spoken often to speak next. The scoring results are used as input. The server identifies the user who has spoken the least and generates a message such as "Would you like to give your next opinion?". As output, this message is sent to the device of the user.

[0221] Step 5:

[0222] The server records the meeting contents in real time and automatically generates minutes. As input, users' speech text is continuously sent to the server and recorded in a database. The server creates minutes based on this. As output, minutes that are updated in real time are generated and distributed to participants after the meeting ends.

[0223] Step 6:

[0224] The server records the action items decided during the meeting and sets the person in charge and deadline. As input, the information of the action items decided during the discussion, the person in charge, and the deadline are sent to the server. The server records this in a database and sets a schedule for following up on the progress. As output, the set action items are saved and reminders are executed according to the schedule.

[0225] Step 7:

[0226] The server translates between different languages ​​in real time during multilingual conferences. Each user's comment text and the language information to be translated are sent to the server as input. The server translates this using a multilingual translation API, and the translation results are displayed on other users' devices as output.

[0227] Through the above steps, the present invention makes the progress of the conference more efficient and facilitates smooth communication between participants.

[0228] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0229] The present invention relates to a system that realizes efficient and neutral meetings, and further combines it with an emotion engine that recognizes the emotions of users. Specific embodiments for carrying out the present invention are described below.

[0230] 1. Discussion Management

[0231] server

[0232] 1. The server sets the purpose and goals of the meeting and stores them in its internal memory, such as "Draw up a market strategy for a new product" or "Decide on a concrete action plan for the next quarter."

[0233] 2. Receive the signal to start the meeting and initialize the discussion history.

[0234] User

[0235] 3. A user speaks during the conference.

[0236] Terminal

[0237] 4. The user's device inputs the message and sends it to the server.

[0238] server

[0239] 5. The server adds the user's comment to the discussion history and uses the generative model to generate the next most appropriate comment.

[0240] 6. The server sends the generated message to the user's device.

[0241] Terminal

[0242] 7. The user terminal displays the next recommended comment sent from the server.

[0243] 2. Measures to correct bias in speech

[0244] server

[0245] 1. The server scores each user's speech record. For example, suppose user 1 spoke three times, user 2 spoke once, and user 3 spoke twice.

[0246] 2. Identify the user with the least number of comments, in this example User 2.

[0247] 3. Generate a notification to encourage a specific user to speak up, such as "User 2, would you like to give your next comment?"

[0248] 4. The server sends this notification to the user terminal.

[0249] Terminal

[0250] 5. The user device displays this notification and prompts the user to make the next statement.

[0251] 3. Automatic generation of minutes

[0252] server

[0253] 1. The server records the discussion history in real time and saves the content of comments sequentially.

[0254] User

[0255] 2. Each user makes a statement.

[0256] server

[0257] 3. The server keeps a real-time log of all statements made during the meeting.

[0258] 4. After the meeting, generate minutes based on the logs.

[0259] 5. The generated minutes are sent to the user's terminal.

[0260] Terminal

[0261] 6. The user terminal displays the minutes, allowing the user to review and save them.

[0262] 4. Action point management methods

[0263] server

[0264] 1. The server initializes the Action Point (AP) list.

[0265] User

[0266] 2. A user proposes a new action point (AP) during a meeting.

[0267] Terminal

[0268] 3. The user device sends the contents of the new AP to the server, including specific details such as "Detailed analysis of market research," the person in charge, and the deadline.

[0269] server

[0270] 4. The server adds the new AP to the AP list and records the assignee and deadline.

[0271] 5. The server will notify the appropriate person of this information.

[0272] Terminal

[0273] 6. The user terminal displays the AP information sent from the server, and the information is confirmed by the person in charge.

[0274] 5. Simultaneous translation function

[0275] server

[0276] 1. The server initializes the multilingual translation API.

[0277] User

[0278] 2. Users speak in different languages ​​during a conference.

[0279] Terminal

[0280] 3. The user terminal inputs a message and sends it to the server.

[0281] server

[0282] 4. The server uses a translation API to translate the speech into a different language.

[0283] 5. The server sends the translation results to the user's device.

[0284] Terminal

[0285] 6. The user's device displays the translation results, supporting communication between different languages.

[0286] 6. Emotion recognition means

[0287] server

[0288] 1. The server initializes the emotion engine that performs facial expression recognition and voice analysis.

[0289] User

[0290] 2. When a user speaks during a meeting, facial expression and voice data are collected by a camera and microphone.

[0291] Terminal

[0292] 3. The user device sends facial expression and voice data to the server.

[0293] server

[0294] 4. The server uses an emotion engine to analyze the user's emotions (e.g., joy, anger, sadness, stress, etc.).

[0295] 5. Based on the emotion recognition results, appropriate responses are taken. For example, if a user is feeling stressed, a message or notification is generated to encourage relaxation.

[0296] 6. The server sends these notifications to the user device.

[0297] Terminal

[0298] 7. The user's device displays a notification and provides support that corresponds to the user's emotions.

[0299] As described above, the present invention realizes efficient, neutral, and user-interactive conference proceedings, and provides a comfortable conference environment through real-time minutes creation, AP management, and user emotion recognition.

[0300] The processing flow will be explained below.

[0301] 1. Discussion Management

[0302] server

[0303] Step 1:

[0304] The server sets the purpose and goals of the meeting and stores them in its internal memory, such as "Draw up a market strategy for a new product" or "Decide on a concrete action plan for the next quarter."

[0305] Step 2:

[0306] The server receives a signal to start a conference and initializes the discussion history.

[0307] User

[0308] Step 3:

[0309] A user speaks during a conference.

[0310] Terminal

[0311] Step 4:

[0312] The user terminal inputs the content of the message and sends it to the server.

[0313] server

[0314] Step 5:

[0315] The server adds the user's comments to the discussion history and uses a generative model to generate the next most appropriate comment.

[0316] Step 6:

[0317] The server sends the generated message to the user terminal.

[0318] Terminal

[0319] Step 7:

[0320] The user terminal displays the next recommended comment sent from the server.

[0321] 2. Measures to correct bias in speech

[0322] server

[0323] Step 1:

[0324] The server scores each user's speech record. For example, suppose user 1 spoke three times, user 2 spoke once, and user 3 spoke twice.

[0325] Step 2:

[0326] The server identifies the user with the least number of comments, in this example user 2.

[0327] Step 3:

[0328] The server generates a notification to prompt a specific user to speak next, such as "User 2, would you like to give your next comment?"

[0329] Step 4:

[0330] The server sends this notification to the user terminal.

[0331] Terminal

[0332] Step 5:

[0333] The user terminal displays this notification and prompts the user to make the next statement.

[0334] 3. Automatic generation of minutes

[0335] server

[0336] Step 1:

[0337] The server records the discussion history in real time and saves the content of comments sequentially.

[0338] User

[0339] Step 2:

[0340] Each user makes a statement.

[0341] server

[0342] Step 3:

[0343] The server keeps a real-time log of everything said during the meeting.

[0344] Step 4:

[0345] After the meeting ends, the server generates minutes based on the log.

[0346] Step 5:

[0347] The server transmits the generated minutes to the user terminal.

[0348] Terminal

[0349] Step 6:

[0350] The user terminal displays the minutes, allowing the user to review and save them.

[0351] 4. Action point management methods

[0352] server

[0353] Step 1:

[0354] The server initializes the Action Point (AP) list.

[0355] User

[0356] Step 2:

[0357] A user proposes a new action point (AP) during a meeting.

[0358] Terminal

[0359] Step 3:

[0360] The user device sends the contents of the new AP to the server, including specific details such as "Detailed analysis of market research," the person in charge, and the deadline.

[0361] server

[0362] Step 4:

[0363] The server adds the new AP to the AP list and records the assignee and deadline.

[0364] Step 5:

[0365] The server will notify the appropriate person of this information.

[0366] Terminal

[0367] Step 6:

[0368] The user terminal displays the AP information sent from the server, and the information is checked by the person in charge.

[0369] 5. Simultaneous translation function

[0370] server

[0371] Step 1:

[0372] The server initializes the multilingual translation API.

[0373] User

[0374] Step 2:

[0375] Users speak in different languages ​​during a conference.

[0376] Terminal

[0377] Step 3:

[0378] A user terminal inputs a message and sends it to the server.

[0379] server

[0380] Step 4:

[0381] The server uses a translation API to translate the speech into different languages.

[0382] Step 5:

[0383] The server sends the translation results to the user terminal.

[0384] Terminal

[0385] Step 6:

[0386] The user's device displays the translation results, supporting communication between different languages.

[0387] 6. Emotion recognition means

[0388] server

[0389] Step 1:

[0390] The server initializes the emotion engine that performs facial expression recognition and voice analysis.

[0391] User

[0392] Step 2:

[0393] When a user speaks during a conference, facial expression and voice data are collected by a camera and microphone.

[0394] Terminal

[0395] Step 3:

[0396] The user terminal transmits facial expression and voice data to the server.

[0397] server

[0398] Step 4:

[0399] The server uses an emotion engine to analyze the user's emotions (e.g., joy, anger, sadness, stress, etc.).

[0400] Step 5:

[0401] Based on the emotion recognition results, appropriate responses are taken. For example, if a user is feeling stressed, a message or notification is generated to encourage relaxation.

[0402] Step 6:

[0403] The server sends these notifications to the user terminal.

[0404] Terminal

[0405] Step 7:

[0406] The user's device displays a notification and provides support that corresponds to the user's emotions.

[0407] As described above, the present invention realizes efficient, neutral, and user-interactive conference proceedings, and provides a comfortable conference environment through real-time minutes creation, AP management, and user emotion recognition.

[0408] Example 2

[0409] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0410] Conventional conference systems lack sufficient discussion progress management, speech bias correction, real-time minutes creation, action point management, and multilingual support, making it difficult to ensure meeting efficiency and neutrality. Furthermore, they do not take participants' emotions into consideration, making it difficult to provide an optimal conference environment. The present invention aims to solve these problems.

[0411] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0412] In this invention, the server includes means for setting the purpose and goal of the meeting and saving it in an internal storage device, means for generating speech content using a generative AI model to manage the progress of the meeting, means for correcting bias in speech based on the number of times participants speak and scoring results, means for recording the content of the meeting in real time and automatically generating minutes after the meeting ends, means for managing action points proposed during the meeting and recording the person in charge and deadline, means for providing a simultaneous translation function in multilingual meetings, and means for recognizing user emotions and taking appropriate action based on the results, thereby enabling efficient, neutral, and user-interactive meeting progress.

[0413] "Means for setting the purpose and goal of the meeting and saving it in the internal memory" is a function for determining the specific purpose and final goal of the meeting before the meeting starts and saving it in the internal memory.

[0414] "Means for generating speech content using a generative AI model to manage the progress of a meeting" refers to a function that uses a generative AI model during a meeting to automatically generate appropriate speech content, helping to ensure the meeting progresses smoothly.

[0415] "Means to correct bias in speech based on the number of times participants speak and the scoring results" is a function that records and analyzes the number of times each participant speaks and their score, and corrects bias in speech by encouraging specific participants to speak.

[0416] "A means of recording meeting content in real time and automatically generating minutes after the meeting" is a function that records statements and discussions made during a meeting in real time and automatically creates minutes after the meeting based on that data.

[0417] "A means to manage action points proposed during a meeting and record the person in charge and deadline" is a function for recording new action points proposed during a meeting and setting and managing the person in charge and deadline.

[0418] "Means for providing simultaneous translation functionality in multilingual conferences" refers to a functionality for translating in real time the statements of conference participants who speak different languages, and simultaneously supporting communication between different languages.

[0419] "Means for recognizing the user's emotions and taking appropriate action based on the results" refers to a function that analyzes the user's facial expressions and voice during a meeting to recognize their emotions, and then takes appropriate action or notifies them based on the results.

[0420] The present invention is a system that integrates meeting efficiency, ensuring neutrality, and user emotion recognition. A specific embodiment of this system will be described below.

[0421] This system is composed of a combination of a server, a terminal, and a user interface. The server is mainly responsible for calculation processing and data storage, while the terminal functions as an interface with the user. Each function is explained in detail below.

[0422] 1. Discussion Management

[0423] The server stores the purpose and goal of the meeting in its internal storage device. For example, it can be set to "formulate a market strategy for a new product." This information is managed based on the meeting start signal that the server receives when the meeting begins. When a user speaks during the meeting, the content of that speech is sent to the server via the user's device. The server adds the received speech to the discussion history, and based on that history, uses a generative AI model to generate the next most appropriate speech. This generated speech is sent back to the user's device and displayed to the user. As a concrete example, if we consider a meeting about a market strategy for a new product, the following actions will occur:

[0424] User 1: "Look at last year's sales data. It shows us new market potential."

[0425] Server: "User 2, next we'd like to hear your thoughts on new target markets."

[0426] 2. Measures to correct bias in speech

[0427] The server records and scores the number of times each user has spoken. For example, if User 1 has spoken three times, User 2 has spoken once, and User 3 has spoken twice, the server generates a notification to encourage the user with the fewest number of speeches (User 2 in this example) to speak next. The notification might be something like, "User 2, would you like to give your opinion next?" These notifications are sent to and displayed on the user's device.

[0428] Example prompt for a generative AI model:

[0429] "Please generate a message to notify User 2 that he has not been posting much and encourage him to post more."

[0430] 3. Automatic generation of minutes

[0431] The server records all statements made during the meeting in real time and saves them sequentially. When the meeting ends, the server automatically generates minutes based on the saved data and sends them to the user's device. The user's device displays the minutes, allowing the user to view and save them.

[0432] 4. Action point management methods

[0433] The server manages new action points (APs) proposed during a conference. When a user proposes a new AP, the user terminal sends the content to the server. The server adds the AP to a list and records the person in charge and the deadline. If necessary, the server sends a notification to the person in charge. This notification is confirmed on the user terminal.

[0434] Example prompt for a generative AI model:

[0435] "Add new action points to your list and record who is responsible and when they are due."

[0436] 5. Simultaneous translation function

[0437] The server initializes a multilingual translation API and translates any comments made in different languages ​​during the meeting. The user's device inputs the comments and sends them to the server, which then uses the translation API to translate the comments into the different language. The translation results are sent to the user's device and displayed.

[0438] Example prompt for a generative AI model:

[0439] "Translate what you say into the specified language and display the results."

[0440] 6. Emotion recognition means

[0441] The server initializes an emotion engine that performs facial expression recognition and voice analysis. When the user speaks, facial expression and voice data is collected by a camera and microphone and sent to the server via the user's device. The server uses the emotion engine to analyze emotions such as joy, anger, sadness, and stress. Based on the analysis results, the server generates appropriate response messages and notifications and sends them to the user's device. The user's device displays these notifications and responds to the user's emotions.

[0442] Example prompt for a generative AI model:

[0443] "If the user is feeling stressed, generate a notification to encourage relaxation."

[0444] As described above, the present invention realizes efficient, neutral, and user-interactive conference proceedings, and provides a comfortable conference environment through real-time minutes creation, action point management, and emotion recognition.

[0445] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0446] Specific processing steps of the system program

[0447] ~Method of managing discussions~

[0448] Step 1:

[0449] The server sets the purpose and goals of the meeting and stores them in an internal storage device.

[0450] Input: Initial setup data, including the purpose and goals of the meeting.

[0451] Output: Meeting objectives and goals data stored in internal memory.

[0452] Specific operation: The server sets goals such as "Draw up a market strategy for a new product" or "Decide on a specific action plan for the next quarter" and stores them in its internal memory.

[0453] Step 2:

[0454] The server receives a signal to start the conference.

[0455] Input: Signal to start a conference.

[0456] Output: Initialized discussion history data.

[0457] Specific operation: The server receives a conference start signal and initializes the internal discussion history.

[0458] Step 3:

[0459] A user speaks during a conference.

[0460] Input: What the user says.

[0461] Output: None.

[0462] Specific action: The user expresses their opinion or information.

[0463] Step 4:

[0464] The user terminal inputs the content of the message and sends it to the server.

[0465] Input: What the user says (voice or text data).

[0466] Output: The speech data sent to the server.

[0467] Specific operation: The user's device converts the spoken content into text using voice recognition technology or keyboard input and sends it to the server.

[0468] Step 5:

[0469] The server adds the user's comments to the discussion history and uses a generative AI model to generate the next most appropriate comment.

[0470] Input: Speech content data.

[0471] Output: The following utterance is generated:

[0472] Specific operation: The server adds the comment to the discussion history and uses the generative AI model to generate the next most appropriate comment.

[0473] Step 6:

[0474] The server sends the generated message to the user terminal.

[0475] Input: The generated utterance.

[0476] Output: The speech sent to the user's device.

[0477] Specific operation: The server sends the generated speech to the user terminal.

[0478] Step 7:

[0479] The user terminal displays the next recommended comment sent from the server.

[0480] Input: The message sent from the server.

[0481] Output: What is displayed to the user.

[0482] Specific operation: The user device displays the recommended comments and shows them to the user.

[0483] ~Methods to correct bias in speech~

[0484] Step 1:

[0485] The server scores each user's speech record.

[0486] Input: Number of speeches per user.

[0487] Output: Scoring result data.

[0488] Specific operation: The server records the number of times each user has spoken and uses that data to perform scoring.

[0489] Step 2:

[0490] Identify users who have spoken the least.

[0491] Input: Scoring result data.

[0492] Output: Specific information about users who speak less frequently.

[0493] Specific operation: The server analyzes the scoring results and identifies the user who has spoken the least.

[0494] Step 3:

[0495] The server generates a notification prompting the user to make the next statement and sends it to the user terminal.

[0496] Input: User specific information.

[0497] Output: Speech promotion notification data.

[0498] Specific operation: The server generates a notification such as "User 2, would you like to give your opinion next?" and sends it to the user terminal.

[0499] Step 4:

[0500] The user terminal displays this notification and prompts the user to make the next statement.

[0501] Input: Notification data sent by the server.

[0502] Output: What is displayed to the user.

[0503] Specific operation: The user device displays a notification and prompts the user to make the next statement.

[0504] -Method for automatically generating meeting minutes-

[0505] Step 1:

[0506] The server records the meeting contents in real time.

[0507] Input: User speech data.

[0508] Output: Real-time recording data.

[0509] Specific operation: The server records and saves what is said in real time.

[0510] Step 2:

[0511] Each user makes a statement.

[0512] Input: What the user says.

[0513] Output: None.

[0514] Specific operation: The user sends the content of the statement to the server via the terminal.

[0515] Step 3:

[0516] The server keeps a real-time log of everything said during the meeting.

[0517] Input: What the user says.

[0518] Output: Recorded log data.

[0519] What it does: The server keeps adding what is said to the record in real time.

[0520] Step 4:

[0521] After the meeting, minutes are generated based on the logs.

[0522] Input: Recorded log data.

[0523] Output: The generated transcript.

[0524] Specific operation: After the meeting ends, the server generates minutes based on the recorded log data.

[0525] Step 5:

[0526] The generated minutes are sent to the user terminal.

[0527] Input: Generated minutes data.

[0528] Output: The minutes sent to the user's terminal.

[0529] Specific operation: The server sends the generated minutes to the user terminal.

[0530] Step 6:

[0531] The user terminal displays the minutes, allowing the user to review and save them.

[0532] Input: Minutes data sent from the server.

[0533] Output: The minutes confirmed by the user.

[0534] Specific operation: The user terminal displays the minutes and allows the user to review and save them.

[0535] ~Action point management method~

[0536] Step 1:

[0537] The server initializes the Action Point (AP) list.

[0538] Input: Conference start signal.

[0539] Output: Initialized AP list.

[0540] Specific operation: The server initializes the AP list when a new conference starts.

[0541] Step 2:

[0542] A user proposes a new action point (AP) during a meeting.

[0543] Input: User suggestions.

[0544] Output: None.

[0545] Specific operation: A user proposes a new AP.

[0546] Step 3:

[0547] The user terminal sends the contents of the new AP to the server.

[0548] Input: Text input of AP by user.

[0549] Output: The AP content sent to the server.

[0550] Specific operation: The user terminal converts the contents of the AP into text and sends it to the server.

[0551] Step 4:

[0552] The server adds the new AP to the AP list and records the assignee and deadline.

[0553] Input: Transmitted AP content data.

[0554] Output: Updated AP list.

[0555] Specific operation: The server adds the received AP to a list and records the person in charge and the deadline.

[0556] Step 5:

[0557] The server will notify the appropriate person of this information.

[0558] Input: Updated AP list.

[0559] Output: Notification sent to assignee.

[0560] Specific operation: The server notifies the person in charge of the AP contents and deadline.

[0561] Step 6:

[0562] The user terminal displays the AP information sent from the server, and the information is checked by the person in charge.

[0563] Input: AP information sent from the server.

[0564] Output: AP information confirmed by the person in charge.

[0565] Specific operation: The user terminal displays the AP information, and the person in charge checks it.

[0566] -Simultaneous translation function-

[0567] Step 1:

[0568] The server initializes the multilingual translation API.

[0569] Input: Translation API initialization directive.

[0570] Output: Initialized translation API.

[0571] Specific operation: The server initializes a multilingual translation API when the conference starts.

[0572] Step 2:

[0573] Users speak in different languages ​​during a conference.

[0574] Input: What users say in different languages.

[0575] Output: None.

[0576] Specific Action: A user speaks in a different language (e.g., English and French).

[0577] Step 3:

[0578] A user terminal inputs a message and sends it to the server.

[0579] Input: What the user says (voice or text data).

[0580] Output: The message sent to the server.

[0581] Specific operation: The user terminal inputs the content of the statement and sends it to the server.

[0582] Step 4:

[0583] The server uses a translation API to translate the speech into different languages.

[0584] Input: User speech data.

[0585] Output: The translated speech.

[0586] Specific operation: The server uses the translation API to translate the speech into the specified language.

[0587] Step 5:

[0588] The server sends the translation results to the user terminal.

[0589] Input: The translated statement.

[0590] Output: The translation results sent to the user's device.

[0591] Specific operation: The server sends the translation results to the user terminal.

[0592] Step 6:

[0593] The user's device displays the translation results, supporting communication between different languages.

[0594] Input: Translation results sent from the server.

[0595] Output: The translation result confirmed by the user.

[0596] Specific operation: The user's device displays the translation results, supporting communication between users who speak different languages.

[0597] ~ Emotion recognition means ~

[0598] Step 1:

[0599] The server initializes the emotion engine that performs facial expression recognition and voice analysis.

[0600] Input: Emotion engine initialization command.

[0601] Output: Initialized emotion engine.

[0602] Specific operation: The server initializes the emotion engine when the conference starts.

[0603] Step 2:

[0604] When a user speaks during a conference, facial expression and voice data are collected by a camera and microphone.

[0605] Input: User's facial and voice data.

[0606] Output: None.

[0607] Specific operation: When a user speaks, facial expression and voice data are collected using the device's camera and microphone.

[0608] Step 3:

[0609] The user terminal transmits facial expression and voice data to the server.

[0610] Input: User's facial and voice data.

[0611] Output: Facial expression and voice data sent to the server.

[0612] Specific operation: The facial expression and voice data collected by the user's device is sent to the server.

[0613] Step 4:

[0614] The server uses an emotion engine to analyze the user's emotions.

[0615] Input: facial expression and voice data.

[0616] Output: Parsed emotion data.

[0617] Specific operation: The server uses an emotion engine to analyze the user's emotions (joy, anger, sadness, stress, etc.).

[0618] Step 5:

[0619] Take appropriate action based on the emotion recognition results.

[0620] Input: Parsed emotion data.

[0621] Output: Appropriate response message or notification.

[0622] Specific operation: Based on the emotion recognition results, the server generates an appropriate response message or notification, such as a "notification encouraging relaxation."

[0623] Step 6:

[0624] The server sends these notifications to the user terminal.

[0625] Input: The generated notification.

[0626] Output: Notification sent to the user's device.

[0627] Specific operation: The server sends the generated notification to the user terminal.

[0628] Step 7:

[0629] The user's device displays a notification and provides support that corresponds to the user's emotions.

[0630] Input: The notification sent by the server.

[0631] Output: The notification that the user acknowledged.

[0632] Specific operation: The user device displays a notification and provides emotional support to the user.

[0633] The above is a specific description of each processing step.

[0634] (Application example 2)

[0635] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0636] Conventional conference systems make it difficult for all participants to speak equally, and management is complicated when there are multilingual participants or when the progress of the meeting needs to take into account the emotional state of the participants. There is also a need for centralized provision of functions such as meeting efficiency, real-time minutes generation, and simultaneous translation for multiple languages. This has resulted in inefficient meeting progress and increased emotional burden on participants.

[0637] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0638] In this invention, the server includes: means for understanding the purpose and goals of a conference and using a generative model to appropriately manage the conference progress; means for correcting bias in speech based on the number of times participants speak and scoring results; means for recording the conference content in real time and automatically generating minutes; means for managing action points and scheduling the next conference; means for providing a simultaneous translation function for multilingual conferences; means for analyzing participants' emotions based on emotion recognition and responding accordingly; and means for generating the next appropriate speech using a generative model and displaying it on a user terminal. This not only streamlines the conference progress, allows all participants to speak equally, but also enables appropriate responses based on emotions. Furthermore, smooth communication can be achieved even when there are participants who speak multiple languages.

[0639] "Meeting objectives and goals" refer to the aims and purposes that are directly related to the progress and decisions of the meeting.

[0640] A "generative model" is a type of algorithm that uses machine learning and artificial intelligence to make predictions and generate information from data.

[0641] "Appropriate management" means choosing the best method depending on the situation and controlling things so that they proceed according to plan.

[0642] The "number of times each participant speaks" refers to the number of times each participant speaks during the conference.

[0643] "Scoring results" refer to the results of evaluating participants' number of comments, content, etc., using numerical values.

[0644] "Correcting bias in speech" refers to making adjustments so that speech is not biased toward a particular participant.

[0645] "Recording in real time" means recording at the same time as the meeting is taking place.

[0646] "Automatically generating minutes" refers to using a system to transcribe meeting contents in real time and create minutes.

[0647] "Action points" refer to specific action items or tasks decided during a meeting.

[0648] "Scheduling" refers to adjusting the date and schedule of the next meeting.

[0649] A "multilingual conference" refers to a conference with participants who speak several different languages.

[0650] "Simultaneous translation function" refers to the function of translating speech content into different languages ​​in real time.

[0651] "Emotion recognition" refers to the technology of analyzing emotions from voice, facial expressions, etc., and taking appropriate action based on the results.

[0652] "Appropriate statements" refer to the most appropriate statements made in accordance with the situation and the progress of the discussion.

[0653] "User terminal" refers to a device such as a computer or smartphone used by a conference participant.

[0654] The present invention relates to a system for efficiently and neutrally conducting a conference, and further provides feedback according to the user's emotions by combining it with an emotion recognition engine. This system includes the following main means.

[0655] 1. Setting and managing meeting objectives and goals

[0656] The server uses a generative model to understand the purpose and goals of the meeting and manage the progress of the meeting appropriately. The server sets the purpose and goals according to the user and the type of meeting, and supports the progress of the meeting based on these.

[0657] 2. Correcting bias in statements

[0658] The server monitors the number of comments made by participants and the scoring results in real time, and prompts participants who have made fewer comments to make the next comment, correcting bias in comments. This allows all participants to express their opinions equally.

[0659] 3. Real-time meeting minutes generation

[0660] The device sends the user's comments to the server, which records them in real time. Meeting minutes are automatically generated from the recorded data, allowing the content of the meeting to be reviewed immediately after the end of the meeting.

[0661] 4. Action Point Management

[0662] Action points proposed during a meeting are sent from the user's device to the server, which manages them. Action points are assigned to specific people and deadlines, and the server also coordinates the schedule for the next meeting.

[0663] 5. Multilingual support and simultaneous translation function

[0664] The server uses a multilingual translation API to translate what is said in different languages ​​in real time, allowing participants who speak different languages ​​to communicate smoothly.

[0665] 6. Emotion Recognition and Feedback

[0666] The server uses an emotion engine that performs facial recognition and voice analysis to recognize the user's emotions from the content of their speech and voice data. If the user is feeling stressed, it will respond appropriately, such as displaying a message encouraging them to relax.

[0667] 7. Suggested utterance corrections using a generative model

[0668] The server uses the generative model to generate the next appropriate utterance based on the current discussion and displays it on the user's device, allowing the discussion to proceed smoothly.

[0669] The system is implemented using the following hardware and software:

[0670] Hardware:

[0671] Microphone: Used to collect what the user says.

[0672] Speaker: Used to provide audio feedback from the server to the user.

[0673] Camera: Used to collect user facial expression data.

[0674] User terminal: A device operated by a user, such as a computer or smartphone.

[0675] software:

[0676] speech_recognition: Converts speech to text using a Python library.

[0677] text_to_speech: Converts text to speech using a Python library.

[0678] EmotionEngine: An engine that analyzes emotions through facial expression recognition and voice analysis.

[0679] Translation API: Translates speech using a multilingual translation API.

[0680] One specific example is a discussion support app for factories. This app is used as follows:

[0681] 1. Meeting to discuss strategies for improving production efficiency:

[0682] The worker speaks through a microphone and the robot converts the speech into text.

[0683] The server uses a generative model to suggest the next appropriate utterance to the worker, facilitating discussion.

[0684] Encourage workers who do not speak up often to speak up and correct bias.

[0685] For example: "Worker 1, please tell us your opinion next."

[0686] 2. If you have workers who speak different languages:

[0687] What workers say is translated in real time so that other workers can understand it.

[0688] Example: "I'd like to hear other opinions on improving production efficiency."

[0689] As described above, this system allows meetings to proceed efficiently and fairly, provides feedback that takes into account the emotional state of participants, and its multilingual support allows smooth communication between participants who speak different languages.

[0690] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0691] Step 1:

[0692] The server sets the purpose and goals of the meeting. The user inputs the purpose and goals of the meeting, and the server stores them in its internal memory. The input data is the specific purpose of the meeting, such as "formulate a market strategy for a new product" or "decide on a specific action plan for the next quarter," and the output is the purpose and goals of the meeting stored in memory.

[0693] Step 2:

[0694] The server receives a signal to start a conference and initializes the discussion history. It receives a signal to start a conference as input and resets the discussion history. The output is the initialized discussion history.

[0695] Step 3:

[0696] A user speaks during a conference. The user speaks through a microphone, and the voice data is input to the user terminal. The input voice data is converted into text in the next step.

[0697] Step 4:

[0698] The device receives the user's speech and converts it into text using speech recognition software. The input speech data is converted into text data using the speech_recognition library. The output is the converted text data.

[0699] Step 5:

[0700] The terminal transmits the converted text data to the server. The input is the text data converted by the speech recognition software, and the output is the text data transmitted to the server.

[0701] Step 6:

[0702] The server adds the received text data to the discussion history and generates the next appropriate utterance using a generative AI model. The input is the text data and the discussion history up to that point, and the output is the generated next utterance.

[0703] Step 7:

[0704] The server sends the generated utterances to the user's device. The input is the utterances generated by the generative AI model, and the output is the utterances sent to the user's device.

[0705] Step 8:

[0706] The terminal displays the next recommended utterance received from the server to the user. The input is the utterance content sent from the server, and the output is the utterance content displayed to the user.

[0707] Step 9:

[0708] The server scores the number of times participants have spoken, identifies users who have spoken less, and generates a notification to encourage them to speak next. The input is the number of times participants have spoken, and the output is the generated notification.

[0709] Step 10:

[0710] The server sends this notification to the user terminal. The input is the generated notification, and the output is the notification sent to the user terminal.

[0711] Step 11:

[0712] The terminal displays this notification and prompts the user to say: The input is the notification sent from the server, and the output is the notification displayed to the user.

[0713] Step 12:

[0714] The server keeps recording the meeting contents in real time and saves them in a log to automatically generate minutes. The input is text data transmitted in real time, and the output is the generated minutes.

[0715] Step 13:

[0716] The server sends the minutes to the user terminal. The input is the generated minutes, and the output is the minutes sent to the user terminal.

[0717] Step 14:

[0718] The terminal displays the received minutes to the user, allowing them to check and save them. The input is the minutes sent from the server, and the output is the minutes displayed to the user.

[0719] Step 15:

[0720] During a meeting, users propose action points (APs) and input them into a terminal. The input includes the AP's specific content, the person in charge, the deadline, and other information, and the output is the AP information entered into the terminal.

[0721] Step 16:

[0722] The terminal sends the proposed AP information to the server. The input is the AP information entered by the user, and the output is the AP information sent to the server.

[0723] Step 17:

[0724] The server adds new APs to the AP list, sets the responsible party and deadline, and manages them. The input is the submitted AP information, and the output is the updated AP list.

[0725] Step 18:

[0726] The server notifies the agent of the configured AP information. The input is the updated AP list, and the output is the notification sent to the agent.

[0727] Step 19:

[0728] The terminal displays the AP information received from the server so that the person in charge can check it. The input is the AP information sent from the server, and the output is the AP information displayed on the terminal.

[0729] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0730] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0731] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0732] [Second embodiment]

[0733] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0734] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0735] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0736] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0737] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0738] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0739] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0740] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0741] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0742] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0743] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0744] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0745] The present invention relates to a system for realizing efficient and neutral meetings. Specific embodiments for carrying out the present invention are described below.

[0746] 1. Discussion Management

[0747] The server first sets the purpose and goals of the meeting. For example, the purpose could be "formulating a market strategy for a new product" or "determining a specific action plan for the next quarter." When the meeting begins, users make comments. The server receives these comments and records them as a discussion history. Then, based on the discussion history and the purpose of the meeting, the server uses a generative model to encourage the next most appropriate comment. This suggestion is displayed on the user's device, ensuring that the meeting discussion progresses in a direction consistent with the purpose.

[0748] 2. Measures to correct bias in speech

[0749] The server scores each user's comment records. This is to count the number of comments each user makes and distinguish between users who speak frequently and those who speak infrequently. To make it easier for users who speak infrequently to express their opinions, the server encourages them to speak next time. For example, by encouraging the user who has spoken the least to speak by asking, "Would you like to give your opinion next?", it corrects bias in the discussion.

[0750] 3. Automatic generation of minutes

[0751] The server records the meeting contents in real time and automatically generates minutes. As the discussion progresses, users' comments are recorded sequentially and documented in one place. After the meeting ends, the server sends the automatically generated minutes to the user's device, where they can be viewed and saved.

[0752] 4. Action point management methods

[0753] The server records and manages action points (APs) decided during meetings in real time. Each AP is assigned a person in charge and a deadline, and is managed as a task to be completed by the next meeting. For example, when an AP such as "detailed analysis of market research" is decided, the server records the content, person in charge, and deadline, and manages it accordingly. The server periodically notifies the person in charge of this information and follows up on the progress.

[0754] 5. Simultaneous translation function

[0755] The server has the ability to translate between different languages ​​in real time in multilingual conferences. For example, it can translate minutes spoken in Japanese into English and display them on the user's device. This allows smooth communication between users who speak different languages.

[0756] By combining the above functions, the present invention provides a system that enables efficient and neutral meeting conduct and automates the management of minutes and action points, allowing all participants to engage in constructive discussions.

[0757] The processing flow will be explained below.

[0758] 1. Discussion Management

[0759] server

[0760] Step 1:

[0761] The server sets the purpose and goals of the meeting and stores them in its internal memory, such as "Draw up a market strategy for a new product" or "Decide on a concrete action plan for the next quarter."

[0762] Step 2:

[0763] The server receives a signal to start a conference and initializes the discussion history.

[0764] User

[0765] Step 3:

[0766] A user speaks during a conference.

[0767] Terminal

[0768] Step 4:

[0769] The user terminal inputs the content of the message and sends it to the server.

[0770] server

[0771] Step 5:

[0772] The server adds the user's comments to the discussion history and uses a generative model to generate the next most appropriate comment.

[0773] Step 6:

[0774] The server sends the generated message to the user terminal.

[0775] Terminal

[0776] Step 7:

[0777] The user terminal displays the next recommended comment sent from the server.

[0778] 2. Measures to correct bias in speech

[0779] server

[0780] Step 1:

[0781] The server scores each user's speech record. For example, suppose user 1 spoke three times, user 2 spoke once, and user 3 spoke twice.

[0782] Step 2:

[0783] The server identifies the user with the least number of comments, which in this example is User 2.

[0784] Step 3:

[0785] The server generates a notification to prompt a specific user to speak next, such as "User 2, would you like to give your next comment?"

[0786] Step 4:

[0787] The server sends this notification to the user terminal.

[0788] Terminal

[0789] Step 5:

[0790] The user terminal displays this notification and prompts the user to make the next statement.

[0791] 3. Automatic generation of minutes

[0792] server

[0793] Step 1:

[0794] The server records the discussion history in real time and saves the content of comments sequentially.

[0795] User

[0796] Step 2:

[0797] Each user makes a statement.

[0798] server

[0799] Step 3:

[0800] The server keeps a real-time log of everything said during the meeting.

[0801] Step 4:

[0802] After the meeting ends, the server generates minutes based on the log.

[0803] Step 5:

[0804] The server transmits the generated minutes to the user terminal.

[0805] Terminal

[0806] Step 6:

[0807] The user terminal displays the minutes, allowing the user to review and save them.

[0808] 4. Action point management methods

[0809] server

[0810] Step 1:

[0811] The server initializes the Action Point (AP) list.

[0812] User

[0813] Step 2:

[0814] A user proposes a new action point (AP) during a meeting.

[0815] Terminal

[0816] Step 3:

[0817] The user device sends the contents of the new AP to the server, including specific details such as "Detailed analysis of market research," the person in charge, and the deadline.

[0818] server

[0819] Step 4:

[0820] The server adds the new AP to the AP list and records the assignee and deadline.

[0821] Step 5:

[0822] The server will notify the appropriate person of this information.

[0823] Terminal

[0824] Step 6:

[0825] The user terminal displays the AP information sent from the server, and the information is checked by the person in charge.

[0826] 5. Simultaneous translation function

[0827] server

[0828] Step 1:

[0829] The server initializes the multilingual translation API.

[0830] User

[0831] Step 2:

[0832] Users speak in different languages ​​during a conference.

[0833] Terminal

[0834] Step 3:

[0835] A user terminal inputs a message and sends it to the server.

[0836] server

[0837] Step 4:

[0838] The server uses a translation API to translate the speech into different languages.

[0839] Step 5:

[0840] The server sends the translation results to the user terminal.

[0841] Terminal

[0842] Step 6:

[0843] The user's device displays the translation results, supporting communication between different languages.

[0844] As described above, each processing step has been specifically described, which will enable efficient and neutral meeting progress, real-time minutes creation, and AP management.

[0845] Example 1

[0846] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0847] Conventional conference systems tend to lead to biased discussions, with discussions going in directions that are not aligned with the objectives of the meeting or with certain participants speaking more than others. Furthermore, the manual process of recording meeting content and managing action points reduces efficiency. Furthermore, communication between multiple languages ​​is difficult, making it difficult to share opinions in international conferences.

[0848] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0849] In this invention, the server includes a means for setting the purpose and goals of the meeting and promoting discussion based on them, a means for recording participants' comments in real time and automatically generating minutes, a means for counting the number of comments and correcting bias in comments, a means for managing action points decided during the meeting and setting responsible parties and deadlines, and a means for real-time translation in multilingual meetings. This allows the meeting to proceed efficiently and neutrally, enabling all participants to engage in constructive discussion. Furthermore, the recording of meeting content and management of action points are automated, realizing smooth communication between multiple languages.

[0850] The "purpose and goals of the meeting" refer to the specific objectives and destinations that the meeting aims to achieve, and are the factors that determine the direction of the discussion.

[0851] "Means to promote discussion" refers to methods and mechanisms for effectively advancing discussion among participants by suggesting appropriate comments and next steps based on the purpose and goals of the meeting.

[0852] "Means for recording speech content in real time" refers to technology or devices that record participants' speech on the spot and save it as digital data.

[0853] "Means for automatically generating minutes" refers to technology or methods for automatically creating official minutes after a meeting based on recorded remarks and meeting progress information.

[0854] "Means for counting speech counts" refers to a mechanism for automatically recording the number of times each participant speaks during a meeting and aggregating that data.

[0855] "Measures to correct bias in speaking" refers to methods and mechanisms to prevent certain participants from speaking too much and to provide all participants with an opportunity to express their opinions fairly.

[0856] "Action point management means" refers to a system or method for recording specific work items decided during meetings, along with the person responsible and deadline, and tracking their progress and completion status.

[0857] "Means for real-time translation in multilingual meetings" refers to techniques and methods for instantly translating speech into other languages ​​to facilitate communication between participants who speak different languages.

[0858] The present invention relates to a system for conducting and managing a multifunctional and automated conference in which a server, terminals, and users cooperate with each other to realize efficient and neutral conferences.

[0859] 1. Setting the purpose and goals of the meeting

[0860] The server first provides a form for the meeting organizer to input the purpose and goals of the meeting. For example, the purpose of the meeting might be to "formulate a market strategy for a new product," and the goal might be to "determine a specific action plan for the next quarter." This data is saved and used to guide the meeting.

[0861] 2. Real-time recording of speech content and automatic generation of meeting minutes

[0862] When a meeting begins, the server recognizes speech in real time and transcribes it. Specific software used is Google Cloud Speech-to-Text, for example. The recognized speech is stored in a database and automatically generated as minutes after the meeting ends. Generative AI models such as OpenAI's GPT-4 are used to generate the minutes.

[0863] 3. Scoring to correct bias in statements

[0864] The server records the number of comments made by each user and performs scoring based on this. When a user who has made few comments is identified, a message is generated to prompt that user to make the next comment. For example, a prompt message such as "Would you like to give your next comment?" is generated and displayed on the user's terminal.

[0865] 4. Action Point Management

[0866] The server records the action points (APs) decided during the meeting and assigns each one a responsible person and deadline. For example, if an AP such as "detailed analysis of market research" is decided, the content, responsible person, and deadline are saved. The server periodically notifies the responsible person of this information and follows up on the progress.

[0867] 5. Multilingual simultaneous translation function

[0868] The server translates between different languages ​​in real time during a multilingual conference. For example, if a user says in Japanese, "Let's discuss the market strategy for the next quarter," the server translates it into English and displays the translated result, "Let's discuss the market strategy for the next quarter," on the user's device.

[0869] Specific examples

[0870] For example, if the purpose of a meeting is set as "Draft a market strategy for a new product" and someone says, "Let's think about what the target market for the new product should be," the server will use speech recognition software to transcribe this and store it in a database.Then, using a generative AI model, it will generate a prompt such as, "Please give us some specific ideas for determining the target market," and display it on the user's device.

[0871] As a result, the present invention provides a system that enables efficient and neutral meeting proceedings and automates the management of minutes and action points, allowing all participants to engage in constructive discussions.

[0872] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0873] Step 1:

[0874] The server provides a form for the meeting organizer to input the purpose and goals of the meeting. In the form, users input the purpose, "to formulate a market strategy for a new product," and the goal, "to determine a specific action plan for the next quarter." The input data is sent to the server and stored in a database. This helps ensure that the progress of discussions during the meeting is in line with the purpose and goals.

[0875] Step 2:

[0876] When a user speaks during a meeting, the speech is sent as audio data via the microphone to the server. The server then uses speech recognition software (e.g., Google Cloud Speech-to-Text) to convert the audio data into text data. This text data is then stored in a database. This allows the content of speech during the meeting to be recorded in real time.

[0877] Step 3:

[0878] The server generates prompts to encourage the next appropriate utterance based on the utterances recorded in real time. The server inputs the recorded utterances and the purpose of the meeting into a generative AI model (e.g., OpenAI's GPT-4) to generate recommendations for the next utterance. The generated prompts (e.g., "Next, let's list specific regions that could be considered as target markets") are displayed on the user's device, thereby encouraging the discussion to proceed in a direction that is in line with the purpose.

[0879] Step 4:

[0880] The server measures the number of comments made by each user in real time and stores it in a database for counting the number of comments. The server scores the data on the number of comments and identifies users who have made few comments. When a user who has made few comments is identified, the server generates a message saying, "Would you like to give your opinion next?" and displays it on the user's device. This corrects any bias in comments.

[0881] Step 5:

[0882] The server records the action points (APs) proposed during the meeting. The server receives the APs proposed by the users (e.g., "Detailed analysis of market research") as input and stores them in a database along with the person in charge and the deadline. After the meeting ends, the server periodically sends notifications to the person in charge to follow up on the progress. This allows for efficient management and execution of action points.

[0883] Step 6:

[0884] The server translates between different languages ​​in real time during a multilingual conference. When a user speaks in Japanese, the server uses translation software to translate this voice data into another language (e.g., English). The translated text is then displayed on the user's device. For example, "Let's discuss the market strategy for the next quarter" is translated as "Let's discuss the market strategy for the next quarter." This facilitates communication between users who speak different languages.

[0885] Through these steps, the system makes meetings more efficient and neutral, providing an environment in which all participants can engage in constructive discussions.

[0886] (Application example 1)

[0887] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0888] Conventional conference systems have problems such as difficulty in managing the progress of discussions, correcting bias in speech, creating minutes in real time, supporting multiple languages, and properly managing action items. In particular, communication breakdowns due to bias in speech and inadequate multilingual support, and insufficient follow-up on decisions are prominent, making it difficult to achieve efficient and neutral conference proceedings.

[0889] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0890] In this invention, the server includes means for using a generative AI model that understands the purpose and goals of a meeting and appropriately manages the progress of the meeting, means for correcting bias in comments based on the number of times participants speak and scoring results, means for recording meeting content in real time and automatically generating minutes, means for managing action items and scheduling the next meeting, means for providing a simultaneous translation function for multilingual meetings, means for a generative AI model to suggest appropriate next comments based on comments made during the meeting, means for generating minutes based on the comments made and distributing them to participants, and means for recording action items decided in the meeting, setting responsible parties and deadlines, and following up on progress. This makes the progress of meetings more efficient and enables neutral and constructive discussions.

[0891] A "meeting" is a gathering of multiple participants to discuss and make decisions in order to achieve a specific purpose or goal.

[0892] A "generative AI model" is an artificial intelligence model that supports speech content and decision-making based on input data.

[0893] A "server" is a computer system that processes data and provides information.

[0894] "Number of speeches" refers to the number of times each participant spoke during the conference.

[0895] The "scoring result" is the result of evaluating the number of statements and their content based on specific criteria and converting them into a score.

[0896] "Bias" refers to a situation in which the fairness of a discussion is compromised when comments and opinions are concentrated among certain participants.

[0897] "Real-time" refers to information processing at the exact moment the meeting is taking place.

[0898] "Minutes" are a written record of what was said and the results of discussions during a meeting.

[0899] "Action items" are specific actions or tasks decided upon during the meeting.

[0900] "Scheduling" refers to adjusting the schedules of participants and deciding the date and time of the next meeting.

[0901] "Multilingual support" is a function that enables participants who speak multiple languages ​​to communicate smoothly.

[0902] The "simultaneous translation function" is a function that translates between different languages ​​in real time.

[0903] "Participants" refers to people who attend meetings and are involved in discussions and decision-making.

[0904] "Appropriate content" means useful and accurate content that is in line with the purpose and goals of the meeting.

[0905] "Progress" indicates the progress and achievement of action items.

[0906] "Follow-up" refers to monitoring the progress of an action item and responding or adjusting accordingly.

[0907] The present invention relates to a system for realizing efficiency and neutrality in a meeting. Specific embodiments for carrying out the present invention are described below.

[0908] 1. System Configuration

[0909] The server uses hardware and software equipped with a generative AI model, database, and multilingual translation function. The server operates in a cloud computing environment, including the OpenAI API. Client devices include smartphones, tablets, and PCs, and users use these to participate in the conference.

[0910] 2. Meeting progress management

[0911] The server receives initial data to set the purpose and goals of the meeting and uses a generative AI model to appropriately manage the progress of the meeting. For example, if the purpose is set as "formulating a market strategy for a new product," the server analyzes the user's comments in real time and suggests the next appropriate comment. This allows the meeting to proceed in a direction that is consistent with the purpose.

[0912] 3. Correcting bias in statements

[0913] The server scores the number of comments made by each user and prompts users who have made fewer comments to make the next comment. For example, by notifying the user who has made the fewest comments, "Would you like to give your opinion next?", it is possible to correct bias in comments and promote neutral discussions.

[0914] 4. Automatic generation of meeting minutes

[0915] The server records what is said during the meeting in real time and automatically generates minutes. These minutes are automatically generated after the meeting ends and distributed to the participants' client devices. For example, if a user says, "I will talk about the results of our market research," this will be automatically recorded in the minutes.

[0916] 5. Managing Action Items

[0917] The server records the action items decided in the meeting and assigns the person in charge and deadline. These are saved in a database and notifications are sent to the person in charge to follow up on the progress as appropriate. For example, if an action item such as "Detailed analysis of market research" is set, the person in charge and deadline are recorded based on that and reminders are sent as appropriate.

[0918] 6. Multilingual simultaneous translation

[0919] In multilingual conferences, the server translates between different languages ​​in real time. For example, it translates what is said in Japanese into English and displays it on the device of an English-speaking user. This allows smooth communication between users who speak different languages.

[0920] 7. Suggested speech using generative AI models

[0921] The server uses a generative AI model to suggest the next most appropriate utterance based on the user's utterance. For example, if the user says, "I'll talk about the results of our market research," the generative AI model will suggest, "Let's consider the next action while looking at specific data."

[0922] Prompt Sentence Examples

[0923] "Suggest appropriate discussion points to follow the statement 'I'll talk about the results of my market research.'"

[0924] As described above, the present invention provides a system that realizes efficient and neutral meetings, corrects bias in discussions, and facilitates multilingual meetings.

[0925] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0926] Step 1:

[0927] The server receives initial data for setting the purpose and goals of the meeting. As input, information about the purpose and goals of the meeting is given and recorded in the database. As output, the set purpose and goals are saved on the server.

[0928] Step 2:

[0929] The server uses a generative AI model to analyze what is said during the meeting in real time. As input, the voice data spoken by the user is sent to the server, which converts it into text data. The generative AI model generates the next appropriate utterance based on this text data. As output, the generated next utterance is displayed on the user's device.

[0930] Step 3:

[0931] The server records the number of times each user has spoken and performs scoring. Information about the content of the speech and the speaker is sent to the server as input. The server records this in a database and counts the number of times each user has spoken. The server saves the scoring results as output.

[0932] Step 4:

[0933] The server generates a message to encourage users who have not spoken often to speak next. The scoring results are used as input. The server identifies the user who has spoken the least and generates a message such as "Would you like to give your next opinion?". As output, this message is sent to the device of the user.

[0934] Step 5:

[0935] The server records the meeting contents in real time and automatically generates minutes. As input, users' speech text is continuously sent to the server and recorded in a database. The server creates minutes based on this. As output, minutes that are updated in real time are generated and distributed to participants after the meeting ends.

[0936] Step 6:

[0937] The server records the action items decided during the meeting and sets the person in charge and deadline. As input, the information of the action items decided during the discussion, the person in charge, and the deadline are sent to the server. The server records this in a database and sets a schedule for following up on the progress. As output, the set action items are saved and reminders are executed according to the schedule.

[0938] Step 7:

[0939] The server translates between different languages ​​in real time during multilingual conferences. Each user's comment text and the language information to be translated are sent to the server as input. The server translates this using a multilingual translation API, and the translation results are displayed on other users' devices as output.

[0940] Through the above steps, the present invention makes the progress of the conference more efficient and facilitates smooth communication between participants.

[0941] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0942] The present invention relates to a system that realizes efficient and neutral meetings, and further combines it with an emotion engine that recognizes the emotions of users. Specific embodiments for carrying out the present invention are described below.

[0943] 1. Discussion Management

[0944] server

[0945] 1. The server sets the purpose and goals of the meeting and stores them in its internal memory, such as "Draw up a market strategy for a new product" or "Decide on a concrete action plan for the next quarter."

[0946] 2. Receive the signal to start the meeting and initialize the discussion history.

[0947] User

[0948] 3. A user speaks during the conference.

[0949] Terminal

[0950] 4. The user's device inputs the message and sends it to the server.

[0951] server

[0952] 5. The server adds the user's comment to the discussion history and uses the generative model to generate the next most appropriate comment.

[0953] 6. The server sends the generated message to the user's device.

[0954] Terminal

[0955] 7. The user terminal displays the next recommended comment sent from the server.

[0956] 2. Measures to correct bias in speech

[0957] server

[0958] 1. The server scores each user's speech record. For example, suppose user 1 spoke three times, user 2 spoke once, and user 3 spoke twice.

[0959] 2. Identify the user with the least number of comments, in this example User 2.

[0960] 3. Generate a notification to encourage a specific user to speak up, such as "User 2, would you like to give your next comment?"

[0961] 4. The server sends this notification to the user terminal.

[0962] Terminal

[0963] 5. The user device displays this notification and prompts the user to make the next statement.

[0964] 3. Automatic generation of minutes

[0965] server

[0966] 1. The server records the discussion history in real time and saves the content of comments sequentially.

[0967] User

[0968] 2. Each user makes a statement.

[0969] server

[0970] 3. The server keeps a real-time log of all statements made during the meeting.

[0971] 4. After the meeting, generate minutes based on the logs.

[0972] 5. The generated minutes are sent to the user's terminal.

[0973] Terminal

[0974] 6. The user terminal displays the minutes, allowing the user to review and save them.

[0975] 4. Action point management methods

[0976] server

[0977] 1. The server initializes the Action Point (AP) list.

[0978] User

[0979] 2. A user proposes a new action point (AP) during a meeting.

[0980] Terminal

[0981] 3. The user device sends the contents of the new AP to the server, including specific details such as "Detailed analysis of market research," the person in charge, and the deadline.

[0982] server

[0983] 4. The server adds the new AP to the AP list and records the assignee and deadline.

[0984] 5. The server will notify the appropriate person of this information.

[0985] Terminal

[0986] 6. The user terminal displays the AP information sent from the server, and the information is confirmed by the person in charge.

[0987] 5. Simultaneous translation function

[0988] server

[0989] 1. The server initializes the multilingual translation API.

[0990] User

[0991] 2. Users speak in different languages ​​during a conference.

[0992] Terminal

[0993] 3. The user terminal inputs a message and sends it to the server.

[0994] server

[0995] 4. The server uses a translation API to translate the speech into a different language.

[0996] 5. The server sends the translation results to the user's device.

[0997] Terminal

[0998] 6. The user's device displays the translation results, supporting communication between different languages.

[0999] 6. Emotion recognition means

[1000] server

[1001] 1. The server initializes the emotion engine that performs facial expression recognition and voice analysis.

[1002] User

[1003] 2. When a user speaks during a meeting, facial expression and voice data are collected by a camera and microphone.

[1004] Terminal

[1005] 3. The user device sends facial expression and voice data to the server.

[1006] server

[1007] 4. The server uses an emotion engine to analyze the user's emotions (e.g., joy, anger, sadness, stress, etc.).

[1008] 5. Based on the emotion recognition results, appropriate responses are taken. For example, if a user is feeling stressed, a message or notification is generated to encourage relaxation.

[1009] 6. The server sends these notifications to the user device.

[1010] Terminal

[1011] 7. The user's device displays a notification and provides support that corresponds to the user's emotions.

[1012] As described above, the present invention realizes efficient, neutral, and user-interactive conference proceedings, and provides a comfortable conference environment through real-time minutes creation, AP management, and user emotion recognition.

[1013] The processing flow will be explained below.

[1014] 1. Discussion Management

[1015] server

[1016] Step 1:

[1017] The server sets the purpose and goals of the meeting and stores them in its internal memory, such as "Draw up a market strategy for a new product" or "Decide on a concrete action plan for the next quarter."

[1018] Step 2:

[1019] The server receives a signal to start a conference and initializes the discussion history.

[1020] User

[1021] Step 3:

[1022] A user speaks during a conference.

[1023] Terminal

[1024] Step 4:

[1025] The user terminal inputs the content of the message and sends it to the server.

[1026] server

[1027] Step 5:

[1028] The server adds the user's comments to the discussion history and uses a generative model to generate the next most appropriate comment.

[1029] Step 6:

[1030] The server sends the generated message to the user terminal.

[1031] Terminal

[1032] Step 7:

[1033] The user terminal displays the next recommended comment sent from the server.

[1034] 2. Measures to correct bias in speech

[1035] server

[1036] Step 1:

[1037] The server scores each user's speech record. For example, suppose user 1 spoke three times, user 2 spoke once, and user 3 spoke twice.

[1038] Step 2:

[1039] The server identifies the user with the least number of comments, in this example user 2.

[1040] Step 3:

[1041] The server generates a notification to prompt a specific user to speak next, such as "User 2, would you like to give your next comment?"

[1042] Step 4:

[1043] The server sends this notification to the user terminal.

[1044] Terminal

[1045] Step 5:

[1046] The user terminal displays this notification and prompts the user to make the next statement.

[1047] 3. Automatic generation of minutes

[1048] server

[1049] Step 1:

[1050] The server records the discussion history in real time and saves the content of comments sequentially.

[1051] User

[1052] Step 2:

[1053] Each user makes a statement.

[1054] server

[1055] Step 3:

[1056] The server keeps a real-time log of everything said during the meeting.

[1057] Step 4:

[1058] After the meeting ends, the server generates minutes based on the log.

[1059] Step 5:

[1060] The server transmits the generated minutes to the user terminal.

[1061] Terminal

[1062] Step 6:

[1063] The user terminal displays the minutes, allowing the user to review and save them.

[1064] 4. Action point management methods

[1065] server

[1066] Step 1:

[1067] The server initializes the Action Point (AP) list.

[1068] User

[1069] Step 2:

[1070] A user proposes a new action point (AP) during a meeting.

[1071] Terminal

[1072] Step 3:

[1073] The user device sends the contents of the new AP to the server, including specific details such as "Detailed analysis of market research," the person in charge, and the deadline.

[1074] server

[1075] Step 4:

[1076] The server adds the new AP to the AP list and records the assignee and deadline.

[1077] Step 5:

[1078] The server will notify the appropriate person of this information.

[1079] Terminal

[1080] Step 6:

[1081] The user terminal displays the AP information sent from the server, and the information is checked by the person in charge.

[1082] 5. Simultaneous translation function

[1083] server

[1084] Step 1:

[1085] The server initializes the multilingual translation API.

[1086] User

[1087] Step 2:

[1088] Users speak in different languages ​​during a conference.

[1089] Terminal

[1090] Step 3:

[1091] A user terminal inputs a message and sends it to the server.

[1092] server

[1093] Step 4:

[1094] The server uses a translation API to translate the speech into different languages.

[1095] Step 5:

[1096] The server sends the translation results to the user terminal.

[1097] Terminal

[1098] Step 6:

[1099] The user's device displays the translation results, supporting communication between different languages.

[1100] 6. Emotion recognition means

[1101] server

[1102] Step 1:

[1103] The server initializes the emotion engine that performs facial expression recognition and voice analysis.

[1104] User

[1105] Step 2:

[1106] When a user speaks during a conference, facial expression and voice data are collected by a camera and microphone.

[1107] Terminal

[1108] Step 3:

[1109] The user terminal transmits facial expression and voice data to the server.

[1110] server

[1111] Step 4:

[1112] The server uses an emotion engine to analyze the user's emotions (e.g., joy, anger, sadness, stress, etc.).

[1113] Step 5:

[1114] Based on the emotion recognition results, appropriate responses are taken. For example, if a user is feeling stressed, a message or notification is generated to encourage relaxation.

[1115] Step 6:

[1116] The server sends these notifications to the user terminal.

[1117] Terminal

[1118] Step 7:

[1119] The user's device displays a notification and provides support that corresponds to the user's emotions.

[1120] As described above, the present invention realizes efficient, neutral, and user-interactive conference proceedings, and provides a comfortable conference environment through real-time minutes creation, AP management, and user emotion recognition.

[1121] Example 2

[1122] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1123] Conventional conference systems lack sufficient discussion progress management, speech bias correction, real-time minutes creation, action point management, and multilingual support, making it difficult to ensure meeting efficiency and neutrality. Furthermore, they do not take participants' emotions into consideration, making it difficult to provide an optimal conference environment. The present invention aims to solve these problems.

[1124] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1125] In this invention, the server includes means for setting the purpose and goal of the meeting and saving it in an internal storage device, means for generating speech content using a generative AI model to manage the progress of the meeting, means for correcting bias in speech based on the number of times participants speak and scoring results, means for recording the content of the meeting in real time and automatically generating minutes after the meeting ends, means for managing action points proposed during the meeting and recording the person in charge and deadline, means for providing a simultaneous translation function in multilingual meetings, and means for recognizing user emotions and taking appropriate action based on the results, thereby enabling efficient, neutral, and user-interactive meeting progress.

[1126] "Means for setting the purpose and goal of the meeting and saving it in the internal memory" is a function for determining the specific purpose and final goal of the meeting before the meeting starts and saving it in the internal memory.

[1127] "Means for generating speech content using a generative AI model to manage the progress of a meeting" refers to a function that uses a generative AI model during a meeting to automatically generate appropriate speech content, helping to ensure the meeting progresses smoothly.

[1128] "Means to correct bias in speech based on the number of times participants speak and the scoring results" is a function that records and analyzes the number of times each participant speaks and their score, and corrects bias in speech by encouraging specific participants to speak.

[1129] "A means of recording meeting content in real time and automatically generating minutes after the meeting" is a function that records statements and discussions made during a meeting in real time and automatically creates minutes after the meeting based on that data.

[1130] "A means to manage action points proposed during a meeting and record the person in charge and deadline" is a function for recording new action points proposed during a meeting and setting and managing the person in charge and deadline.

[1131] "Means for providing simultaneous translation functionality in multilingual conferences" refers to a functionality for translating in real time the statements of conference participants who speak different languages, and simultaneously supporting communication between different languages.

[1132] "Means for recognizing the user's emotions and taking appropriate action based on the results" refers to a function that analyzes the user's facial expressions and voice during a meeting to recognize their emotions, and then takes appropriate action or notifies them based on the results.

[1133] The present invention is a system that integrates meeting efficiency, ensuring neutrality, and user emotion recognition. A specific embodiment of this system will be described below.

[1134] This system is composed of a combination of a server, a terminal, and a user interface. The server is mainly responsible for calculation processing and data storage, while the terminal functions as an interface with the user. Each function is explained in detail below.

[1135] 1. Discussion Management

[1136] The server stores the purpose and goal of the meeting in its internal storage device. For example, it can be set to "formulate a market strategy for a new product." This information is managed based on the meeting start signal that the server receives when the meeting begins. When a user speaks during the meeting, the content of that speech is sent to the server via the user's device. The server adds the received speech to the discussion history, and based on that history, uses a generative AI model to generate the next most appropriate speech. This generated speech is sent back to the user's device and displayed to the user. As a concrete example, if we consider a meeting about a market strategy for a new product, the following actions will occur:

[1137] User 1: "Look at last year's sales data. It shows us new market potential."

[1138] Server: "User 2, next we'd like to hear your thoughts on new target markets."

[1139] 2. Measures to correct bias in speech

[1140] The server records and scores the number of times each user has spoken. For example, if User 1 has spoken three times, User 2 has spoken once, and User 3 has spoken twice, the server generates a notification to encourage the user with the fewest number of speeches (User 2 in this example) to speak next. The notification might be something like, "User 2, would you like to give your opinion next?" These notifications are sent to and displayed on the user's device.

[1141] Example prompt for a generative AI model:

[1142] "Please generate a message to notify User 2 that he has not been posting much and encourage him to post more."

[1143] 3. Automatic generation of minutes

[1144] The server records all statements made during the meeting in real time and saves them sequentially. When the meeting ends, the server automatically generates minutes based on the saved data and sends them to the user's device. The user's device displays the minutes, allowing the user to view and save them.

[1145] 4. Action point management methods

[1146] The server manages new action points (APs) proposed during a conference. When a user proposes a new AP, the user terminal sends the content to the server. The server adds the AP to a list and records the person in charge and the deadline. If necessary, the server sends a notification to the person in charge. This notification is confirmed on the user terminal.

[1147] Example prompt for a generative AI model:

[1148] "Add new action points to your list and record who is responsible and when they are due."

[1149] 5. Simultaneous translation function

[1150] The server initializes a multilingual translation API and translates any comments made in different languages ​​during the meeting. The user's device inputs the comments and sends them to the server, which then uses the translation API to translate the comments into the different language. The translation results are sent to the user's device and displayed.

[1151] Example prompt for a generative AI model:

[1152] "Translate what you say into the specified language and display the results."

[1153] 6. Emotion recognition means

[1154] The server initializes an emotion engine that performs facial expression recognition and voice analysis. When the user speaks, facial expression and voice data is collected by a camera and microphone and sent to the server via the user's device. The server uses the emotion engine to analyze emotions such as joy, anger, sadness, and stress. Based on the analysis results, the server generates appropriate response messages and notifications and sends them to the user's device. The user's device displays these notifications and responds to the user's emotions.

[1155] Example prompt for a generative AI model:

[1156] "If the user is feeling stressed, generate a notification to encourage relaxation."

[1157] As described above, the present invention realizes efficient, neutral, and user-interactive conference proceedings, and provides a comfortable conference environment through real-time minutes creation, action point management, and emotion recognition.

[1158] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1159] Specific processing steps of the system program

[1160] ~Method of managing discussions~

[1161] Step 1:

[1162] The server sets the purpose and goals of the meeting and stores them in an internal storage device.

[1163] Input: Initial setup data, including the purpose and goals of the meeting.

[1164] Output: Meeting objectives and goals data stored in internal memory.

[1165] Specific operation: The server sets goals such as "Draw up a market strategy for a new product" or "Decide on a specific action plan for the next quarter" and stores them in its internal memory.

[1166] Step 2:

[1167] The server receives a signal to start the conference.

[1168] Input: Signal to start a conference.

[1169] Output: Initialized discussion history data.

[1170] Specific operation: The server receives a conference start signal and initializes the internal discussion history.

[1171] Step 3:

[1172] A user speaks during a conference.

[1173] Input: What the user says.

[1174] Output: None.

[1175] Specific action: The user expresses their opinion or information.

[1176] Step 4:

[1177] The user terminal inputs the content of the message and sends it to the server.

[1178] Input: What the user says (voice or text data).

[1179] Output: The speech data sent to the server.

[1180] Specific operation: The user's device converts the spoken content into text using voice recognition technology or keyboard input and sends it to the server.

[1181] Step 5:

[1182] The server adds the user's comments to the discussion history and uses a generative AI model to generate the next most appropriate comment.

[1183] Input: Speech content data.

[1184] Output: The following utterance is generated:

[1185] Specific operation: The server adds the comment to the discussion history and uses the generative AI model to generate the next most appropriate comment.

[1186] Step 6:

[1187] The server sends the generated message to the user terminal.

[1188] Input: The generated utterance.

[1189] Output: The speech sent to the user's device.

[1190] Specific operation: The server sends the generated speech to the user terminal.

[1191] Step 7:

[1192] The user terminal displays the next recommended comment sent from the server.

[1193] Input: The message sent from the server.

[1194] Output: What is displayed to the user.

[1195] Specific operation: The user device displays the recommended comments and shows them to the user.

[1196] ~Methods to correct bias in speech~

[1197] Step 1:

[1198] The server scores each user's speech record.

[1199] Input: Number of speeches per user.

[1200] Output: Scoring result data.

[1201] Specific operation: The server records the number of times each user has spoken and uses that data to perform scoring.

[1202] Step 2:

[1203] Identify users who have spoken the least.

[1204] Input: Scoring result data.

[1205] Output: Specific information about users who speak less frequently.

[1206] Specific operation: The server analyzes the scoring results and identifies the user who has spoken the least.

[1207] Step 3:

[1208] The server generates a notification prompting the user to make the next statement and sends it to the user terminal.

[1209] Input: User specific information.

[1210] Output: Speech promotion notification data.

[1211] Specific operation: The server generates a notification such as "User 2, would you like to give your opinion next?" and sends it to the user terminal.

[1212] Step 4:

[1213] The user terminal displays this notification and prompts the user to make the next statement.

[1214] Input: Notification data sent by the server.

[1215] Output: What is displayed to the user.

[1216] Specific operation: The user device displays a notification and prompts the user to make the next statement.

[1217] -Method for automatically generating meeting minutes-

[1218] Step 1:

[1219] The server records the meeting contents in real time.

[1220] Input: User speech data.

[1221] Output: Real-time recording data.

[1222] Specific operation: The server records and saves what is said in real time.

[1223] Step 2:

[1224] Each user makes a statement.

[1225] Input: What the user says.

[1226] Output: None.

[1227] Specific operation: The user sends the content of the statement to the server via the terminal.

[1228] Step 3:

[1229] The server keeps a real-time log of everything said during the meeting.

[1230] Input: What the user says.

[1231] Output: Recorded log data.

[1232] What it does: The server keeps adding what is said to the record in real time.

[1233] Step 4:

[1234] After the meeting, minutes are generated based on the logs.

[1235] Input: Recorded log data.

[1236] Output: The generated transcript.

[1237] Specific operation: After the meeting ends, the server generates minutes based on the recorded log data.

[1238] Step 5:

[1239] The generated minutes are sent to the user terminal.

[1240] Input: Generated minutes data.

[1241] Output: The minutes sent to the user's terminal.

[1242] Specific operation: The server sends the generated minutes to the user terminal.

[1243] Step 6:

[1244] The user terminal displays the minutes, allowing the user to review and save them.

[1245] Input: Minutes data sent from the server.

[1246] Output: The minutes confirmed by the user.

[1247] Specific operation: The user terminal displays the minutes and allows the user to review and save them.

[1248] ~Action point management method~

[1249] Step 1:

[1250] The server initializes the Action Point (AP) list.

[1251] Input: Conference start signal.

[1252] Output: Initialized AP list.

[1253] Specific operation: The server initializes the AP list when a new conference starts.

[1254] Step 2:

[1255] A user proposes a new action point (AP) during a meeting.

[1256] Input: User suggestions.

[1257] Output: None.

[1258] Specific operation: A user proposes a new AP.

[1259] Step 3:

[1260] The user terminal sends the contents of the new AP to the server.

[1261] Input: Text input of AP by user.

[1262] Output: The AP content sent to the server.

[1263] Specific operation: The user terminal converts the contents of the AP into text and sends it to the server.

[1264] Step 4:

[1265] The server adds the new AP to the AP list and records the assignee and deadline.

[1266] Input: Transmitted AP content data.

[1267] Output: Updated AP list.

[1268] Specific operation: The server adds the received AP to a list and records the person in charge and the deadline.

[1269] Step 5:

[1270] The server will notify the appropriate person of this information.

[1271] Input: Updated AP list.

[1272] Output: Notification sent to assignee.

[1273] Specific operation: The server notifies the person in charge of the AP contents and deadline.

[1274] Step 6:

[1275] The user terminal displays the AP information sent from the server, and the information is checked by the person in charge.

[1276] Input: AP information sent from the server.

[1277] Output: AP information confirmed by the person in charge.

[1278] Specific operation: The user terminal displays the AP information, and the person in charge checks it.

[1279] -Simultaneous translation function-

[1280] Step 1:

[1281] The server initializes the multilingual translation API.

[1282] Input: Translation API initialization directive.

[1283] Output: Initialized translation API.

[1284] Specific operation: The server initializes a multilingual translation API when the conference starts.

[1285] Step 2:

[1286] Users speak in different languages ​​during a conference.

[1287] Input: What users say in different languages.

[1288] Output: None.

[1289] Specific Action: A user speaks in a different language (e.g., English and French).

[1290] Step 3:

[1291] A user terminal inputs a message and sends it to the server.

[1292] Input: What the user says (voice or text data).

[1293] Output: The message sent to the server.

[1294] Specific operation: The user terminal inputs the content of the statement and sends it to the server.

[1295] Step 4:

[1296] The server uses a translation API to translate the speech into different languages.

[1297] Input: User speech data.

[1298] Output: The translated speech.

[1299] Specific operation: The server uses the translation API to translate the speech into the specified language.

[1300] Step 5:

[1301] The server sends the translation results to the user terminal.

[1302] Input: The translated statement.

[1303] Output: The translation results sent to the user's device.

[1304] Specific operation: The server sends the translation results to the user terminal.

[1305] Step 6:

[1306] The user's device displays the translation results, supporting communication between different languages.

[1307] Input: Translation results sent from the server.

[1308] Output: The translation result confirmed by the user.

[1309] Specific operation: The user's device displays the translation results, supporting communication between users who speak different languages.

[1310] ~ Emotion recognition means ~

[1311] Step 1:

[1312] The server initializes the emotion engine that performs facial expression recognition and voice analysis.

[1313] Input: Emotion engine initialization command.

[1314] Output: Initialized emotion engine.

[1315] Specific operation: The server initializes the emotion engine when the conference starts.

[1316] Step 2:

[1317] When a user speaks during a conference, facial expression and voice data are collected by a camera and microphone.

[1318] Input: User's facial and voice data.

[1319] Output: None.

[1320] Specific operation: When a user speaks, facial expression and voice data are collected using the device's camera and microphone.

[1321] Step 3:

[1322] The user terminal transmits facial expression and voice data to the server.

[1323] Input: User's facial and voice data.

[1324] Output: Facial expression and voice data sent to the server.

[1325] Specific operation: The facial expression and voice data collected by the user's device is sent to the server.

[1326] Step 4:

[1327] The server uses an emotion engine to analyze the user's emotions.

[1328] Input: facial expression and voice data.

[1329] Output: Parsed emotion data.

[1330] Specific operation: The server uses an emotion engine to analyze the user's emotions (joy, anger, sadness, stress, etc.).

[1331] Step 5:

[1332] Take appropriate action based on the emotion recognition results.

[1333] Input: Parsed emotion data.

[1334] Output: Appropriate response message or notification.

[1335] Specific operation: Based on the emotion recognition results, the server generates an appropriate response message or notification, such as a "notification encouraging relaxation."

[1336] Step 6:

[1337] The server sends these notifications to the user terminal.

[1338] Input: The generated notification.

[1339] Output: Notification sent to the user's device.

[1340] Specific operation: The server sends the generated notification to the user terminal.

[1341] Step 7:

[1342] The user's device displays a notification and provides support that corresponds to the user's emotions.

[1343] Input: The notification sent by the server.

[1344] Output: The notification that the user acknowledged.

[1345] Specific operation: The user device displays a notification and provides emotional support to the user.

[1346] The above is a specific description of each processing step.

[1347] (Application example 2)

[1348] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1349] Conventional conference systems make it difficult for all participants to speak equally, and management is complicated when there are multilingual participants or when the progress of the meeting needs to take into account the emotional state of the participants. There is also a need for centralized provision of functions such as meeting efficiency, real-time minutes generation, and simultaneous translation for multiple languages. This has resulted in inefficient meeting progress and increased emotional burden on participants.

[1350] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1351] In this invention, the server includes: means for understanding the purpose and goals of a conference and using a generative model to appropriately manage the conference progress; means for correcting bias in speech based on the number of times participants speak and scoring results; means for recording the conference content in real time and automatically generating minutes; means for managing action points and scheduling the next conference; means for providing a simultaneous translation function for multilingual conferences; means for analyzing participants' emotions based on emotion recognition and responding accordingly; and means for generating the next appropriate speech using a generative model and displaying it on a user terminal. This not only streamlines the conference progress, allows all participants to speak equally, but also enables appropriate responses based on emotions. Furthermore, smooth communication can be achieved even when there are participants who speak multiple languages.

[1352] "Meeting objectives and goals" refer to the aims and purposes that are directly related to the progress and decisions of the meeting.

[1353] A "generative model" is a type of algorithm that uses machine learning and artificial intelligence to make predictions and generate information from data.

[1354] "Appropriate management" means choosing the best method depending on the situation and controlling things so that they proceed according to plan.

[1355] The "number of times each participant speaks" refers to the number of times each participant speaks during the conference.

[1356] "Scoring results" refer to the results of evaluating participants' number of comments, content, etc., using numerical values.

[1357] "Correcting bias in speech" refers to making adjustments so that speech is not biased toward a particular participant.

[1358] "Recording in real time" means recording at the same time as the meeting is taking place.

[1359] "Automatically generating minutes" refers to using a system to transcribe meeting contents in real time and create minutes.

[1360] "Action points" refer to specific action items or tasks decided during a meeting.

[1361] "Scheduling" refers to adjusting the date and schedule of the next meeting.

[1362] A "multilingual conference" refers to a conference with participants who speak several different languages.

[1363] "Simultaneous translation function" refers to the function of translating speech content into different languages ​​in real time.

[1364] "Emotion recognition" refers to the technology of analyzing emotions from voice, facial expressions, etc., and taking appropriate action based on the results.

[1365] "Appropriate statements" refer to the most appropriate statements made in accordance with the situation and the progress of the discussion.

[1366] "User terminal" refers to a device such as a computer or smartphone used by a conference participant.

[1367] The present invention relates to a system for efficiently and neutrally conducting a conference, and further provides feedback according to the user's emotions by combining it with an emotion recognition engine. This system includes the following main means.

[1368] 1. Setting and managing meeting objectives and goals

[1369] The server uses a generative model to understand the purpose and goals of the meeting and manage the progress of the meeting appropriately. The server sets the purpose and goals according to the user and the type of meeting, and supports the progress of the meeting based on these.

[1370] 2. Correcting bias in statements

[1371] The server monitors the number of comments made by participants and the scoring results in real time, and prompts participants who have made fewer comments to make the next comment, correcting bias in comments. This allows all participants to express their opinions equally.

[1372] 3. Real-time meeting minutes generation

[1373] The device sends the user's comments to the server, which records them in real time. Meeting minutes are automatically generated from the recorded data, allowing the content of the meeting to be reviewed immediately after the end of the meeting.

[1374] 4. Action Point Management

[1375] Action points proposed during a meeting are sent from the user's device to the server, which manages them. Action points are assigned to specific people and deadlines, and the server also coordinates the schedule for the next meeting.

[1376] 5. Multilingual support and simultaneous translation function

[1377] The server uses a multilingual translation API to translate what is said in different languages ​​in real time, allowing participants who speak different languages ​​to communicate smoothly.

[1378] 6. Emotion Recognition and Feedback

[1379] The server uses an emotion engine that performs facial recognition and voice analysis to recognize the user's emotions from the content of their speech and voice data. If the user is feeling stressed, it will respond appropriately, such as displaying a message encouraging them to relax.

[1380] 7. Suggested utterance corrections using a generative model

[1381] The server uses the generative model to generate the next appropriate utterance based on the current discussion and displays it on the user's device, allowing the discussion to proceed smoothly.

[1382] The system is implemented using the following hardware and software:

[1383] Hardware:

[1384] Microphone: Used to collect what the user says.

[1385] Speaker: Used to provide audio feedback from the server to the user.

[1386] Camera: Used to collect user facial expression data.

[1387] User terminal: A device operated by a user, such as a computer or smartphone.

[1388] software:

[1389] speech_recognition: Converts speech to text using a Python library.

[1390] text_to_speech: Converts text to speech using a Python library.

[1391] EmotionEngine: An engine that analyzes emotions through facial expression recognition and voice analysis.

[1392] Translation API: Translates speech using a multilingual translation API.

[1393] One specific example is a discussion support app for factories. This app is used as follows:

[1394] 1. Meeting to discuss strategies for improving production efficiency:

[1395] The worker speaks through a microphone and the robot converts the speech into text.

[1396] The server uses a generative model to suggest the next appropriate utterance to the worker, facilitating discussion.

[1397] Encourage workers who do not speak up often to speak up and correct bias.

[1398] For example: "Worker 1, please tell us your opinion next."

[1399] 2. If you have workers who speak different languages:

[1400] What workers say is translated in real time so that other workers can understand it.

[1401] Example: "I'd like to hear other opinions on improving production efficiency."

[1402] As described above, this system allows meetings to proceed efficiently and fairly, provides feedback that takes into account the emotional state of participants, and its multilingual support allows smooth communication between participants who speak different languages.

[1403] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1404] Step 1:

[1405] The server sets the purpose and goals of the meeting. The user inputs the purpose and goals of the meeting, and the server stores them in its internal memory. The input data is the specific purpose of the meeting, such as "formulate a market strategy for a new product" or "decide on a specific action plan for the next quarter," and the output is the purpose and goals of the meeting stored in memory.

[1406] Step 2:

[1407] The server receives a signal to start a conference and initializes the discussion history. It receives a signal to start a conference as input and resets the discussion history. The output is the initialized discussion history.

[1408] Step 3:

[1409] A user speaks during a conference. The user speaks through a microphone, and the voice data is input to the user terminal. The input voice data is converted into text in the next step.

[1410] Step 4:

[1411] The device receives the user's speech and converts it into text using speech recognition software. The input speech data is converted into text data using the speech_recognition library. The output is the converted text data.

[1412] Step 5:

[1413] The terminal transmits the converted text data to the server. The input is the text data converted by the speech recognition software, and the output is the text data transmitted to the server.

[1414] Step 6:

[1415] The server adds the received text data to the discussion history and generates the next appropriate utterance using a generative AI model. The input is the text data and the discussion history up to that point, and the output is the generated next utterance.

[1416] Step 7:

[1417] The server sends the generated utterances to the user's device. The input is the utterances generated by the generative AI model, and the output is the utterances sent to the user's device.

[1418] Step 8:

[1419] The terminal displays the next recommended utterance received from the server to the user. The input is the utterance content sent from the server, and the output is the utterance content displayed to the user.

[1420] Step 9:

[1421] The server scores the number of times participants have spoken, identifies users who have spoken less, and generates a notification to encourage them to speak next. The input is the number of times participants have spoken, and the output is the generated notification.

[1422] Step 10:

[1423] The server sends this notification to the user terminal. The input is the generated notification, and the output is the notification sent to the user terminal.

[1424] Step 11:

[1425] The terminal displays this notification and prompts the user to say: The input is the notification sent from the server, and the output is the notification displayed to the user.

[1426] Step 12:

[1427] The server keeps recording the meeting contents in real time and saves them in a log to automatically generate minutes. The input is text data transmitted in real time, and the output is the generated minutes.

[1428] Step 13:

[1429] The server sends the minutes to the user terminal. The input is the generated minutes, and the output is the minutes sent to the user terminal.

[1430] Step 14:

[1431] The terminal displays the received minutes to the user, allowing them to check and save them. The input is the minutes sent from the server, and the output is the minutes displayed to the user.

[1432] Step 15:

[1433] During a meeting, users propose action points (APs) and input them into a terminal. The input includes the AP's specific content, the person in charge, the deadline, and other information, and the output is the AP information entered into the terminal.

[1434] Step 16:

[1435] The terminal sends the proposed AP information to the server. The input is the AP information entered by the user, and the output is the AP information sent to the server.

[1436] Step 17:

[1437] The server adds new APs to the AP list, sets the responsible party and deadline, and manages them. The input is the submitted AP information, and the output is the updated AP list.

[1438] Step 18:

[1439] The server notifies the agent of the configured AP information. The input is the updated AP list, and the output is the notification sent to the agent.

[1440] Step 19:

[1441] The terminal displays the AP information received from the server so that the person in charge can check it. The input is the AP information sent from the server, and the output is the AP information displayed on the terminal.

[1442] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1443] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1444] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1445] [Third embodiment]

[1446] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1447] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1448] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1449] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1450] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1451] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1452] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1453] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1454] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1455] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1456] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1457] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1458] The present invention relates to a system for realizing efficient and neutral meetings. Specific embodiments for carrying out the present invention are described below.

[1459] 1. Discussion Management

[1460] The server first sets the purpose and goals of the meeting. For example, the purpose could be "formulating a market strategy for a new product" or "determining a specific action plan for the next quarter." When the meeting begins, users make comments. The server receives these comments and records them as a discussion history. Then, based on the discussion history and the purpose of the meeting, the server uses a generative model to encourage the next most appropriate comment. This suggestion is displayed on the user's device, ensuring that the meeting discussion progresses in a direction consistent with the purpose.

[1461] 2. Measures to correct bias in speech

[1462] The server scores each user's comment records. This is to count the number of comments each user makes and distinguish between users who speak frequently and those who speak infrequently. To make it easier for users who speak infrequently to express their opinions, the server encourages them to speak next time. For example, by encouraging the user who has spoken the least to speak by asking, "Would you like to give your opinion next?", it corrects bias in the discussion.

[1463] 3. Automatic generation of minutes

[1464] The server records the meeting contents in real time and automatically generates minutes. As the discussion progresses, users' comments are recorded sequentially and documented in one place. After the meeting ends, the server sends the automatically generated minutes to the user's device, where they can be viewed and saved.

[1465] 4. Action point management methods

[1466] The server records and manages action points (APs) decided during meetings in real time. Each AP is assigned a person in charge and a deadline, and is managed as a task to be completed by the next meeting. For example, when an AP such as "detailed analysis of market research" is decided, the server records the content, person in charge, and deadline, and manages it accordingly. The server periodically notifies the person in charge of this information and follows up on the progress.

[1467] 5. Simultaneous translation function

[1468] The server has the ability to translate between different languages ​​in real time in multilingual conferences. For example, it can translate minutes spoken in Japanese into English and display them on the user's device. This allows smooth communication between users who speak different languages.

[1469] By combining the above functions, the present invention provides a system that enables efficient and neutral meeting conduct and automates the management of minutes and action points, allowing all participants to engage in constructive discussions.

[1470] The processing flow will be explained below.

[1471] 1. Discussion Management

[1472] server

[1473] Step 1:

[1474] The server sets the purpose and goals of the meeting and stores them in its internal memory, such as "Draw up a market strategy for a new product" or "Decide on a concrete action plan for the next quarter."

[1475] Step 2:

[1476] The server receives a signal to start a conference and initializes the discussion history.

[1477] User

[1478] Step 3:

[1479] A user speaks during a conference.

[1480] Terminal

[1481] Step 4:

[1482] The user terminal inputs the content of the message and sends it to the server.

[1483] server

[1484] Step 5:

[1485] The server adds the user's comments to the discussion history and uses a generative model to generate the next most appropriate comment.

[1486] Step 6:

[1487] The server sends the generated message to the user terminal.

[1488] Terminal

[1489] Step 7:

[1490] The user terminal displays the next recommended comment sent from the server.

[1491] 2. Measures to correct bias in speech

[1492] server

[1493] Step 1:

[1494] The server scores each user's speech record. For example, suppose user 1 spoke three times, user 2 spoke once, and user 3 spoke twice.

[1495] Step 2:

[1496] The server identifies the user with the least number of comments, which in this example is User 2.

[1497] Step 3:

[1498] The server generates a notification to prompt a specific user to speak next, such as "User 2, would you like to give your next comment?"

[1499] Step 4:

[1500] The server sends this notification to the user terminal.

[1501] Terminal

[1502] Step 5:

[1503] The user terminal displays this notification and prompts the user to make the next statement.

[1504] 3. Automatic generation of minutes

[1505] server

[1506] Step 1:

[1507] The server records the discussion history in real time and saves the content of comments sequentially.

[1508] User

[1509] Step 2:

[1510] Each user makes a statement.

[1511] server

[1512] Step 3:

[1513] The server keeps a real-time log of everything said during the meeting.

[1514] Step 4:

[1515] After the meeting ends, the server generates minutes based on the log.

[1516] Step 5:

[1517] The server transmits the generated minutes to the user terminal.

[1518] Terminal

[1519] Step 6:

[1520] The user terminal displays the minutes, allowing the user to review and save them.

[1521] 4. Action point management methods

[1522] server

[1523] Step 1:

[1524] The server initializes the Action Point (AP) list.

[1525] User

[1526] Step 2:

[1527] A user proposes a new action point (AP) during a meeting.

[1528] Terminal

[1529] Step 3:

[1530] The user device sends the contents of the new AP to the server, including specific details such as "Detailed analysis of market research," the person in charge, and the deadline.

[1531] server

[1532] Step 4:

[1533] The server adds the new AP to the AP list and records the assignee and deadline.

[1534] Step 5:

[1535] The server will notify the appropriate person of this information.

[1536] Terminal

[1537] Step 6:

[1538] The user terminal displays the AP information sent from the server, and the information is checked by the person in charge.

[1539] 5. Simultaneous translation function

[1540] server

[1541] Step 1:

[1542] The server initializes the multilingual translation API.

[1543] User

[1544] Step 2:

[1545] Users speak in different languages ​​during a conference.

[1546] Terminal

[1547] Step 3:

[1548] A user terminal inputs a message and sends it to the server.

[1549] server

[1550] Step 4:

[1551] The server uses a translation API to translate the speech into different languages.

[1552] Step 5:

[1553] The server sends the translation results to the user terminal.

[1554] Terminal

[1555] Step 6:

[1556] The user's device displays the translation results, supporting communication between different languages.

[1557] As described above, each processing step has been specifically described, which will enable efficient and neutral meeting progress, real-time minutes creation, and AP management.

[1558] Example 1

[1559] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1560] Conventional conference systems tend to lead to biased discussions, with discussions going in directions that are not aligned with the objectives of the meeting or with certain participants speaking more than others. Furthermore, the manual process of recording meeting content and managing action points reduces efficiency. Furthermore, communication between multiple languages ​​is difficult, making it difficult to share opinions in international conferences.

[1561] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1562] In this invention, the server includes a means for setting the purpose and goals of the meeting and promoting discussion based on them, a means for recording participants' comments in real time and automatically generating minutes, a means for counting the number of comments and correcting bias in comments, a means for managing action points decided during the meeting and setting responsible parties and deadlines, and a means for real-time translation in multilingual meetings. This allows the meeting to proceed efficiently and neutrally, enabling all participants to engage in constructive discussion. Furthermore, the recording of meeting content and management of action points are automated, realizing smooth communication between multiple languages.

[1563] The "purpose and goals of the meeting" refer to the specific objectives and destinations that the meeting aims to achieve, and are the factors that determine the direction of the discussion.

[1564] "Means to promote discussion" refers to methods and mechanisms for effectively advancing discussion among participants by suggesting appropriate comments and next steps based on the purpose and goals of the meeting.

[1565] "Means for recording speech content in real time" refers to technology or devices that record participants' speech on the spot and save it as digital data.

[1566] "Means for automatically generating minutes" refers to technology or methods for automatically creating official minutes after a meeting based on recorded remarks and meeting progress information.

[1567] "Means for counting speech counts" refers to a mechanism for automatically recording the number of times each participant speaks during a meeting and aggregating that data.

[1568] "Measures to correct bias in speaking" refers to methods and mechanisms to prevent certain participants from speaking too much and to provide all participants with an opportunity to express their opinions fairly.

[1569] "Action point management means" refers to a system or method for recording specific work items decided during meetings, along with the person responsible and deadline, and tracking their progress and completion status.

[1570] "Means for real-time translation in multilingual meetings" refers to techniques and methods for instantly translating speech into other languages ​​to facilitate communication between participants who speak different languages.

[1571] The present invention relates to a system for conducting and managing a multifunctional and automated conference in which a server, terminals, and users cooperate with each other to realize efficient and neutral conferences.

[1572] 1. Setting the purpose and goals of the meeting

[1573] The server first provides a form for the meeting organizer to input the purpose and goals of the meeting. For example, the purpose of the meeting might be to "formulate a market strategy for a new product," and the goal might be to "determine a specific action plan for the next quarter." This data is saved and used to guide the meeting.

[1574] 2. Real-time recording of speech content and automatic generation of meeting minutes

[1575] When a meeting begins, the server recognizes speech in real time and transcribes it. Specific software used is Google Cloud Speech-to-Text, for example. The recognized speech is stored in a database and automatically generated as minutes after the meeting ends. Generative AI models such as OpenAI's GPT-4 are used to generate the minutes.

[1576] 3. Scoring to correct bias in statements

[1577] The server records the number of comments made by each user and performs scoring based on this. When a user who has made few comments is identified, a message is generated to prompt that user to make the next comment. For example, a prompt message such as "Would you like to give your next comment?" is generated and displayed on the user's terminal.

[1578] 4. Action Point Management

[1579] The server records the action points (APs) decided during the meeting and assigns each one a responsible person and deadline. For example, if an AP such as "detailed analysis of market research" is decided, the content, responsible person, and deadline are saved. The server periodically notifies the responsible person of this information and follows up on the progress.

[1580] 5. Multilingual simultaneous translation function

[1581] The server translates between different languages ​​in real time during a multilingual conference. For example, if a user says in Japanese, "Let's discuss the market strategy for the next quarter," the server translates it into English and displays the translated result, "Let's discuss the market strategy for the next quarter," on the user's device.

[1582] Specific examples

[1583] For example, if the purpose of a meeting is set as "Draft a market strategy for a new product" and someone says, "Let's think about what the target market for the new product should be," the server will use speech recognition software to transcribe this and store it in a database.Then, using a generative AI model, it will generate a prompt such as, "Please give us some specific ideas for determining the target market," and display it on the user's device.

[1584] As a result, the present invention provides a system that enables efficient and neutral meeting proceedings and automates the management of minutes and action points, allowing all participants to engage in constructive discussions.

[1585] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1586] Step 1:

[1587] The server provides a form for the meeting organizer to input the purpose and goals of the meeting. In the form, users input the purpose, "to formulate a market strategy for a new product," and the goal, "to determine a specific action plan for the next quarter." The input data is sent to the server and stored in a database. This helps ensure that the progress of discussions during the meeting is in line with the purpose and goals.

[1588] Step 2:

[1589] When a user speaks during a meeting, the speech is sent as audio data via the microphone to the server. The server then uses speech recognition software (e.g., Google Cloud Speech-to-Text) to convert the audio data into text data. This text data is then stored in a database. This allows the content of speech during the meeting to be recorded in real time.

[1590] Step 3:

[1591] The server generates prompts to encourage the next appropriate utterance based on the utterances recorded in real time. The server inputs the recorded utterances and the purpose of the meeting into a generative AI model (e.g., OpenAI's GPT-4) to generate recommendations for the next utterance. The generated prompts (e.g., "Next, let's list specific regions that could be considered as target markets") are displayed on the user's device, thereby encouraging the discussion to proceed in a direction that is in line with the purpose.

[1592] Step 4:

[1593] The server measures the number of comments made by each user in real time and stores it in a database for counting the number of comments. The server scores the data on the number of comments and identifies users who have made few comments. When a user who has made few comments is identified, the server generates a message saying, "Would you like to give your opinion next?" and displays it on the user's device. This corrects any bias in comments.

[1594] Step 5:

[1595] The server records the action points (APs) proposed during the meeting. The server receives the APs proposed by the users (e.g., "Detailed analysis of market research") as input and stores them in a database along with the person in charge and the deadline. After the meeting ends, the server periodically sends notifications to the person in charge to follow up on the progress. This allows for efficient management and execution of action points.

[1596] Step 6:

[1597] The server translates between different languages ​​in real time during a multilingual conference. When a user speaks in Japanese, the server uses translation software to translate this voice data into another language (e.g., English). The translated text is then displayed on the user's device. For example, "Let's discuss the market strategy for the next quarter" is translated as "Let's discuss the market strategy for the next quarter." This facilitates communication between users who speak different languages.

[1598] Through these steps, the system makes meetings more efficient and neutral, providing an environment in which all participants can engage in constructive discussions.

[1599] (Application example 1)

[1600] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1601] Conventional conference systems have problems such as difficulty in managing the progress of discussions, correcting bias in speech, creating minutes in real time, supporting multiple languages, and properly managing action items. In particular, communication breakdowns due to bias in speech and inadequate multilingual support, and insufficient follow-up on decisions are prominent, making it difficult to achieve efficient and neutral conference proceedings.

[1602] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1603] In this invention, the server includes means for using a generative AI model that understands the purpose and goals of a meeting and appropriately manages the progress of the meeting, means for correcting bias in comments based on the number of times participants speak and scoring results, means for recording meeting content in real time and automatically generating minutes, means for managing action items and scheduling the next meeting, means for providing a simultaneous translation function for multilingual meetings, means for a generative AI model to suggest appropriate next comments based on comments made during the meeting, means for generating minutes based on the comments made and distributing them to participants, and means for recording action items decided in the meeting, setting responsible parties and deadlines, and following up on progress. This makes the progress of meetings more efficient and enables neutral and constructive discussions.

[1604] A "meeting" is a gathering of multiple participants to discuss and make decisions in order to achieve a specific purpose or goal.

[1605] A "generative AI model" is an artificial intelligence model that supports speech content and decision-making based on input data.

[1606] A "server" is a computer system that processes data and provides information.

[1607] "Number of speeches" refers to the number of times each participant spoke during the conference.

[1608] The "scoring result" is the result of evaluating the number of statements and their content based on specific criteria and converting them into a score.

[1609] "Bias" refers to a situation in which the fairness of a discussion is compromised when comments and opinions are concentrated among certain participants.

[1610] "Real-time" refers to information processing at the exact moment the meeting is taking place.

[1611] "Minutes" are a written record of what was said and the results of discussions during a meeting.

[1612] "Action items" are specific actions or tasks decided upon during the meeting.

[1613] "Scheduling" refers to adjusting the schedules of participants and deciding the date and time of the next meeting.

[1614] "Multilingual support" is a function that enables participants who speak multiple languages ​​to communicate smoothly.

[1615] The "simultaneous translation function" is a function that translates between different languages ​​in real time.

[1616] "Participants" refers to people who attend meetings and are involved in discussions and decision-making.

[1617] "Appropriate content" means useful and accurate content that is in line with the purpose and goals of the meeting.

[1618] "Progress" indicates the progress and achievement of action items.

[1619] "Follow-up" refers to monitoring the progress of an action item and responding or adjusting accordingly.

[1620] The present invention relates to a system for realizing efficiency and neutrality in a meeting. Specific embodiments for carrying out the present invention are described below.

[1621] 1. System Configuration

[1622] The server uses hardware and software equipped with a generative AI model, database, and multilingual translation function. The server operates in a cloud computing environment, including the OpenAI API. Client devices include smartphones, tablets, and PCs, and users use these to participate in the conference.

[1623] 2. Meeting progress management

[1624] The server receives initial data to set the purpose and goals of the meeting and uses a generative AI model to appropriately manage the progress of the meeting. For example, if the purpose is set as "formulating a market strategy for a new product," the server analyzes the user's comments in real time and suggests the next appropriate comment. This allows the meeting to proceed in a direction that is consistent with the purpose.

[1625] 3. Correcting bias in statements

[1626] The server scores the number of comments made by each user and prompts users who have made fewer comments to make the next comment. For example, by notifying the user who has made the fewest comments, "Would you like to give your opinion next?", it is possible to correct bias in comments and promote neutral discussions.

[1627] 4. Automatic generation of meeting minutes

[1628] The server records what is said during the meeting in real time and automatically generates minutes. These minutes are automatically generated after the meeting ends and distributed to the participants' client devices. For example, if a user says, "I will talk about the results of our market research," this will be automatically recorded in the minutes.

[1629] 5. Managing Action Items

[1630] The server records the action items decided in the meeting and assigns the person in charge and deadline. These are saved in a database and notifications are sent to the person in charge to follow up on the progress as appropriate. For example, if an action item such as "Detailed analysis of market research" is set, the person in charge and deadline are recorded based on that and reminders are sent as appropriate.

[1631] 6. Multilingual simultaneous translation

[1632] In multilingual conferences, the server translates between different languages ​​in real time. For example, it translates what is said in Japanese into English and displays it on the device of an English-speaking user. This allows smooth communication between users who speak different languages.

[1633] 7. Suggested speech using generative AI models

[1634] The server uses a generative AI model to suggest the next most appropriate utterance based on the user's utterance. For example, if the user says, "I'll talk about the results of our market research," the generative AI model will suggest, "Let's consider the next action while looking at specific data."

[1635] Prompt Sentence Examples

[1636] "Suggest appropriate discussion points to follow the statement 'I'll talk about the results of my market research.'"

[1637] As described above, the present invention provides a system that realizes efficient and neutral meetings, corrects bias in discussions, and facilitates multilingual meetings.

[1638] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1639] Step 1:

[1640] The server receives initial data for setting the purpose and goals of the meeting. As input, information about the purpose and goals of the meeting is given and recorded in the database. As output, the set purpose and goals are saved on the server.

[1641] Step 2:

[1642] The server uses a generative AI model to analyze what is said during the meeting in real time. As input, the voice data spoken by the user is sent to the server, which converts it into text data. The generative AI model generates the next appropriate utterance based on this text data. As output, the generated next utterance is displayed on the user's device.

[1643] Step 3:

[1644] The server records the number of times each user has spoken and performs scoring. Information about the content of the speech and the speaker is sent to the server as input. The server records this in a database and counts the number of times each user has spoken. The server saves the scoring results as output.

[1645] Step 4:

[1646] The server generates a message to encourage users who have not spoken often to speak next. The scoring results are used as input. The server identifies the user who has spoken the least and generates a message such as "Would you like to give your next opinion?". As output, this message is sent to the device of the user.

[1647] Step 5:

[1648] The server records the meeting contents in real time and automatically generates minutes. As input, users' speech text is continuously sent to the server and recorded in a database. The server creates minutes based on this. As output, minutes that are updated in real time are generated and distributed to participants after the meeting ends.

[1649] Step 6:

[1650] The server records the action items decided during the meeting and sets the person in charge and deadline. As input, the information of the action items decided during the discussion, the person in charge, and the deadline are sent to the server. The server records this in a database and sets a schedule for following up on the progress. As output, the set action items are saved and reminders are executed according to the schedule.

[1651] Step 7:

[1652] The server translates between different languages ​​in real time during multilingual conferences. Each user's comment text and the language information to be translated are sent to the server as input. The server translates this using a multilingual translation API, and the translation results are displayed on other users' devices as output.

[1653] Through the above steps, the present invention makes the progress of the conference more efficient and facilitates smooth communication between participants.

[1654] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1655] The present invention relates to a system that realizes efficient and neutral meetings, and further combines it with an emotion engine that recognizes the emotions of users. Specific embodiments for carrying out the present invention are described below.

[1656] 1. Discussion Management

[1657] server

[1658] 1. The server sets the purpose and goals of the meeting and stores them in its internal memory, such as "Draw up a market strategy for a new product" or "Decide on a concrete action plan for the next quarter."

[1659] 2. Receive the signal to start the meeting and initialize the discussion history.

[1660] User

[1661] 3. A user speaks during the conference.

[1662] Terminal

[1663] 4. The user's device inputs the message and sends it to the server.

[1664] server

[1665] 5. The server adds the user's comment to the discussion history and uses the generative model to generate the next most appropriate comment.

[1666] 6. The server sends the generated message to the user's device.

[1667] Terminal

[1668] 7. The user terminal displays the next recommended comment sent from the server.

[1669] 2. Measures to correct bias in speech

[1670] server

[1671] 1. The server scores each user's speech record. For example, suppose user 1 spoke three times, user 2 spoke once, and user 3 spoke twice.

[1672] 2. Identify the user with the least number of comments, in this example User 2.

[1673] 3. Generate a notification to encourage a specific user to speak up, such as "User 2, would you like to give your next comment?"

[1674] 4. The server sends this notification to the user terminal.

[1675] Terminal

[1676] 5. The user device displays this notification and prompts the user to make the next statement.

[1677] 3. Automatic generation of minutes

[1678] server

[1679] 1. The server records the discussion history in real time and saves the content of comments sequentially.

[1680] User

[1681] 2. Each user makes a statement.

[1682] server

[1683] 3. The server keeps a real-time log of all statements made during the meeting.

[1684] 4. After the meeting, generate minutes based on the logs.

[1685] 5. The generated minutes are sent to the user's terminal.

[1686] Terminal

[1687] 6. The user terminal displays the minutes, allowing the user to review and save them.

[1688] 4. Action point management methods

[1689] server

[1690] 1. The server initializes the Action Point (AP) list.

[1691] User

[1692] 2. A user proposes a new action point (AP) during a meeting.

[1693] Terminal

[1694] 3. The user device sends the contents of the new AP to the server, including specific details such as "Detailed analysis of market research," the person in charge, and the deadline.

[1695] server

[1696] 4. The server adds the new AP to the AP list and records the assignee and deadline.

[1697] 5. The server will notify the appropriate person of this information.

[1698] Terminal

[1699] 6. The user terminal displays the AP information sent from the server, and the information is confirmed by the person in charge.

[1700] 5. Simultaneous translation function

[1701] server

[1702] 1. The server initializes the multilingual translation API.

[1703] User

[1704] 2. Users speak in different languages ​​during a conference.

[1705] Terminal

[1706] 3. The user terminal inputs a message and sends it to the server.

[1707] server

[1708] 4. The server uses a translation API to translate the speech into a different language.

[1709] 5. The server sends the translation results to the user's device.

[1710] Terminal

[1711] 6. The user's device displays the translation results, supporting communication between different languages.

[1712] 6. Emotion recognition means

[1713] server

[1714] 1. The server initializes the emotion engine that performs facial expression recognition and voice analysis.

[1715] User

[1716] 2. When a user speaks during a meeting, facial expression and voice data are collected by a camera and microphone.

[1717] Terminal

[1718] 3. The user device sends facial expression and voice data to the server.

[1719] server

[1720] 4. The server uses an emotion engine to analyze the user's emotions (e.g., joy, anger, sadness, stress, etc.).

[1721] 5. Based on the emotion recognition results, appropriate responses are taken. For example, if a user is feeling stressed, a message or notification is generated to encourage relaxation.

[1722] 6. The server sends these notifications to the user device.

[1723] Terminal

[1724] 7. The user's device displays a notification and provides support that corresponds to the user's emotions.

[1725] As described above, the present invention realizes efficient, neutral, and user-interactive conference proceedings, and provides a comfortable conference environment through real-time minutes creation, AP management, and user emotion recognition.

[1726] The processing flow will be explained below.

[1727] 1. Discussion Management

[1728] server

[1729] Step 1:

[1730] The server sets the purpose and goals of the meeting and stores them in its internal memory, such as "Draw up a market strategy for a new product" or "Decide on a concrete action plan for the next quarter."

[1731] Step 2:

[1732] The server receives a signal to start a conference and initializes the discussion history.

[1733] User

[1734] Step 3:

[1735] A user speaks during a conference.

[1736] Terminal

[1737] Step 4:

[1738] The user terminal inputs the content of the message and sends it to the server.

[1739] server

[1740] Step 5:

[1741] The server adds the user's comments to the discussion history and uses a generative model to generate the next most appropriate comment.

[1742] Step 6:

[1743] The server sends the generated message to the user terminal.

[1744] Terminal

[1745] Step 7:

[1746] The user terminal displays the next recommended comment sent from the server.

[1747] 2. Measures to correct bias in speech

[1748] server

[1749] Step 1:

[1750] The server scores each user's speech record. For example, suppose user 1 spoke three times, user 2 spoke once, and user 3 spoke twice.

[1751] Step 2:

[1752] The server identifies the user with the least number of comments, in this example user 2.

[1753] Step 3:

[1754] The server generates a notification to prompt a specific user to speak next, such as "User 2, would you like to give your next comment?"

[1755] Step 4:

[1756] The server sends this notification to the user terminal.

[1757] Terminal

[1758] Step 5:

[1759] The user terminal displays this notification and prompts the user to make the next statement.

[1760] 3. Automatic generation of minutes

[1761] server

[1762] Step 1:

[1763] The server records the discussion history in real time and saves the content of comments sequentially.

[1764] User

[1765] Step 2:

[1766] Each user makes a statement.

[1767] server

[1768] Step 3:

[1769] The server keeps a real-time log of everything said during the meeting.

[1770] Step 4:

[1771] After the meeting ends, the server generates minutes based on the log.

[1772] Step 5:

[1773] The server transmits the generated minutes to the user terminal.

[1774] Terminal

[1775] Step 6:

[1776] The user terminal displays the minutes, allowing the user to review and save them.

[1777] 4. Action point management methods

[1778] server

[1779] Step 1:

[1780] The server initializes the Action Point (AP) list.

[1781] User

[1782] Step 2:

[1783] A user proposes a new action point (AP) during a meeting.

[1784] Terminal

[1785] Step 3:

[1786] The user device sends the contents of the new AP to the server, including specific details such as "Detailed analysis of market research," the person in charge, and the deadline.

[1787] server

[1788] Step 4:

[1789] The server adds the new AP to the AP list and records the assignee and deadline.

[1790] Step 5:

[1791] The server will notify the appropriate person of this information.

[1792] Terminal

[1793] Step 6:

[1794] The user terminal displays the AP information sent from the server, and the information is checked by the person in charge.

[1795] 5. Simultaneous translation function

[1796] server

[1797] Step 1:

[1798] The server initializes the multilingual translation API.

[1799] User

[1800] Step 2:

[1801] Users speak in different languages ​​during a conference.

[1802] Terminal

[1803] Step 3:

[1804] A user terminal inputs a message and sends it to the server.

[1805] server

[1806] Step 4:

[1807] The server uses a translation API to translate the speech into different languages.

[1808] Step 5:

[1809] The server sends the translation results to the user terminal.

[1810] Terminal

[1811] Step 6:

[1812] The user's device displays the translation results, supporting communication between different languages.

[1813] 6. Emotion recognition means

[1814] server

[1815] Step 1:

[1816] The server initializes the emotion engine that performs facial expression recognition and voice analysis.

[1817] User

[1818] Step 2:

[1819] When a user speaks during a conference, facial expression and voice data are collected by a camera and microphone.

[1820] Terminal

[1821] Step 3:

[1822] The user terminal transmits facial expression and voice data to the server.

[1823] server

[1824] Step 4:

[1825] The server uses an emotion engine to analyze the user's emotions (e.g., joy, anger, sadness, stress, etc.).

[1826] Step 5:

[1827] Based on the emotion recognition results, appropriate responses are taken. For example, if a user is feeling stressed, a message or notification is generated to encourage relaxation.

[1828] Step 6:

[1829] The server sends these notifications to the user terminal.

[1830] Terminal

[1831] Step 7:

[1832] The user's device displays a notification and provides support that corresponds to the user's emotions.

[1833] As described above, the present invention realizes efficient, neutral, and user-interactive conference proceedings, and provides a comfortable conference environment through real-time minutes creation, AP management, and user emotion recognition.

[1834] Example 2

[1835] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1836] Conventional conference systems lack sufficient discussion progress management, speech bias correction, real-time minutes creation, action point management, and multilingual support, making it difficult to ensure meeting efficiency and neutrality. Furthermore, they do not take participants' emotions into consideration, making it difficult to provide an optimal conference environment. The present invention aims to solve these problems.

[1837] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1838] In this invention, the server includes means for setting the purpose and goal of the meeting and saving it in an internal storage device, means for generating speech content using a generative AI model to manage the progress of the meeting, means for correcting bias in speech based on the number of times participants speak and scoring results, means for recording the content of the meeting in real time and automatically generating minutes after the meeting ends, means for managing action points proposed during the meeting and recording the person in charge and deadline, means for providing a simultaneous translation function in multilingual meetings, and means for recognizing user emotions and taking appropriate action based on the results, thereby enabling efficient, neutral, and user-interactive meeting progress.

[1839] "Means for setting the purpose and goal of the meeting and saving it in the internal memory" is a function for determining the specific purpose and final goal of the meeting before the meeting starts and saving it in the internal memory.

[1840] "Means for generating speech content using a generative AI model to manage the progress of a meeting" refers to a function that uses a generative AI model during a meeting to automatically generate appropriate speech content, helping to ensure the meeting progresses smoothly.

[1841] "Means to correct bias in speech based on the number of times participants speak and the scoring results" is a function that records and analyzes the number of times each participant speaks and their score, and corrects bias in speech by encouraging specific participants to speak.

[1842] "A means of recording meeting content in real time and automatically generating minutes after the meeting" is a function that records statements and discussions made during a meeting in real time and automatically creates minutes after the meeting based on that data.

[1843] "A means to manage action points proposed during a meeting and record the person in charge and deadline" is a function for recording new action points proposed during a meeting and setting and managing the person in charge and deadline.

[1844] "Means for providing simultaneous translation functionality in multilingual conferences" refers to a functionality for translating in real time the statements of conference participants who speak different languages, and simultaneously supporting communication between different languages.

[1845] "Means for recognizing the user's emotions and taking appropriate action based on the results" refers to a function that analyzes the user's facial expressions and voice during a meeting to recognize their emotions, and then takes appropriate action or notifies them based on the results.

[1846] The present invention is a system that integrates meeting efficiency, ensuring neutrality, and user emotion recognition. A specific embodiment of this system will be described below.

[1847] This system is composed of a combination of a server, a terminal, and a user interface. The server is mainly responsible for calculation processing and data storage, while the terminal functions as an interface with the user. Each function is explained in detail below.

[1848] 1. Discussion Management

[1849] The server stores the purpose and goal of the meeting in its internal storage device. For example, it can be set to "formulate a market strategy for a new product." This information is managed based on the meeting start signal that the server receives when the meeting begins. When a user speaks during the meeting, the content of that speech is sent to the server via the user's device. The server adds the received speech to the discussion history, and based on that history, uses a generative AI model to generate the next most appropriate speech. This generated speech is sent back to the user's device and displayed to the user. As a concrete example, if we consider a meeting about a market strategy for a new product, the following actions will occur:

[1850] User 1: "Look at last year's sales data. It shows us new market potential."

[1851] Server: "User 2, next we'd like to hear your thoughts on new target markets."

[1852] 2. Measures to correct bias in speech

[1853] The server records and scores the number of times each user has spoken. For example, if User 1 has spoken three times, User 2 has spoken once, and User 3 has spoken twice, the server generates a notification to encourage the user with the fewest number of speeches (User 2 in this example) to speak next. The notification might be something like, "User 2, would you like to give your opinion next?" These notifications are sent to and displayed on the user's device.

[1854] Example prompt for a generative AI model:

[1855] "Please generate a message to notify User 2 that he has not been posting much and encourage him to post more."

[1856] 3. Automatic generation of minutes

[1857] The server records all statements made during the meeting in real time and saves them sequentially. When the meeting ends, the server automatically generates minutes based on the saved data and sends them to the user's device. The user's device displays the minutes, allowing the user to view and save them.

[1858] 4. Action point management methods

[1859] The server manages new action points (APs) proposed during a conference. When a user proposes a new AP, the user terminal sends the content to the server. The server adds the AP to a list and records the person in charge and the deadline. If necessary, the server sends a notification to the person in charge. This notification is confirmed on the user terminal.

[1860] Example prompt for a generative AI model:

[1861] "Add new action points to your list and record who is responsible and when they are due."

[1862] 5. Simultaneous translation function

[1863] The server initializes a multilingual translation API and translates any comments made in different languages ​​during the meeting. The user's device inputs the comments and sends them to the server, which then uses the translation API to translate the comments into the different language. The translation results are sent to the user's device and displayed.

[1864] Example prompt for a generative AI model:

[1865] "Translate what you say into the specified language and display the results."

[1866] 6. Emotion recognition means

[1867] The server initializes an emotion engine that performs facial expression recognition and voice analysis. When the user speaks, facial expression and voice data is collected by a camera and microphone and sent to the server via the user's device. The server uses the emotion engine to analyze emotions such as joy, anger, sadness, and stress. Based on the analysis results, the server generates appropriate response messages and notifications and sends them to the user's device. The user's device displays these notifications and responds to the user's emotions.

[1868] Example prompt for a generative AI model:

[1869] "If the user is feeling stressed, generate a notification to encourage relaxation."

[1870] As described above, the present invention realizes efficient, neutral, and user-interactive conference proceedings, and provides a comfortable conference environment through real-time minutes creation, action point management, and emotion recognition.

[1871] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1872] Specific processing steps of the system program

[1873] ~Method of managing discussions~

[1874] Step 1:

[1875] The server sets the purpose and goals of the meeting and stores them in an internal storage device.

[1876] Input: Initial setup data, including the purpose and goals of the meeting.

[1877] Output: Meeting objectives and goals data stored in internal memory.

[1878] Specific operation: The server sets goals such as "Draw up a market strategy for a new product" or "Decide on a specific action plan for the next quarter" and stores them in its internal memory.

[1879] Step 2:

[1880] The server receives a signal to start the conference.

[1881] Input: Signal to start a conference.

[1882] Output: Initialized discussion history data.

[1883] Specific operation: The server receives a conference start signal and initializes the internal discussion history.

[1884] Step 3:

[1885] A user speaks during a conference.

[1886] Input: What the user says.

[1887] Output: None.

[1888] Specific action: The user expresses their opinion or information.

[1889] Step 4:

[1890] The user terminal inputs the content of the message and sends it to the server.

[1891] Input: What the user says (voice or text data).

[1892] Output: The speech data sent to the server.

[1893] Specific operation: The user's device converts the spoken content into text using voice recognition technology or keyboard input and sends it to the server.

[1894] Step 5:

[1895] The server adds the user's comments to the discussion history and uses a generative AI model to generate the next most appropriate comment.

[1896] Input: Speech content data.

[1897] Output: The following utterance is generated:

[1898] Specific operation: The server adds the comment to the discussion history and uses the generative AI model to generate the next most appropriate comment.

[1899] Step 6:

[1900] The server sends the generated message to the user terminal.

[1901] Input: The generated utterance.

[1902] Output: The speech sent to the user's device.

[1903] Specific operation: The server sends the generated speech to the user terminal.

[1904] Step 7:

[1905] The user terminal displays the next recommended comment sent from the server.

[1906] Input: The message sent from the server.

[1907] Output: What is displayed to the user.

[1908] Specific operation: The user device displays the recommended comments and shows them to the user.

[1909] ~Methods to correct bias in speech~

[1910] Step 1:

[1911] The server scores each user's speech record.

[1912] Input: Number of speeches per user.

[1913] Output: Scoring result data.

[1914] Specific operation: The server records the number of times each user has spoken and uses that data to perform scoring.

[1915] Step 2:

[1916] Identify users who have spoken the least.

[1917] Input: Scoring result data.

[1918] Output: Specific information about users who speak less frequently.

[1919] Specific operation: The server analyzes the scoring results and identifies the user who has spoken the least.

[1920] Step 3:

[1921] The server generates a notification prompting the user to make the next statement and sends it to the user terminal.

[1922] Input: User specific information.

[1923] Output: Speech promotion notification data.

[1924] Specific operation: The server generates a notification such as "User 2, would you like to give your opinion next?" and sends it to the user terminal.

[1925] Step 4:

[1926] The user terminal displays this notification and prompts the user to make the next statement.

[1927] Input: Notification data sent by the server.

[1928] Output: What is displayed to the user.

[1929] Specific operation: The user device displays a notification and prompts the user to make the next statement.

[1930] -Method for automatically generating meeting minutes-

[1931] Step 1:

[1932] The server records the meeting contents in real time.

[1933] Input: User speech data.

[1934] Output: Real-time recording data.

[1935] Specific operation: The server records and saves what is said in real time.

[1936] Step 2:

[1937] Each user makes a statement.

[1938] Input: What the user says.

[1939] Output: None.

[1940] Specific operation: The user sends the content of the statement to the server via the terminal.

[1941] Step 3:

[1942] The server keeps a real-time log of everything said during the meeting.

[1943] Input: What the user says.

[1944] Output: Recorded log data.

[1945] What it does: The server keeps adding what is said to the record in real time.

[1946] Step 4:

[1947] After the meeting, minutes are generated based on the logs.

[1948] Input: Recorded log data.

[1949] Output: The generated transcript.

[1950] Specific operation: After the meeting ends, the server generates minutes based on the recorded log data.

[1951] Step 5:

[1952] The generated minutes are sent to the user terminal.

[1953] Input: Generated minutes data.

[1954] Output: The minutes sent to the user's terminal.

[1955] Specific operation: The server sends the generated minutes to the user terminal.

[1956] Step 6:

[1957] The user terminal displays the minutes, allowing the user to review and save them.

[1958] Input: Minutes data sent from the server.

[1959] Output: The minutes confirmed by the user.

[1960] Specific operation: The user terminal displays the minutes and allows the user to review and save them.

[1961] ~Action point management method~

[1962] Step 1:

[1963] The server initializes the Action Point (AP) list.

[1964] Input: Conference start signal.

[1965] Output: Initialized AP list.

[1966] Specific operation: The server initializes the AP list when a new conference starts.

[1967] Step 2:

[1968] A user proposes a new action point (AP) during a meeting.

[1969] Input: User suggestions.

[1970] Output: None.

[1971] Specific operation: A user proposes a new AP.

[1972] Step 3:

[1973] The user terminal sends the contents of the new AP to the server.

[1974] Input: Text input of AP by user.

[1975] Output: The AP content sent to the server.

[1976] Specific operation: The user terminal converts the contents of the AP into text and sends it to the server.

[1977] Step 4:

[1978] The server adds the new AP to the AP list and records the assignee and deadline.

[1979] Input: Transmitted AP content data.

[1980] Output: Updated AP list.

[1981] Specific operation: The server adds the received AP to a list and records the person in charge and the deadline.

[1982] Step 5:

[1983] The server will notify the appropriate person of this information.

[1984] Input: Updated AP list.

[1985] Output: Notification sent to assignee.

[1986] Specific operation: The server notifies the person in charge of the AP contents and deadline.

[1987] Step 6:

[1988] The user terminal displays the AP information sent from the server, and the information is checked by the person in charge.

[1989] Input: AP information sent from the server.

[1990] Output: AP information confirmed by the person in charge.

[1991] Specific operation: The user terminal displays the AP information, and the person in charge checks it.

[1992] -Simultaneous translation function-

[1993] Step 1:

[1994] The server initializes the multilingual translation API.

[1995] Input: Translation API initialization directive.

[1996] Output: Initialized translation API.

[1997] Specific operation: The server initializes a multilingual translation API when the conference starts.

[1998] Step 2:

[1999] Users speak in different languages ​​during a conference.

[2000] Input: What users say in different languages.

[2001] Output: None.

[2002] Specific Action: A user speaks in a different language (e.g., English and French).

[2003] Step 3:

[2004] A user terminal inputs a message and sends it to the server.

[2005] Input: What the user says (voice or text data).

[2006] Output: The message sent to the server.

[2007] Specific operation: The user terminal inputs the content of the statement and sends it to the server.

[2008] Step 4:

[2009] The server uses a translation API to translate the speech into different languages.

[2010] Input: User speech data.

[2011] Output: The translated speech.

[2012] Specific operation: The server uses the translation API to translate the speech into the specified language.

[2013] Step 5:

[2014] The server sends the translation results to the user terminal.

[2015] Input: The translated statement.

[2016] Output: The translation results sent to the user's device.

[2017] Specific operation: The server sends the translation results to the user terminal.

[2018] Step 6:

[2019] The user's device displays the translation results, supporting communication between different languages.

[2020] Input: Translation results sent from the server.

[2021] Output: The translation result confirmed by the user.

[2022] Specific operation: The user's device displays the translation results, supporting communication between users who speak different languages.

[2023] ~ Emotion recognition means ~

[2024] Step 1:

[2025] The server initializes the emotion engine that performs facial expression recognition and voice analysis.

[2026] Input: Emotion engine initialization command.

[2027] Output: Initialized emotion engine.

[2028] Specific operation: The server initializes the emotion engine when the conference starts.

[2029] Step 2:

[2030] When a user speaks during a conference, facial expression and voice data are collected by a camera and microphone.

[2031] Input: User's facial and voice data.

[2032] Output: None.

[2033] Specific operation: When a user speaks, facial expression and voice data are collected using the device's camera and microphone.

[2034] Step 3:

[2035] The user terminal transmits facial expression and voice data to the server.

[2036] Input: User's facial and voice data.

[2037] Output: Facial expression and voice data sent to the server.

[2038] Specific operation: The facial expression and voice data collected by the user's device is sent to the server.

[2039] Step 4:

[2040] The server uses an emotion engine to analyze the user's emotions.

[2041] Input: facial expression and voice data.

[2042] Output: Parsed emotion data.

[2043] Specific operation: The server uses an emotion engine to analyze the user's emotions (joy, anger, sadness, stress, etc.).

[2044] Step 5:

[2045] Take appropriate action based on the emotion recognition results.

[2046] Input: Parsed emotion data.

[2047] Output: Appropriate response message or notification.

[2048] Specific operation: Based on the emotion recognition results, the server generates an appropriate response message or notification, such as a "notification encouraging relaxation."

[2049] Step 6:

[2050] The server sends these notifications to the user terminal.

[2051] Input: The generated notification.

[2052] Output: Notification sent to the user's device.

[2053] Specific operation: The server sends the generated notification to the user terminal.

[2054] Step 7:

[2055] The user's device displays a notification and provides support that corresponds to the user's emotions.

[2056] Input: The notification sent by the server.

[2057] Output: The notification that the user acknowledged.

[2058] Specific operation: The user device displays a notification and provides emotional support to the user.

[2059] The above is a specific description of each processing step.

[2060] (Application example 2)

[2061] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[2062] Conventional conference systems make it difficult for all participants to speak equally, and management is complicated when there are multilingual participants or when the progress of the meeting needs to take into account the emotional state of the participants. There is also a need for centralized provision of functions such as meeting efficiency, real-time minutes generation, and simultaneous translation for multiple languages. This has resulted in inefficient meeting progress and increased emotional burden on participants.

[2063] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[2064] In this invention, the server includes: means for understanding the purpose and goals of a conference and using a generative model to appropriately manage the conference progress; means for correcting bias in speech based on the number of times participants speak and scoring results; means for recording the conference content in real time and automatically generating minutes; means for managing action points and scheduling the next conference; means for providing a simultaneous translation function for multilingual conferences; means for analyzing participants' emotions based on emotion recognition and responding accordingly; and means for generating the next appropriate speech using a generative model and displaying it on a user terminal. This not only streamlines the conference progress, allows all participants to speak equally, but also enables appropriate responses based on emotions. Furthermore, smooth communication can be achieved even when there are participants who speak multiple languages.

[2065] "Meeting objectives and goals" refer to the aims and purposes that are directly related to the progress and decisions of the meeting.

[2066] A "generative model" is a type of algorithm that uses machine learning and artificial intelligence to make predictions and generate information from data.

[2067] "Appropriate management" means choosing the best method depending on the situation and controlling things so that they proceed according to plan.

[2068] The "number of times each participant speaks" refers to the number of times each participant speaks during the conference.

[2069] "Scoring results" refer to the results of evaluating participants' number of comments, content, etc., using numerical values.

[2070] "Correcting bias in speech" refers to making adjustments so that speech is not biased toward a particular participant.

[2071] "Recording in real time" means recording at the same time as the meeting is taking place.

[2072] "Automatically generating minutes" refers to using a system to transcribe meeting contents in real time and create minutes.

[2073] "Action points" refer to specific action items or tasks decided during a meeting.

[2074] "Scheduling" refers to adjusting the date and schedule of the next meeting.

[2075] A "multilingual conference" refers to a conference with participants who speak several different languages.

[2076] "Simultaneous translation function" refers to the function of translating speech content into different languages ​​in real time.

[2077] "Emotion recognition" refers to the technology of analyzing emotions from voice, facial expressions, etc., and taking appropriate action based on the results.

[2078] "Appropriate statements" refer to the most appropriate statements made in accordance with the situation and the progress of the discussion.

[2079] "User terminal" refers to a device such as a computer or smartphone used by a conference participant.

[2080] The present invention relates to a system for efficiently and neutrally conducting a conference, and further provides feedback according to the user's emotions by combining it with an emotion recognition engine. This system includes the following main means.

[2081] 1. Setting and managing meeting objectives and goals

[2082] The server uses a generative model to understand the purpose and goals of the meeting and manage the progress of the meeting appropriately. The server sets the purpose and goals according to the user and the type of meeting, and supports the progress of the meeting based on these.

[2083] 2. Correcting bias in statements

[2084] The server monitors the number of comments made by participants and the scoring results in real time, and prompts participants who have made fewer comments to make the next comment, correcting bias in comments. This allows all participants to express their opinions equally.

[2085] 3. Real-time meeting minutes generation

[2086] The device sends the user's comments to the server, which records them in real time. Meeting minutes are automatically generated from the recorded data, allowing the content of the meeting to be reviewed immediately after the end of the meeting.

[2087] 4. Action Point Management

[2088] Action points proposed during a meeting are sent from the user's device to the server, which manages them. Action points are assigned to specific people and deadlines, and the server also coordinates the schedule for the next meeting.

[2089] 5. Multilingual support and simultaneous translation function

[2090] The server uses a multilingual translation API to translate what is said in different languages ​​in real time, allowing participants who speak different languages ​​to communicate smoothly.

[2091] 6. Emotion Recognition and Feedback

[2092] The server uses an emotion engine that performs facial recognition and voice analysis to recognize the user's emotions from the content of their speech and voice data. If the user is feeling stressed, it will respond appropriately, such as displaying a message encouraging them to relax.

[2093] 7. Suggested utterance corrections using a generative model

[2094] The server uses the generative model to generate the next appropriate utterance based on the current discussion and displays it on the user's device, allowing the discussion to proceed smoothly.

[2095] The system is implemented using the following hardware and software:

[2096] Hardware:

[2097] Microphone: Used to collect what the user says.

[2098] Speaker: Used to provide audio feedback from the server to the user.

[2099] Camera: Used to collect user facial expression data.

[2100] User terminal: A device operated by a user, such as a computer or smartphone.

[2101] software:

[2102] speech_recognition: Converts speech to text using a Python library.

[2103] text_to_speech: Converts text to speech using a Python library.

[2104] EmotionEngine: An engine that analyzes emotions through facial expression recognition and voice analysis.

[2105] Translation API: Translates speech using a multilingual translation API.

[2106] One specific example is a discussion support app for factories. This app is used as follows:

[2107] 1. Meeting to discuss strategies for improving production efficiency:

[2108] The worker speaks through a microphone and the robot converts the speech into text.

[2109] The server uses a generative model to suggest the next appropriate utterance to the worker, facilitating discussion.

[2110] Encourage workers who do not speak up often to speak up and correct bias.

[2111] For example: "Worker 1, please tell us your opinion next."

[2112] 2. If you have workers who speak different languages:

[2113] What workers say is translated in real time so that other workers can understand it.

[2114] Example: "I'd like to hear other opinions on improving production efficiency."

[2115] As described above, this system allows meetings to proceed efficiently and fairly, provides feedback that takes into account the emotional state of participants, and its multilingual support allows smooth communication between participants who speak different languages.

[2116] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2117] Step 1:

[2118] The server sets the purpose and goals of the meeting. The user inputs the purpose and goals of the meeting, and the server stores them in its internal memory. The input data is the specific purpose of the meeting, such as "formulate a market strategy for a new product" or "decide on a specific action plan for the next quarter," and the output is the purpose and goals of the meeting stored in memory.

[2119] Step 2:

[2120] The server receives a signal to start a conference and initializes the discussion history. It receives a signal to start a conference as input and resets the discussion history. The output is the initialized discussion history.

[2121] Step 3:

[2122] A user speaks during a conference. The user speaks through a microphone, and the voice data is input to the user terminal. The input voice data is converted into text in the next step.

[2123] Step 4:

[2124] The device receives the user's speech and converts it into text using speech recognition software. The input speech data is converted into text data using the speech_recognition library. The output is the converted text data.

[2125] Step 5:

[2126] The terminal transmits the converted text data to the server. The input is the text data converted by the speech recognition software, and the output is the text data transmitted to the server.

[2127] Step 6:

[2128] The server adds the received text data to the discussion history and generates the next appropriate utterance using a generative AI model. The input is the text data and the discussion history up to that point, and the output is the generated next utterance.

[2129] Step 7:

[2130] The server sends the generated utterances to the user's device. The input is the utterances generated by the generative AI model, and the output is the utterances sent to the user's device.

[2131] Step 8:

[2132] The terminal displays the next recommended utterance received from the server to the user. The input is the utterance content sent from the server, and the output is the utterance content displayed to the user.

[2133] Step 9:

[2134] The server scores the number of times participants have spoken, identifies users who have spoken less, and generates a notification to encourage them to speak next. The input is the number of times participants have spoken, and the output is the generated notification.

[2135] Step 10:

[2136] The server sends this notification to the user terminal. The input is the generated notification, and the output is the notification sent to the user terminal.

[2137] Step 11:

[2138] The terminal displays this notification and prompts the user to say: The input is the notification sent from the server, and the output is the notification displayed to the user.

[2139] Step 12:

[2140] The server keeps recording the meeting contents in real time and saves them in a log to automatically generate minutes. The input is text data transmitted in real time, and the output is the generated minutes.

[2141] Step 13:

[2142] The server sends the minutes to the user terminal. The input is the generated minutes, and the output is the minutes sent to the user terminal.

[2143] Step 14:

[2144] The terminal displays the received minutes to the user, allowing them to check and save them. The input is the minutes sent from the server, and the output is the minutes displayed to the user.

[2145] Step 15:

[2146] During a meeting, users propose action points (APs) and input them into a terminal. The input includes the AP's specific content, the person in charge, the deadline, and other information, and the output is the AP information entered into the terminal.

[2147] Step 16:

[2148] The terminal sends the proposed AP information to the server. The input is the AP information entered by the user, and the output is the AP information sent to the server.

[2149] Step 17:

[2150] The server adds new APs to the AP list, sets the responsible party and deadline, and manages them. The input is the submitted AP information, and the output is the updated AP list.

[2151] Step 18:

[2152] The server notifies the agent of the configured AP information. The input is the updated AP list, and the output is the notification sent to the agent.

[2153] Step 19:

[2154] The terminal displays the AP information received from the server so that the person in charge can check it. The input is the AP information sent from the server, and the output is the AP information displayed on the terminal.

[2155] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[2156] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2157] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[2158] [Fourth embodiment]

[2159] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[2160] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[2161] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[2162] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[2163] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[2164] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[2165] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[2166] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[2167] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[2168] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[2169] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[2170] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[2171] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2172] The present invention relates to a system for realizing efficient and neutral meetings. Specific embodiments for carrying out the present invention are described below.

[2173] 1. Discussion Management

[2174] The server first sets the purpose and goals of the meeting. For example, the purpose could be "formulating a market strategy for a new product" or "determining a specific action plan for the next quarter." When the meeting begins, users make comments. The server receives these comments and records them as a discussion history. Then, based on the discussion history and the purpose of the meeting, the server uses a generative model to encourage the next most appropriate comment. This suggestion is displayed on the user's device, ensuring that the meeting discussion progresses in a direction consistent with the purpose.

[2175] 2. Measures to correct bias in speech

[2176] The server scores each user's comment records. This is to count the number of comments each user makes and distinguish between users who speak frequently and those who speak infrequently. To make it easier for users who speak infrequently to express their opinions, the server encourages them to speak next time. For example, by encouraging the user who has spoken the least to speak by asking, "Would you like to give your opinion next?", it corrects bias in the discussion.

[2177] 3. Automatic generation of minutes

[2178] The server records the meeting contents in real time and automatically generates minutes. As the discussion progresses, users' comments are recorded sequentially and documented in one place. After the meeting ends, the server sends the automatically generated minutes to the user's device, where they can be viewed and saved.

[2179] 4. Action point management methods

[2180] The server records and manages action points (APs) decided during meetings in real time. Each AP is assigned a person in charge and a deadline, and is managed as a task to be completed by the next meeting. For example, when an AP such as "detailed analysis of market research" is decided, the server records the content, person in charge, and deadline, and manages it accordingly. The server periodically notifies the person in charge of this information and follows up on the progress.

[2181] 5. Simultaneous translation function

[2182] The server has the ability to translate between different languages ​​in real time in multilingual conferences. For example, it can translate minutes spoken in Japanese into English and display them on the user's device. This allows smooth communication between users who speak different languages.

[2183] By combining the above functions, the present invention provides a system that enables efficient and neutral meeting conduct and automates the management of minutes and action points, allowing all participants to engage in constructive discussions.

[2184] The processing flow will be explained below.

[2185] 1. Discussion Management

[2186] server

[2187] Step 1:

[2188] The server sets the purpose and goals of the meeting and stores them in its internal memory, such as "Draw up a market strategy for a new product" or "Decide on a concrete action plan for the next quarter."

[2189] Step 2:

[2190] The server receives a signal to start a conference and initializes the discussion history.

[2191] User

[2192] Step 3:

[2193] A user speaks during a conference.

[2194] Terminal

[2195] Step 4:

[2196] The user terminal inputs the content of the message and sends it to the server.

[2197] server

[2198] Step 5:

[2199] The server adds the user's comments to the discussion history and uses a generative model to generate the next most appropriate comment.

[2200] Step 6:

[2201] The server sends the generated message to the user terminal.

[2202] Terminal

[2203] Step 7:

[2204] The user terminal displays the next recommended comment sent from the server.

[2205] 2. Measures to correct bias in speech

[2206] server

[2207] Step 1:

[2208] The server scores each user's speech record. For example, suppose user 1 spoke three times, user 2 spoke once, and user 3 spoke twice.

[2209] Step 2:

[2210] The server identifies the user with the least number of comments, which in this example is User 2.

[2211] Step 3:

[2212] The server generates a notification to prompt a specific user to speak next, such as "User 2, would you like to give your next comment?"

[2213] Step 4:

[2214] The server sends this notification to the user terminal.

[2215] Terminal

[2216] Step 5:

[2217] The user terminal displays this notification and prompts the user to make the next statement.

[2218] 3. Automatic generation of minutes

[2219] server

[2220] Step 1:

[2221] The server records the discussion history in real time and saves the content of comments sequentially.

[2222] User

[2223] Step 2:

[2224] Each user makes a statement.

[2225] server

[2226] Step 3:

[2227] The server keeps a real-time log of everything said during the meeting.

[2228] Step 4:

[2229] After the meeting ends, the server generates minutes based on the log.

[2230] Step 5:

[2231] The server transmits the generated minutes to the user terminal.

[2232] Terminal

[2233] Step 6:

[2234] The user terminal displays the minutes, allowing the user to review and save them.

[2235] 4. Action point management methods

[2236] server

[2237] Step 1:

[2238] The server initializes the Action Point (AP) list.

[2239] User

[2240] Step 2:

[2241] A user proposes a new action point (AP) during a meeting.

[2242] Terminal

[2243] Step 3:

[2244] The user device sends the contents of the new AP to the server, including specific details such as "Detailed analysis of market research," the person in charge, and the deadline.

[2245] server

[2246] Step 4:

[2247] The server adds the new AP to the AP list and records the assignee and deadline.

[2248] Step 5:

[2249] The server will notify the appropriate person of this information.

[2250] Terminal

[2251] Step 6:

[2252] The user terminal displays the AP information sent from the server, and the information is checked by the person in charge.

[2253] 5. Simultaneous translation function

[2254] server

[2255] Step 1:

[2256] The server initializes the multilingual translation API.

[2257] User

[2258] Step 2:

[2259] Users speak in different languages ​​during a conference.

[2260] Terminal

[2261] Step 3:

[2262] A user terminal inputs a message and sends it to the server.

[2263] server

[2264] Step 4:

[2265] The server uses a translation API to translate the speech into different languages.

[2266] Step 5:

[2267] The server sends the translation results to the user terminal.

[2268] Terminal

[2269] Step 6:

[2270] The user's device displays the translation results, supporting communication between different languages.

[2271] As described above, each processing step has been specifically described, which will enable efficient and neutral meeting progress, real-time minutes creation, and AP management.

[2272] Example 1

[2273] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2274] Conventional conference systems tend to lead to biased discussions, with discussions going in directions that are not aligned with the objectives of the meeting or with certain participants speaking more than others. Furthermore, the manual process of recording meeting content and managing action points reduces efficiency. Furthermore, communication between multiple languages ​​is difficult, making it difficult to share opinions in international conferences.

[2275] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[2276] In this invention, the server includes a means for setting the purpose and goals of the meeting and promoting discussion based on them, a means for recording participants' comments in real time and automatically generating minutes, a means for counting the number of comments and correcting bias in comments, a means for managing action points decided during the meeting and setting responsible parties and deadlines, and a means for real-time translation in multilingual meetings. This allows the meeting to proceed efficiently and neutrally, enabling all participants to engage in constructive discussion. Furthermore, the recording of meeting content and management of action points are automated, realizing smooth communication between multiple languages.

[2277] The "purpose and goals of the meeting" refer to the specific objectives and destinations that the meeting aims to achieve, and are the factors that determine the direction of the discussion.

[2278] "Means to promote discussion" refers to methods and mechanisms for effectively advancing discussion among participants by suggesting appropriate comments and next steps based on the purpose and goals of the meeting.

[2279] "Means for recording speech content in real time" refers to technology or devices that record participants' speech on the spot and save it as digital data.

[2280] "Means for automatically generating minutes" refers to technology or methods for automatically creating official minutes after a meeting based on recorded remarks and meeting progress information.

[2281] "Means for counting speech counts" refers to a mechanism for automatically recording the number of times each participant speaks during a meeting and aggregating that data.

[2282] "Measures to correct bias in speaking" refers to methods and mechanisms to prevent certain participants from speaking too much and to provide all participants with an opportunity to express their opinions fairly.

[2283] "Action point management means" refers to a system or method for recording specific work items decided during meetings, along with the person responsible and deadline, and tracking their progress and completion status.

[2284] "Means for real-time translation in multilingual meetings" refers to techniques and methods for instantly translating speech into other languages ​​to facilitate communication between participants who speak different languages.

[2285] The present invention relates to a system for conducting and managing a multifunctional and automated conference in which a server, terminals, and users cooperate with each other to realize efficient and neutral conferences.

[2286] 1. Setting the purpose and goals of the meeting

[2287] The server first provides a form for the meeting organizer to input the purpose and goals of the meeting. For example, the purpose of the meeting might be to "formulate a market strategy for a new product," and the goal might be to "determine a specific action plan for the next quarter." This data is saved and used to guide the meeting.

[2288] 2. Real-time recording of speech content and automatic generation of meeting minutes

[2289] When a meeting begins, the server recognizes speech in real time and transcribes it. Specific software used is Google Cloud Speech-to-Text, for example. The recognized speech is stored in a database and automatically generated as minutes after the meeting ends. Generative AI models such as OpenAI's GPT-4 are used to generate the minutes.

[2290] 3. Scoring to correct bias in statements

[2291] The server records the number of comments made by each user and performs scoring based on this. When a user who has made few comments is identified, a message is generated to prompt that user to make the next comment. For example, a prompt message such as "Would you like to give your next comment?" is generated and displayed on the user's terminal.

[2292] 4. Action Point Management

[2293] The server records the action points (APs) decided during the meeting and assigns each one a responsible person and deadline. For example, if an AP such as "detailed analysis of market research" is decided, the content, responsible person, and deadline are saved. The server periodically notifies the responsible person of this information and follows up on the progress.

[2294] 5. Multilingual simultaneous translation function

[2295] The server translates between different languages ​​in real time during a multilingual conference. For example, if a user says in Japanese, "Let's discuss the market strategy for the next quarter," the server translates it into English and displays the translated result, "Let's discuss the market strategy for the next quarter," on the user's device.

[2296] Specific examples

[2297] For example, if the purpose of a meeting is set as "Draft a market strategy for a new product" and someone says, "Let's think about what the target market for the new product should be," the server will use speech recognition software to transcribe this and store it in a database.Then, using a generative AI model, it will generate a prompt such as, "Please give us some specific ideas for determining the target market," and display it on the user's device.

[2298] As a result, the present invention provides a system that enables efficient and neutral meeting proceedings and automates the management of minutes and action points, allowing all participants to engage in constructive discussions.

[2299] The flow of the identification process in the first embodiment will be described with reference to FIG.

[2300] Step 1:

[2301] The server provides a form for the meeting organizer to input the purpose and goals of the meeting. In the form, users input the purpose, "to formulate a market strategy for a new product," and the goal, "to determine a specific action plan for the next quarter." The input data is sent to the server and stored in a database. This helps ensure that the progress of discussions during the meeting is in line with the purpose and goals.

[2302] Step 2:

[2303] When a user speaks during a meeting, the speech is sent as audio data via the microphone to the server. The server then uses speech recognition software (e.g., Google Cloud Speech-to-Text) to convert the audio data into text data. This text data is then stored in a database. This allows the content of speech during the meeting to be recorded in real time.

[2304] Step 3:

[2305] The server generates prompts to encourage the next appropriate utterance based on the utterances recorded in real time. The server inputs the recorded utterances and the purpose of the meeting into a generative AI model (e.g., OpenAI's GPT-4) to generate recommendations for the next utterance. The generated prompts (e.g., "Next, let's list specific regions that could be considered as target markets") are displayed on the user's device, thereby encouraging the discussion to proceed in a direction that is in line with the purpose.

[2306] Step 4:

[2307] The server measures the number of comments made by each user in real time and stores it in a database for counting the number of comments. The server scores the data on the number of comments and identifies users who have made few comments. When a user who has made few comments is identified, the server generates a message saying, "Would you like to give your opinion next?" and displays it on the user's device. This corrects any bias in comments.

[2308] Step 5:

[2309] The server records the action points (APs) proposed during the meeting. The server receives the APs proposed by the users (e.g., "Detailed analysis of market research") as input and stores them in a database along with the person in charge and the deadline. After the meeting ends, the server periodically sends notifications to the person in charge to follow up on the progress. This allows for efficient management and execution of action points.

[2310] Step 6:

[2311] The server translates between different languages ​​in real time during a multilingual conference. When a user speaks in Japanese, the server uses translation software to translate this voice data into another language (e.g., English). The translated text is then displayed on the user's device. For example, "Let's discuss the market strategy for the next quarter" is translated as "Let's discuss the market strategy for the next quarter." This facilitates communication between users who speak different languages.

[2312] Through these steps, the system makes meetings more efficient and neutral, providing an environment in which all participants can engage in constructive discussions.

[2313] (Application example 1)

[2314] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2315] Conventional conference systems have problems such as difficulty in managing the progress of discussions, correcting bias in speech, creating minutes in real time, supporting multiple languages, and properly managing action items. In particular, communication breakdowns due to bias in speech and inadequate multilingual support, and insufficient follow-up on decisions are prominent, making it difficult to achieve efficient and neutral conference proceedings.

[2316] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2317] In this invention, the server includes means for using a generative AI model that understands the purpose and goals of a meeting and appropriately manages the progress of the meeting, means for correcting bias in comments based on the number of times participants speak and scoring results, means for recording meeting content in real time and automatically generating minutes, means for managing action items and scheduling the next meeting, means for providing a simultaneous translation function for multilingual meetings, means for a generative AI model to suggest appropriate next comments based on comments made during the meeting, means for generating minutes based on the comments made and distributing them to participants, and means for recording action items decided in the meeting, setting responsible parties and deadlines, and following up on progress. This makes the progress of meetings more efficient and enables neutral and constructive discussions.

[2318] A "meeting" is a gathering of multiple participants to discuss and make decisions in order to achieve a specific purpose or goal.

[2319] A "generative AI model" is an artificial intelligence model that supports speech content and decision-making based on input data.

[2320] A "server" is a computer system that processes data and provides information.

[2321] "Number of speeches" refers to the number of times each participant spoke during the conference.

[2322] The "scoring result" is the result of evaluating the number of statements and their content based on specific criteria and converting them into a score.

[2323] "Bias" refers to a situation in which the fairness of a discussion is compromised when comments and opinions are concentrated among certain participants.

[2324] "Real-time" refers to information processing at the exact moment the meeting is taking place.

[2325] "Minutes" are a written record of what was said and the results of discussions during a meeting.

[2326] "Action items" are specific actions or tasks decided upon during the meeting.

[2327] "Scheduling" refers to adjusting the schedules of participants and deciding the date and time of the next meeting.

[2328] "Multilingual support" is a function that enables participants who speak multiple languages ​​to communicate smoothly.

[2329] The "simultaneous translation function" is a function that translates between different languages ​​in real time.

[2330] "Participants" refers to people who attend meetings and are involved in discussions and decision-making.

[2331] "Appropriate content" means useful and accurate content that is in line with the purpose and goals of the meeting.

[2332] "Progress" indicates the progress and achievement of action items.

[2333] "Follow-up" refers to monitoring the progress of an action item and responding or adjusting accordingly.

[2334] The present invention relates to a system for realizing efficiency and neutrality in a meeting. Specific embodiments for carrying out the present invention are described below.

[2335] 1. System Configuration

[2336] The server uses hardware and software equipped with a generative AI model, database, and multilingual translation function. The server operates in a cloud computing environment, including the OpenAI API. Client devices include smartphones, tablets, and PCs, and users use these to participate in the conference.

[2337] 2. Meeting progress management

[2338] The server receives initial data to set the purpose and goals of the meeting and uses a generative AI model to appropriately manage the progress of the meeting. For example, if the purpose is set as "formulating a market strategy for a new product," the server analyzes the user's comments in real time and suggests the next appropriate comment. This allows the meeting to proceed in a direction that is consistent with the purpose.

[2339] 3. Correcting bias in statements

[2340] The server scores the number of comments made by each user and prompts users who have made fewer comments to make the next comment. For example, by notifying the user who has made the fewest comments, "Would you like to give your opinion next?", it is possible to correct bias in comments and promote neutral discussions.

[2341] 4. Automatic generation of meeting minutes

[2342] The server records what is said during the meeting in real time and automatically generates minutes. These minutes are automatically generated after the meeting ends and distributed to the participants' client devices. For example, if a user says, "I will talk about the results of our market research," this will be automatically recorded in the minutes.

[2343] 5. Managing Action Items

[2344] The server records the action items decided in the meeting and assigns the person in charge and deadline. These are saved in a database and notifications are sent to the person in charge to follow up on the progress as appropriate. For example, if an action item such as "Detailed analysis of market research" is set, the person in charge and deadline are recorded based on that and reminders are sent as appropriate.

[2345] 6. Multilingual simultaneous translation

[2346] In multilingual conferences, the server translates between different languages ​​in real time. For example, it translates what is said in Japanese into English and displays it on the device of an English-speaking user. This allows smooth communication between users who speak different languages.

[2347] 7. Suggested speech using generative AI models

[2348] The server uses a generative AI model to suggest the next most appropriate utterance based on the user's utterance. For example, if the user says, "I'll talk about the results of our market research," the generative AI model will suggest, "Let's consider the next action while looking at specific data."

[2349] Prompt Sentence Examples

[2350] "Suggest appropriate discussion points to follow the statement 'I'll talk about the results of my market research.'"

[2351] As described above, the present invention provides a system that realizes efficient and neutral meetings, corrects bias in discussions, and facilitates multilingual meetings.

[2352] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2353] Step 1:

[2354] The server receives initial data for setting the purpose and goals of the meeting. As input, information about the purpose and goals of the meeting is given and recorded in the database. As output, the set purpose and goals are saved on the server.

[2355] Step 2:

[2356] The server uses a generative AI model to analyze what is said during the meeting in real time. As input, the voice data spoken by the user is sent to the server, which converts it into text data. The generative AI model generates the next appropriate utterance based on this text data. As output, the generated next utterance is displayed on the user's device.

[2357] Step 3:

[2358] The server records the number of times each user has spoken and performs scoring. Information about the content of the speech and the speaker is sent to the server as input. The server records this in a database and counts the number of times each user has spoken. The server saves the scoring results as output.

[2359] Step 4:

[2360] The server generates a message to encourage users who have not spoken often to speak next. The scoring results are used as input. The server identifies the user who has spoken the least and generates a message such as "Would you like to give your next opinion?". As output, this message is sent to the device of the user.

[2361] Step 5:

[2362] The server records the meeting contents in real time and automatically generates minutes. As input, users' speech text is continuously sent to the server and recorded in a database. The server creates minutes based on this. As output, minutes that are updated in real time are generated and distributed to participants after the meeting ends.

[2363] Step 6:

[2364] The server records the action items decided during the meeting and sets the person in charge and deadline. As input, the information of the action items decided during the discussion, the person in charge, and the deadline are sent to the server. The server records this in a database and sets a schedule for following up on the progress. As output, the set action items are saved and reminders are executed according to the schedule.

[2365] Step 7:

[2366] The server translates between different languages ​​in real time during multilingual conferences. Each user's comment text and the language information to be translated are sent to the server as input. The server translates this using a multilingual translation API, and the translation results are displayed on other users' devices as output.

[2367] Through the above steps, the present invention makes the progress of the conference more efficient and facilitates smooth communication between participants.

[2368] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2369] The present invention relates to a system that realizes efficient and neutral meetings, and further combines it with an emotion engine that recognizes the emotions of users. Specific embodiments for carrying out the present invention are described below.

[2370] 1. Discussion Management

[2371] server

[2372] 1. The server sets the purpose and goals of the meeting and stores them in its internal memory, such as "Draw up a market strategy for a new product" or "Decide on a concrete action plan for the next quarter."

[2373] 2. Receive the signal to start the meeting and initialize the discussion history.

[2374] User

[2375] 3. A user speaks during the conference.

[2376] Terminal

[2377] 4. The user's device inputs the message and sends it to the server.

[2378] server

[2379] 5. The server adds the user's comment to the discussion history and uses the generative model to generate the next most appropriate comment.

[2380] 6. The server sends the generated message to the user's device.

[2381] Terminal

[2382] 7. The user terminal displays the next recommended comment sent from the server.

[2383] 2. Measures to correct bias in speech

[2384] server

[2385] 1. The server scores each user's speech record. For example, suppose user 1 spoke three times, user 2 spoke once, and user 3 spoke twice.

[2386] 2. Identify the user with the least number of comments, in this example User 2.

[2387] 3. Generate a notification to encourage a specific user to speak up, such as "User 2, would you like to give your next comment?"

[2388] 4. The server sends this notification to the user terminal.

[2389] Terminal

[2390] 5. The user device displays this notification and prompts the user to make the next statement.

[2391] 3. Automatic generation of minutes

[2392] server

[2393] 1. The server records the discussion history in real time and saves the content of comments sequentially.

[2394] User

[2395] 2. Each user makes a statement.

[2396] server

[2397] 3. The server keeps a real-time log of all statements made during the meeting.

[2398] 4. After the meeting, generate minutes based on the logs.

[2399] 5. The generated minutes are sent to the user's terminal.

[2400] Terminal

[2401] 6. The user terminal displays the minutes, allowing the user to review and save them.

[2402] 4. Action point management methods

[2403] server

[2404] 1. The server initializes the Action Point (AP) list.

[2405] User

[2406] 2. A user proposes a new action point (AP) during a meeting.

[2407] Terminal

[2408] 3. The user device sends the contents of the new AP to the server, including specific details such as "Detailed analysis of market research," the person in charge, and the deadline.

[2409] server

[2410] 4. The server adds the new AP to the AP list and records the assignee and deadline.

[2411] 5. The server will notify the appropriate person of this information.

[2412] Terminal

[2413] 6. The user terminal displays the AP information sent from the server, and the information is confirmed by the person in charge.

[2414] 5. Simultaneous translation function

[2415] server

[2416] 1. The server initializes the multilingual translation API.

[2417] User

[2418] 2. Users speak in different languages ​​during a conference.

[2419] Terminal

[2420] 3. The user terminal inputs a message and sends it to the server.

[2421] server

[2422] 4. The server uses a translation API to translate the speech into a different language.

[2423] 5. The server sends the translation results to the user's device.

[2424] Terminal

[2425] 6. The user's device displays the translation results, supporting communication between different languages.

[2426] 6. Emotion recognition means

[2427] server

[2428] 1. The server initializes the emotion engine that performs facial expression recognition and voice analysis.

[2429] User

[2430] 2. When a user speaks during a meeting, facial expression and voice data are collected by a camera and microphone.

[2431] Terminal

[2432] 3. The user device sends facial expression and voice data to the server.

[2433] server

[2434] 4. The server uses an emotion engine to analyze the user's emotions (e.g., joy, anger, sadness, stress, etc.).

[2435] 5. Based on the emotion recognition results, appropriate responses are taken. For example, if a user is feeling stressed, a message or notification is generated to encourage relaxation.

[2436] 6. The server sends these notifications to the user device.

[2437] Terminal

[2438] 7. The user's device displays a notification and provides support that corresponds to the user's emotions.

[2439] As described above, the present invention realizes efficient, neutral, and user-interactive conference proceedings, and provides a comfortable conference environment through real-time minutes creation, AP management, and user emotion recognition.

[2440] The processing flow will be explained below.

[2441] 1. Discussion Management

[2442] server

[2443] Step 1:

[2444] The server sets the purpose and goals of the meeting and stores them in its internal memory, such as "Draw up a market strategy for a new product" or "Decide on a concrete action plan for the next quarter."

[2445] Step 2:

[2446] The server receives a signal to start a conference and initializes the discussion history.

[2447] User

[2448] Step 3:

[2449] A user speaks during a conference.

[2450] Terminal

[2451] Step 4:

[2452] The user terminal inputs the content of the message and sends it to the server.

[2453] server

[2454] Step 5:

[2455] The server adds the user's comments to the discussion history and uses a generative model to generate the next most appropriate comment.

[2456] Step 6:

[2457] The server sends the generated message to the user terminal.

[2458] Terminal

[2459] Step 7:

[2460] The user terminal displays the next recommended comment sent from the server.

[2461] 2. Measures to correct bias in speech

[2462] server

[2463] Step 1:

[2464] The server scores each user's speech record. For example, suppose user 1 spoke three times, user 2 spoke once, and user 3 spoke twice.

[2465] Step 2:

[2466] The server identifies the user with the least number of comments, in this example user 2.

[2467] Step 3:

[2468] The server generates a notification to prompt a specific user to speak next, such as "User 2, would you like to give your next comment?"

[2469] Step 4:

[2470] The server sends this notification to the user terminal.

[2471] Terminal

[2472] Step 5:

[2473] The user terminal displays this notification and prompts the user to make the next statement.

[2474] 3. Automatic generation of minutes

[2475] server

[2476] Step 1:

[2477] The server records the discussion history in real time and saves the content of comments sequentially.

[2478] User

[2479] Step 2:

[2480] Each user makes a statement.

[2481] server

[2482] Step 3:

[2483] The server keeps a real-time log of everything said during the meeting.

[2484] Step 4:

[2485] After the meeting ends, the server generates minutes based on the log.

[2486] Step 5:

[2487] The server transmits the generated minutes to the user terminal.

[2488] Terminal

[2489] Step 6:

[2490] The user terminal displays the minutes, allowing the user to review and save them.

[2491] 4. Action point management methods

[2492] server

[2493] Step 1:

[2494] The server initializes the Action Point (AP) list.

[2495] User

[2496] Step 2:

[2497] A user proposes a new action point (AP) during a meeting.

[2498] Terminal

[2499] Step 3:

[2500] The user device sends the contents of the new AP to the server, including specific details such as "Detailed analysis of market research," the person in charge, and the deadline.

[2501] server

[2502] Step 4:

[2503] The server adds the new AP to the AP list and records the assignee and deadline.

[2504] Step 5:

[2505] The server will notify the appropriate person of this information.

[2506] Terminal

[2507] Step 6:

[2508] The user terminal displays the AP information sent from the server, and the information is checked by the person in charge.

[2509] 5. Simultaneous translation function

[2510] server

[2511] Step 1:

[2512] The server initializes the multilingual translation API.

[2513] User

[2514] Step 2:

[2515] Users speak in different languages ​​during a conference.

[2516] Terminal

[2517] Step 3:

[2518] A user terminal inputs a message and sends it to the server.

[2519] server

[2520] Step 4:

[2521] The server uses a translation API to translate the speech into different languages.

[2522] Step 5:

[2523] The server sends the translation results to the user terminal.

[2524] Terminal

[2525] Step 6:

[2526] The user's device displays the translation results, supporting communication between different languages.

[2527] 6. Emotion recognition means

[2528] server

[2529] Step 1:

[2530] The server initializes the emotion engine that performs facial expression recognition and voice analysis.

[2531] User

[2532] Step 2:

[2533] When a user speaks during a conference, facial expression and voice data are collected by a camera and microphone.

[2534] Terminal

[2535] Step 3:

[2536] The user terminal transmits facial expression and voice data to the server.

[2537] server

[2538] Step 4:

[2539] The server uses an emotion engine to analyze the user's emotions (e.g., joy, anger, sadness, stress, etc.).

[2540] Step 5:

[2541] Based on the emotion recognition results, appropriate responses are taken. For example, if a user is feeling stressed, a message or notification is generated to encourage relaxation.

[2542] Step 6:

[2543] The server sends these notifications to the user terminal.

[2544] Terminal

[2545] Step 7:

[2546] The user's device displays a notification and provides support that corresponds to the user's emotions.

[2547] As described above, the present invention realizes efficient, neutral, and user-interactive conference proceedings, and provides a comfortable conference environment through real-time minutes creation, AP management, and user emotion recognition.

[2548] Example 2

[2549] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2550] Conventional conference systems lack sufficient discussion progress management, speech bias correction, real-time minutes creation, action point management, and multilingual support, making it difficult to ensure meeting efficiency and neutrality. Furthermore, they do not take participants' emotions into consideration, making it difficult to provide an optimal conference environment. The present invention aims to solve these problems.

[2551] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2552] In this invention, the server includes means for setting the purpose and goal of the meeting and saving it in an internal storage device, means for generating speech content using a generative AI model to manage the progress of the meeting, means for correcting bias in speech based on the number of times participants speak and scoring results, means for recording the content of the meeting in real time and automatically generating minutes after the meeting ends, means for managing action points proposed during the meeting and recording the person in charge and deadline, means for providing a simultaneous translation function in multilingual meetings, and means for recognizing user emotions and taking appropriate action based on the results, thereby enabling efficient, neutral, and user-interactive meeting progress.

[2553] "Means for setting the purpose and goal of the meeting and saving it in the internal memory" is a function for determining the specific purpose and final goal of the meeting before the meeting starts and saving it in the internal memory.

[2554] "Means for generating speech content using a generative AI model to manage the progress of a meeting" refers to a function that uses a generative AI model during a meeting to automatically generate appropriate speech content, helping to ensure the meeting progresses smoothly.

[2555] "Means to correct bias in speech based on the number of times participants speak and the scoring results" is a function that records and analyzes the number of times each participant speaks and their score, and corrects bias in speech by encouraging specific participants to speak.

[2556] "A means of recording meeting content in real time and automatically generating minutes after the meeting" is a function that records statements and discussions made during a meeting in real time and automatically creates minutes after the meeting based on that data.

[2557] "A means to manage action points proposed during a meeting and record the person in charge and deadline" is a function for recording new action points proposed during a meeting and setting and managing the person in charge and deadline.

[2558] "Means for providing simultaneous translation functionality in multilingual conferences" refers to a functionality for translating in real time the statements of conference participants who speak different languages, and simultaneously supporting communication between different languages.

[2559] "Means for recognizing the user's emotions and taking appropriate action based on the results" refers to a function that analyzes the user's facial expressions and voice during a meeting to recognize their emotions, and then takes appropriate action or notifies them based on the results.

[2560] The present invention is a system that integrates meeting efficiency, ensuring neutrality, and user emotion recognition. A specific embodiment of this system will be described below.

[2561] This system is composed of a combination of a server, a terminal, and a user interface. The server is mainly responsible for calculation processing and data storage, while the terminal functions as an interface with the user. Each function is explained in detail below.

[2562] 1. Discussion Management

[2563] The server stores the purpose and goal of the meeting in its internal storage device. For example, it can be set to "formulate a market strategy for a new product." This information is managed based on the meeting start signal that the server receives when the meeting begins. When a user speaks during the meeting, the content of that speech is sent to the server via the user's device. The server adds the received speech to the discussion history, and based on that history, uses a generative AI model to generate the next most appropriate speech. This generated speech is sent back to the user's device and displayed to the user. As a concrete example, if we consider a meeting about a market strategy for a new product, the following actions will occur:

[2564] User 1: "Look at last year's sales data. It shows us new market potential."

[2565] Server: "User 2, next we'd like to hear your thoughts on new target markets."

[2566] 2. Measures to correct bias in speech

[2567] The server records and scores the number of times each user has spoken. For example, if User 1 has spoken three times, User 2 has spoken once, and User 3 has spoken twice, the server generates a notification to encourage the user with the fewest number of speeches (User 2 in this example) to speak next. The notification might be something like, "User 2, would you like to give your opinion next?" These notifications are sent to and displayed on the user's device.

[2568] Example prompt for a generative AI model:

[2569] "Please generate a message to notify User 2 that he has not been posting much and encourage him to post more."

[2570] 3. Automatic generation of minutes

[2571] The server records all statements made during the meeting in real time and saves them sequentially. When the meeting ends, the server automatically generates minutes based on the saved data and sends them to the user's device. The user's device displays the minutes, allowing the user to view and save them.

[2572] 4. Action point management methods

[2573] The server manages new action points (APs) proposed during a conference. When a user proposes a new AP, the user terminal sends the content to the server. The server adds the AP to a list and records the person in charge and the deadline. If necessary, the server sends a notification to the person in charge. This notification is confirmed on the user terminal.

[2574] Example prompt for a generative AI model:

[2575] "Add new action points to your list and record who is responsible and when they are due."

[2576] 5. Simultaneous translation function

[2577] The server initializes a multilingual translation API and translates any comments made in different languages ​​during the meeting. The user's device inputs the comments and sends them to the server, which then uses the translation API to translate the comments into the different language. The translation results are sent to the user's device and displayed.

[2578] Example prompt for a generative AI model:

[2579] "Translate what you say into the specified language and display the results."

[2580] 6. Emotion recognition means

[2581] The server initializes an emotion engine that performs facial expression recognition and voice analysis. When the user speaks, facial expression and voice data is collected by a camera and microphone and sent to the server via the user's device. The server uses the emotion engine to analyze emotions such as joy, anger, sadness, and stress. Based on the analysis results, the server generates appropriate response messages and notifications and sends them to the user's device. The user's device displays these notifications and responds to the user's emotions.

[2582] Example prompt for a generative AI model:

[2583] "If the user is feeling stressed, generate a notification to encourage relaxation."

[2584] As described above, the present invention realizes efficient, neutral, and user-interactive conference proceedings, and provides a comfortable conference environment through real-time minutes creation, action point management, and emotion recognition.

[2585] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2586] Specific processing steps of the system program

[2587] ~Method of managing discussions~

[2588] Step 1:

[2589] The server sets the purpose and goals of the meeting and stores them in an internal storage device.

[2590] Input: Initial setup data, including the purpose and goals of the meeting.

[2591] Output: Meeting objectives and goals data stored in internal memory.

[2592] Specific operation: The server sets goals such as "Draw up a market strategy for a new product" or "Decide on a specific action plan for the next quarter" and stores them in its internal memory.

[2593] Step 2:

[2594] The server receives a signal to start the conference.

[2595] Input: Signal to start a conference.

[2596] Output: Initialized discussion history data.

[2597] Specific operation: The server receives a conference start signal and initializes the internal discussion history.

[2598] Step 3:

[2599] A user speaks during a conference.

[2600] Input: What the user says.

[2601] Output: None.

[2602] Specific action: The user expresses their opinion or information.

[2603] Step 4:

[2604] The user terminal inputs the content of the message and sends it to the server.

[2605] Input: What the user says (voice or text data).

[2606] Output: The speech data sent to the server.

[2607] Specific operation: The user's device converts the spoken co...

Claims

1. A means to use a generative model to understand the purpose and goals of a meeting and appropriately manage the progress of the meeting; A method to correct bias in speech based on the number of times participants speak and the scoring results, A means to record meeting content in real time and automatically generate minutes, A way to manage action points and schedule the next meeting, A means for providing simultaneous translation capabilities in multilingual conferences; A system including:

2. The system according to claim 1, further comprising means for setting a purpose and goal of a meeting and promoting a discussion based thereon, the means using a generative model.

3. 2. The system according to claim 1, further comprising means for scoring the number of times a participant speaks in order to encourage a participant who has spoken less to speak next.

Images