system

Abstract

Provide a system. 【Solution means】 An input means for inputting the purpose information of the meeting, A generation means for generating predicted topics based on the input purpose information, A presentation means for presenting the generated predicted topics, An analysis means for analyzing the voices of participants during the meeting and supporting the progress of the meeting, A voice recognition means for converting the voice data during the meeting into text data and monitoring the speech, An arrangement means for automatically arranging the determined matters after the meeting and generating a meeting minutes, A sharing means for sharing the generated meeting minutes with the participants, An information providing means for integrating smart environment information for improving the efficiency of meetings related to urban planning and social management and presenting relevant information, A system including the above.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, the method including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance as a response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] A problem is that meetings proceed inefficiently, participants lose sight of the direction of the discussion, and the meeting ends with unclear decisions and next actions. Therefore, it is necessary to clarify the purpose of the meeting and smooth its progress so that participants can make effective decisions and clarify the actions after the meeting.

Means for Solving the Problems

[0005] The present invention includes means for inputting purpose information of a meeting, means for generating and presenting predicted topics based on the input information, and means for analyzing the speech of participants during the meeting. Thereby, it is possible to support the progress of the meeting, automatically organize the decisions after the meeting, generate and share the minutes of the meeting, and improve the efficiency and productivity of the meeting.

[0006] "Purpose information" refers to information that indicates the ultimate goal or desired outcome of the meeting.

[0007] An "input means" is an interface or device for a user to provide target information to a system.

[0008] A "predicted agenda" is a list of topics and discussion items that should be addressed in the meeting, generated by AI based on the input objective information.

[0009] "Generation means" refers to an AI algorithm or software module that creates predictive topics based on target information.

[0010] "Presentation means" refers to a screen or device used to display the generated predicted topics and related information to the user.

[0011] "Analysis means" refers to a technology or system that processes and understands participants' voices and statements in real time during a meeting, thereby supporting the progress of the meeting.

[0012] A "organizational tool" is a process or function that automatically summarizes decisions and next actions after a meeting.

[0013] "Sharing means" refers to a communication method or system for providing meeting participants with generated minutes or decisions. [Brief explanation of the drawing]

[0014] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4]It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Embodiments for Carrying out the Invention

[0015] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, the signed processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Furthermore, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), and APU (Accelerated Processing Unit).

[0018] In the following embodiments, signed RAM (Random Access Memory) is a memory that temporarily stores information and is used as work memory by the processor.

[0019] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0020] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0022] [First Embodiment]

[0023] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0024] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0027] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0030] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0034] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0035] The system for implementing this invention aims to improve the efficiency and effectiveness of meetings, and achieves this objective by providing multiple functions. The system begins with the user entering meeting objective information into a terminal. This information specifically indicates the goals to be achieved at the meeting and is transmitted to the server.

[0036] The server activates an AI algorithm based on the received target information, analyzing past data and related information to generate predicted agenda items. These predicted agenda items are a list of important topics that should be discussed in the meeting. The generated agenda items are sent from the server to the terminal and presented to the user.

[0037] Users can review the presented agenda and edit it as needed. This clarifies the direction of the meeting and allows for smoother progress.

[0038] Once the meeting begins, the server analyzes the meeting's audio data through a speech recognition system and converts it into text data in real time. This analysis records participants' statements and simultaneously monitors the progress of the meeting. The server checks whether the discussion is on topic and provides participants with relevant information as needed.

[0039] After the meeting, the server automatically organizes the decisions and next actions based on the analyzed data. This data is compiled into meeting minutes, and the organized information is automatically shared with participants. These minutes include specific action plans, responsible parties, and deadlines, enabling more effective follow-up after the meeting.

[0040] For example, if a user enters the objective information as "determine the marketing strategy for a new product," the server will generate specific predictive agenda items such as "competitor analysis," "target market selection," and "advertising channel selection." If the discussion strays from the main topic during the meeting, the server will provide participants with a notification such as "Let's return to the current agenda." In this way, it is possible to consistently improve the quality and efficiency of meetings.

[0041] The following describes the processing flow.

[0042] Step 1:

[0043] The user uses their device to input meeting objectives. The user describes the specific goals of the meeting on the input screen and presses the submit button.

[0044] Step 2:

[0045] The terminal receives the target information entered by the user and sends it to the server. The data is encrypted and sent to the server via a secure communication channel.

[0046] Step 3:

[0047] The server receives the target information and uses an AI algorithm to generate predicted agenda items based on that information. It refers to past meeting data and related information to list appropriate items.

[0048] Step 4:

[0049] The server sends the generated predicted agenda items to the terminal. The generated agenda items are displayed in the user interface.

[0050] Step 5:

[0051] Users review the predicted agenda items presented on their device and edit them as needed. Once the user confirms the final agenda, it is sent to the server.

[0052] Step 6:

[0053] During the meeting, the server uses speech recognition technology to analyze participants' voices in real time. It converts their speech into text data and monitors the meeting's progress.

[0054] Step 7:

[0055] The server provides relevant data and support information to speakers as needed during the meeting. If the agenda deviates from its intended direction, it will propose corrections.

[0056] Step 8:

[0057] Once the meeting ends, the server automatically organizes the decisions and next actions based on the audio and text data.

[0058] Step 9:

[0059] The server generates meeting minutes based on the information it has compiled, and automatically distributes these minutes to all participants. Distribution is done via email or a sharing platform.

[0060] (Example 1)

[0061] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0062] Traditional meeting systems often rely on participants' skills and preparation to ensure effective use of time. Furthermore, discussions frequently veer off-topic, making progress management difficult. Additionally, decisions and action plans after meetings may not be reliably shared, leading to insufficient follow-up.

[0063] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0064] In this invention, the server includes a device for inputting meeting purpose information, a device for generating predicted agenda items based on the input purpose information, and a device for presenting the generated predicted agenda items. This makes it possible to improve the efficiency and results of meetings and make effective use of time. It also supports progress management and ensures that follow-up after meetings is carried out reliably.

[0065] A "device" is a set of hardware and software components designed to perform a specific function.

[0066] "Meeting objective information" refers to information that represents the specific goals and objectives that the meeting aims to achieve.

[0067] A "predicted agenda" is a list of important topics that should be discussed at the meeting, based on past data and relevant information.

[0068] "Presentation" refers to the act of visually displaying generated information or data to the user.

[0069] "Analysis" is the process of understanding meaning and obtaining information based on collected data.

[0070] "Audio information" refers to data that includes audio recordings made by participants during a meeting.

[0071] "Text information" refers to the information format obtained after converting audio information into text data.

[0072] A "record" is a document that details the proceedings and decisions made at a meeting.

[0073] "Sharing" refers to the act of distributing generated information or data among participants and making it accessible to them.

[0074] This system is an integrated platform designed to support more efficient and productive meetings. Users first input meeting objective information into a terminal, including specific meeting goals such as "determine the marketing strategy for the new product." The terminal then sends this objective information to a server. The server receives the information and activates an AI algorithm. This AI algorithm analyzes historical databases and relevant information sources to generate a predictive agenda of key topics to be discussed in the meeting. For example, a prompt such as "Please suggest the agenda necessary to achieve the meeting's objectives" can be entered into the AI ​​model.

[0075] The server uses Google® Cloud AI and various speech recognition services (e.g., Google Speech-to-Text) to carry out this process. The generated predicted topics are sent by the server to the terminal and presented to the user. The user can review the presented topics and edit them as needed.

[0076] During the meeting, the server converts participants' voice data into text data via a speech recognition system and monitors the progress according to the agenda. It supports efficient progress by providing reminders such as "Let's return to the current agenda" as needed.

[0077] After the meeting concludes, the server automatically organizes the analyzed data and generates a record containing decisions and action plans. This record is shared with all participants, enabling effective post-meeting follow-up.

[0078] In this way, by integrating multiple functions, this system can support the entire process of a meeting, achieving a higher level of effectiveness and efficiency.

[0079] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0080] Step 1:

[0081] The user enters meeting objectives into a terminal. This input is text information that linguistically expresses specific goals and topics. The terminal sends the entered information as digital data to the server. This step provides the basic data needed to initially set the focus of the meeting.

[0082] Step 2:

[0083] The server activates an AI algorithm based on the objective information received from the terminal. During this process, a prompt (e.g., "Please suggest the agenda items necessary to achieve the objectives of this meeting") is input to the generating AI model. The server analyzes past databases and related information to generate predicted agenda items. This data processing generates a list of important topics to be discussed in the meeting. This list is output as an agenda item tailored to the user's needs.

[0084] Step 3:

[0085] The server sends the generated predicted agenda to the terminal, which then presents the content to the user. The user can review the presented agenda and edit it as needed. The edited agenda is output as application information to help the user adjust the direction of the discussion.

[0086] Step 4:

[0087] Once the meeting begins, the server retrieves participants' voice data through a speech recognition system. This voice data is converted into text data through a service like Google Speech-to-Text. The server analyzes this text and monitors whether the meeting is progressing according to the agenda. This analysis checks the consistency between what is said and the agenda, and provides notifications such as "Let's return to the current agenda" if necessary.

[0088] Step 5:

[0089] After the meeting concludes, the server organizes the data collected during the meeting and generates a record containing decisions and action plans. This includes extracting key information based on real-time converted text data. The generated record is automatically shared with participants via their devices. This step enables efficient post-meeting follow-up.

[0090] (Application Example 1)

[0091] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0092] In fields such as urban development and public transportation, meetings involving diverse stakeholders require efficient information sharing and decision-making. However, discussions can be lengthy and inefficient, necessitating optimization of meeting procedures. Furthermore, systems are needed to quickly and accurately follow up on decisions.

[0093] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0094] This invention includes a server that provides information by aggregating smart environmental information to improve the efficiency of meetings related to urban planning and social management, and presenting relevant information; a voice recognition means that converts voice data into text data and monitors speech during meetings; and a means that cooperates with an urban management system to provide support information for the implementation of plans and activities decided at meetings. This improves the efficiency of meetings and enables rapid and accurate decision-making and support for the implementation of those decisions.

[0095] "Meeting objective information" refers to information that describes the specific goals and topics to be achieved in the meeting.

[0096] A "predicted agenda" is an agenda generated by an AI algorithm based on past data and objective information, indicating important topics for the meeting.

[0097] "Presentation means" refers to methods or devices for showing information to participants visually or audibly.

[0098] "Speech recognition means" refers to technology that converts participants' voices into text data and analyzes the progress of the meeting and the content of their statements.

[0099] "Organizational methods" refer to methods or processes for automatically organizing decisions and minutes after a meeting and compiling them into a report.

[0100] A "sharing method" refers to a system for distributing generated meeting minutes and related information to participants.

[0101] "Information provision methods" refer to methods of gathering smart environment information related to urban planning and social management, and presenting participants with the information necessary to support the efficiency of meetings.

[0102] A "city management system" is a management system that uses information technology to support the implementation of planning and activities in a city.

[0103] To implement this invention, it is necessary to build a system that aims to improve the efficiency and effectiveness of meetings. This system is centered around a server, terminals, and users, each playing a specific role.

[0104] First, the user inputs meeting objectives using their device. This defines the specific goals and topics to be addressed in the meeting. Based on this objective information, the server uses an AI algorithm to generate a predicted agenda. The predicted agenda is a list of important topics for the meeting, compiled by analyzing past data and related information.

[0105] The server sends the generated predicted agenda to the terminal and presents it to the user. The user can review this agenda and edit it as needed, thereby clarifying the direction of the meeting.

[0106] During the meeting, the server converts participants' voice data into text in real time via a speech recognition system (e.g., Google Cloud Speech-to-Text API). This process allows for recording participants' statements and monitoring the progress of the meeting. Furthermore, information provision tools are used to present relevant information as needed, providing timely information necessary for urban planning and social management.

[0107] After the meeting ends, the server automatically organizes the decisions made based on the data recorded during the meeting and generates meeting minutes. The generated minutes, which include specific action plans, responsible parties, and deadlines, are shared with participants via their devices. As a result, post-meeting follow-up can be conducted more effectively.

[0108] For example, in a meeting about improving urban transportation systems, instructions such as "Please proceed to the next proposal point" could be given. An example of a prompt for a generative AI model would be, "Please present a specific action plan that is expected as an outcome of this meeting."

[0109] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0110] Step 1:

[0111] The user enters meeting objective information using a terminal. This information specifically outlines the goals and agenda of the meeting and is sent to the server. The input here includes the user's desired outcomes and discussion topics.

[0112] Step 2:

[0113] Based on the received purpose information, the server activates the AI ​​algorithm. The prompt "Generate an agenda based on the purpose of this meeting" is input to the AI, which then analyzes past data and relevant information to generate a predicted agenda. This creates a list of important topics to be discussed at the meeting, which is then sent from the server to the terminal.

[0114] Step 3:

[0115] Users review the predicted agenda generated on their devices and edit it as needed. This editing process is intended to improve the content of the agenda, and the newly edited agenda becomes the final one. This editing includes adding or deleting items from the agenda.

[0116] Step 4:

[0117] After the meeting begins, the server uses a speech recognition system to convert the meeting's audio data into text data in real time. The audio input is the participants' statements, and the output is the corresponding text data. This conversion allows participants' statements to be recorded in text format, while simultaneously monitoring their alignment with the current agenda.

[0118] Step 5:

[0119] The server continues its analysis, checking whether the meeting is progressing according to the agenda. If necessary, it provides relevant information in real time to assist in the meeting's progress. At this time, it uses the prompt "Please provide relevant materials" to request relevant information.

[0120] Step 6:

[0121] At the end of the meeting, the server automatically organizes the decisions made based on the text data and generates meeting minutes. This process verifies the statements made against the agenda and clearly outlines the final action plan.

[0122] Step 7:

[0123] The generated meeting minutes are shared with participants via their devices. This allows for post-meeting follow-up and ensures that all participants understand the agreed-upon future action plan. Sending meeting minutes is typically done using electronic messaging systems.

[0124] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0125] This invention proposes a system incorporating an emotion engine to recognize and analyze participants' emotions in real time during a meeting. This system begins with the user inputting meeting objective information using a terminal, which is then transmitted to a server. Based on this objective information, the server generates predicted agenda items using an AI algorithm and presents them to the terminal.

[0126] During the meeting, the server uses speech recognition to convert participants' voices into text, while an emotion engine analyzes participants' emotions from the audio and video. This analysis is used as data to control the progress of the meeting. For example, if it determines that participants' interest is waning, the server will suggest changing the agenda or taking a break to ensure smooth progress.

[0127] Furthermore, the server organizes all decisions, including sentiment data, after the meeting and generates meeting minutes. The generated minutes are automatically shared with participants, and the sentiment data is used to prepare for the next meeting.

[0128] For example, if a user enters the objective information as "discuss the market strategy for a new product," the server will generate and present agenda items such as "analysis of competing products" and "identification of the target market" based on this information. If the emotion engine determines that participants' levels of interest have decreased during the meeting or presentation, the server will send a notification to the terminal saying, "Consider sharing relevant success stories at this point." This allows for efficient decision-making while maintaining the energy of the meeting.

[0129] The following describes the processing flow.

[0130] Step 1:

[0131] The user uses a terminal to input meeting purpose information. The user enters the specific meeting goals in the input field on the terminal and presses the submit button.

[0132] Step 2:

[0133] The terminal sends the target information entered by the user to the server. The data is sent to the server in an encrypted form for security purposes.

[0134] Step 3:

[0135] The server receives the target information and generates predicted agenda items using an AI algorithm. The server then refers to a historical database, extracts relevant topics, and lists them as agenda items.

[0136] Step 4:

[0137] The server sends the generated predicted agenda items to the terminal and presents them to the user.

[0138] Step 5:

[0139] The user reviews the predicted agenda presented on their device and edits it if necessary. Once the user confirms the final agenda, they send that information to the server.

[0140] Step 6:

[0141] Once the meeting begins, the server activates the speech recognition system and emotion engine, analyzing participants' audio and video data in real time.

[0142] Step 7:

[0143] The server references data from the emotion engine to analyze the emotional state of the participants. For example, if it detects signs that a participant is bored, the server will then present suggestions on the terminal to improve the meeting's flow.

[0144] Step 8:

[0145] Once the meeting concludes, the server organizes the decisions and next actions based on audio and sentiment data. It then generates meeting minutes summarizing the results.

[0146] Step 9:

[0147] The server automatically distributes meeting minutes to all participants. The minutes also include sentiment analysis results, which can be used to prepare for future meetings.

[0148] (Example 2)

[0149] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0150] In meetings, there are challenges in understanding participants' changing emotions and interests in real time and appropriately adjusting the meeting's progress. Furthermore, there is a problem in automatically generating detailed meeting minutes, including emotional information, after the meeting, making effective preparation for the next meeting difficult. In addition, accurately recording what is said during the meeting and managing it in a way that allows for easy later reference is also challenging.

[0151] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0152] This invention includes a server configuration that acquires participants' voices as audio data during a meeting and analyzes their emotions using natural language processing technology; a configuration that generates instructions to support the progress of the meeting and notifies participants' devices; and a configuration that automatically organizes and documents all decisions, including emotion data, after the meeting. This makes it possible to grasp changes in participants' emotions and interests in real time and adjust the meeting progress appropriately based on that data. In addition, the automatically generated detailed minutes enable effective preparation for the next meeting, and statements made during the meeting can be accurately recorded and managed.

[0153] A "device" refers to a combination of hardware and software used for information processing, particularly those that enable input and output manipulation.

[0154] "Configuration" refers to the arrangement of elements or processes combined to achieve a specific function, or the effect achieved by that arrangement.

[0155] "Audio data" refers to data that represents speech in digital format and contains fundamental information for speech recognition and analysis.

[0156] "Natural language processing technology" refers to artificial intelligence technology that enables computers to understand, appropriately analyze, and generate human language.

[0157] "Analyzing emotions" refers to the process of identifying emotional states from input data and quantifying or categorizing those states.

[0158] "Instructions" are pieces of information or recommendations provided to encourage specific behaviors, and they involve suggestions or requests for action.

[0159] "Documenting" refers to the process of systematically recording information in document form, making it easy to refer to and analyze later.

[0160] "Real-time" means that processing and responses occur simultaneously with the occurrence of an event, and refers to a state in which information is transmitted or processed without delay.

[0161] This invention provides a system that analyzes the emotions and statements of participants during a meeting in real time to support the progress of the meeting, and at the same time automatically generates detailed meeting minutes that include emotions after the meeting. This system consists of three main elements: users, terminals, and servers.

[0162] The user uses a terminal to input information about the purpose of the meeting. For example, they might input information such as "Discuss the market strategy for the new product." This terminal is used to receive user input and send the entered information to the server.

[0163] The server uses a generative AI model to generate relevant topics based on the received target information. This AI model analyzes text data using natural language processing techniques and generates predictive topics based on user input. Specific software used here includes machine learning frameworks such as TENSORFLOW® and PyTorch. As a result, topics such as "Analysis of Competitor Products" and "Identification of Target Markets" are generated and presented to the terminal.

[0164] During the meeting, the server uses speech recognition technology (e.g., Google Cloud Speech-to-Text) to convert participants' voices into text data, while simultaneously performing sentiment analysis that includes video data. An AI-based sentiment engine is used for the analysis. This allows for real-time monitoring of participants' emotional states and generates instructions appropriate for the meeting's progress. For example, if a participant's interest wanes, the server sends a notification to their device saying, "Consider sharing relevant success stories at this point."

[0165] After the meeting, the server uses AI technology to automatically generate detailed meeting minutes based on sentiment data and decisions made during the meeting. These minutes are shared with participants via email or cloud services, allowing them to accurately understand the meeting content and contribute to preparing for future meetings.

[0166] An example of a prompt to a generative AI model would be: "Meeting purpose information has been entered. Please list the predicted topics. For example, if the purpose is to 'discuss the market strategy for a new product,' what topics would you suggest?" Through this prompt, the server can generate the most appropriate topics.

[0167] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0168] Step 1:

[0169] The user uses a terminal to input meeting objective information. The entered objective information is sent from the terminal to the server. Specifically, the user uses the input field on the terminal and the keyboard to type "Discuss the market strategy for the new product" and presses the send button. The input is text data, and the output is data sent to the server.

[0170] Step 2:

[0171] The server sends prompts to the generating AI model based on the received objective information, and generates relevant agenda items. Here, natural language processing technology is used to analyze text data and generate predicted agenda items. Specifically, in response to the prompt "Discuss market strategy for new product," the AI ​​model on the server generates agenda items such as "Analyze competing products" and "Identify target market." The input is the text data of the objective information, and the output is a list of generated agenda items.

[0172] Step 3:

[0173] The server sends the generated agenda items to the terminal, and the terminal displays the list on its screen. Specifically, the server transfers the generated agenda items as data packets to the terminal, and the agenda list is displayed on the terminal's display. The input is agenda data from the server, and the output is the display of agenda items on the user interface.

[0174] Step 4:

[0175] During the meeting, the server uses speech recognition technology to collect participants' voices as audio data and converts it to text. It also uses an emotion engine to analyze participants' emotions from the audio and video data. Specifically, the server processes data acquired from microphones and cameras installed in the meeting room in real time. The input is the audio and video data from the meeting, and the output is the transcribed speech and the results of the emotion analysis.

[0176] Step 5:

[0177] The server generates instructions appropriate for the meeting's progress based on the sentiment analysis results and notifies the terminals. Specifically, if the server receives an analysis result indicating "participant interest is declining," it creates an instruction such as "consider sharing relevant success stories at this point" and notifies the participants' terminals. The input is the sentiment analysis result, and the output is the notification of the instruction content.

[0178] Step 6:

[0179] After the meeting, the server organizes the meeting data and uses generative AI technology to create detailed meeting minutes. This includes sentiment data and decisions made. Specifically, all aggregated data is stored on the server, and the AI ​​model automatically creates a list and documented meeting minutes. The input is all the data from the meeting, and the output is a complete meeting minute.

[0180] Step 7:

[0181] The server shares the generated meeting minutes with the participants. Specifically, it uses email sending functions or cloud storage to distribute the minutes to participants' email addresses or designated folders. The input is the meeting minutes data, and the output is distribution to each participant.

[0182] (Application Example 2)

[0183] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0184] In modern community-participatory meetings, there is a need for methods to efficiently gather diverse opinions and ensure smooth proceedings while understanding participants' emotional states in real time. Furthermore, there is a demand for providing high-quality feedback after the meeting that takes into account participants' level of interest and emotions. Currently, there is a problem in that emotional awareness during meetings is insufficient, making it difficult to adjust the proceedings and provide appropriate feedback.

[0185] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0186] This invention includes a server that analyzes participants' voice and video during a meeting, recognizes and analyzes emotional data in real time to support the progress of the meeting, generates and presents relevant suggestions in real time, and automatically organizes decisions and emotional data after the meeting and generates meeting minutes. This enables accurate understanding of participants' emotions and facilitates efficient and smooth meeting management and feedback provision.

[0187] "Purpose information" refers to information about the specific goals and topics that participants aim to achieve in a meeting or gathering.

[0188] "Predicted agenda items" are themes and topics that should be discussed during the meeting, generated by an AI algorithm based on the input objective information.

[0189] "Presentation means" refers to a system or device for visually or audibly communicating generated predicted agenda items and related proposals to participants.

[0190] The "analysis means" is a component that collects participants' voice and video data in real time and performs emotion recognition and analysis of the content of their statements.

[0191] "Emotional data" refers to information representing the emotional state extracted from participants' voices and facial expressions.

[0192] "Real-time recognition and analysis" refers to a process where data collection and processing occur almost simultaneously, allowing for immediate results.

[0193] A "proposal generation tool" is a system that creates and presents specific proposals to revitalize or adjust a meeting, depending on the emotional state of the participants and the progress of the discussion.

[0194] "Organizational tools" refer to the function of organizing data collected after a meeting and creating meeting minutes in a format that is easy for participants to understand.

[0195] Meeting minutes are documents that summarize what was said, decided, and how people felt during a meeting, and are shared with the participants.

[0196] To realize this invention, the server is first connected to a system equipped with a speech recognition API and an emotion recognition engine. This allows it to acquire and analyze in real time the audio and video data transmitted from terminals during a meeting. Specifically, it converts the audio data into text using Google Cloud Speech-to-Text API and Microsoft® Azure® Cognitive Services. It also extracts emotion data from participants' facial expressions using Amazon Rekognition and other facial recognition APIs.

[0197] The terminal functions as an interface for participants joining a meeting, providing a means for inputting purpose information. This allows participants to send the meeting topic and desired outcomes to the server in advance. Based on this information, the server uses an AI algorithm to generate predicted agenda items and presents them to participants through the terminal, thereby streamlining the meeting process.

[0198] As the meeting progresses, the server uses collected sentiment data to generate relevant suggestions in real time and notify participants' terminals if their interest wanes. This helps maintain the meeting's energy and effectively supports decision-making.

[0199] After the meeting concludes, the server organizes all the comments and sentiment data, and automatically generates meeting minutes. These minutes are shared with participants via their devices, allowing them to easily review past meeting content. Sentiment data is also used to prepare for future meetings.

[0200] For example, when a user holds a meeting to "consider proposals for renovating a local park," the system can suggest predictive agenda items such as "improving methods for gathering citizen opinions" and "the possibility of installing new playground equipment." Furthermore, if participants' interest wanes, the server can send a notification suggesting they "consider sharing successful examples from neighboring areas."

[0201] An example of a prompt might be: "In a community-participatory meeting, we are discussing park renovations. Participants are losing interest. Please suggest ways to rekindle their interest."

[0202] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0203] Step 1:

[0204] Users input meeting objectives using a terminal. This input includes specific themes such as "Considering proposals for the renovation of local parks." The terminal sends this information to the server, which receives it. This information serves as the basis for generating predicted agenda items to be discussed at the meeting.

[0205] Step 2:

[0206] Based on the received objective information, the server uses an AI algorithm to generate predicted agenda items. This process involves referencing past relevant meeting data and templates to create agenda items that are helpful in addressing the issues. The generated predicted agenda items include topics such as "improving methods for collecting citizen opinions" and "the possibility of installing new playground equipment." The server then sends these to the terminal.

[0207] Step 3:

[0208] During the meeting, the terminal continuously transmits participants' audio and video to the server. The server uses a speech recognition API to convert the audio data into text and a facial recognition API to extract emotional data from the video. Specifically, it utilizes the Google Cloud Speech-to-Text API and Amazon Rekognition to transcribe spoken content and analyze emotional states. This allows for an understanding of the participants' levels of interest and engagement.

[0209] Step 4:

[0210] The server evaluates the progress of the meeting based on the analyzed sentiment data. If it determines that participants' interest is waning, it uses a generative AI model to create relevant suggestions. For example, it might generate a message such as, "Consider sharing success stories from your local area." This suggestion is then notified to the user via their device.

[0211] Step 5:

[0212] After the meeting, the server organizes all spoken text and sentiment data and automatically generates meeting minutes. These minutes are shared on participants' devices so they can view them at any time. Specifically, a text editing algorithm is used to organize the spoken content into a summarized format, and it includes an evaluation of the importance of each agenda item based on changes in sentiment.

[0213] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0214] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0215] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0216] [Second Embodiment]

[0217] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0218] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0219] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0220] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0221] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0222] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0223] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0224] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0225] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0226] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0227] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0228] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0229] The system for implementing this invention aims to improve the efficiency and effectiveness of meetings, and achieves this objective by providing multiple functions. The system begins with the user entering meeting objective information into a terminal. This information specifically indicates the goals to be achieved at the meeting and is transmitted to the server.

[0230] The server activates an AI algorithm based on the received target information, analyzing past data and related information to generate predicted agenda items. These predicted agenda items are a list of important topics that should be discussed in the meeting. The generated agenda items are sent from the server to the terminal and presented to the user.

[0231] Users can review the presented agenda and edit it as needed. This clarifies the direction of the meeting and allows for smoother progress.

[0232] Once the meeting begins, the server analyzes the meeting's audio data through a speech recognition system and converts it into text data in real time. This analysis records participants' statements and simultaneously monitors the progress of the meeting. The server checks whether the discussion is on topic and provides participants with relevant information as needed.

[0233] After the meeting, the server automatically organizes the decisions and next actions based on the analyzed data. This data is compiled into meeting minutes, and the organized information is automatically shared with participants. These minutes include specific action plans, responsible parties, and deadlines, enabling more effective follow-up after the meeting.

[0234] For example, if a user enters the objective information as "determine the marketing strategy for a new product," the server will generate specific predictive agenda items such as "competitor analysis," "target market selection," and "advertising channel selection." If the discussion strays from the main topic during the meeting, the server will provide participants with a notification such as "Let's return to the current agenda." In this way, it is possible to consistently improve the quality and efficiency of meetings.

[0235] The following describes the processing flow.

[0236] Step 1:

[0237] The user uses their device to input meeting objectives. The user describes the specific goals of the meeting on the input screen and presses the submit button.

[0238] Step 2:

[0239] The terminal receives the target information entered by the user and sends it to the server. The data is encrypted and sent to the server via a secure communication channel.

[0240] Step 3:

[0241] The server receives the target information and uses an AI algorithm to generate predicted agenda items based on that information. It refers to past meeting data and related information to list appropriate items.

[0242] Step 4:

[0243] The server sends the generated predicted agenda items to the terminal. The generated agenda items are displayed in the user interface.

[0244] Step 5:

[0245] Users review the predicted agenda items presented on their device and edit them as needed. Once the user confirms the final agenda, it is sent to the server.

[0246] Step 6:

[0247] During the meeting, the server uses speech recognition technology to analyze participants' voices in real time. It converts their speech into text data and monitors the meeting's progress.

[0248] Step 7:

[0249] The server provides relevant data and support information to speakers as needed during the meeting. If the agenda deviates from its intended direction, it will propose corrections.

[0250] Step 8:

[0251] Once the meeting ends, the server automatically organizes the decisions and next actions based on the audio and text data.

[0252] Step 9:

[0253] The server generates meeting minutes based on the information it has compiled, and automatically distributes these minutes to all participants. Distribution is done via email or a sharing platform.

[0254] (Example 1)

[0255] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0256] Traditional meeting systems often rely on participants' skills and preparation to ensure effective use of time. Furthermore, discussions frequently veer off-topic, making progress management difficult. Additionally, decisions and action plans after meetings may not be reliably shared, leading to insufficient follow-up.

[0257] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0258] In this invention, the server includes a device for inputting meeting purpose information, a device for generating predicted agenda items based on the input purpose information, and a device for presenting the generated predicted agenda items. This makes it possible to improve the efficiency and results of meetings and make effective use of time. It also supports progress management and ensures that follow-up after meetings is carried out reliably.

[0259] A "device" is a set of hardware and software components designed to perform a specific function.

[0260] "Meeting objective information" refers to information that represents the specific goals and objectives that the meeting aims to achieve.

[0261] A "predicted agenda" is a list of important topics that should be discussed at the meeting, based on past data and relevant information.

[0262] "Presentation" refers to the act of visually displaying generated information or data to the user.

[0263] "Analysis" is the process of understanding meaning and obtaining information based on collected data.

[0264] "Audio information" refers to data that includes audio recordings made by participants during a meeting.

[0265] "Text information" refers to the information format obtained after converting audio information into text data.

[0266] A "record" is a document that details the proceedings and decisions made at a meeting.

[0267] "Sharing" refers to the act of distributing generated information or data among participants and making it accessible to them.

[0268] This system is an integrated platform designed to support more efficient and productive meetings. Users first input meeting objective information into a terminal, including specific meeting goals such as "determine the marketing strategy for the new product." The terminal then sends this objective information to a server. The server receives the information and activates an AI algorithm. This AI algorithm analyzes historical databases and relevant information sources to generate a predictive agenda of key topics to be discussed in the meeting. For example, a prompt such as "Please suggest the agenda necessary to achieve the meeting's objectives" can be entered into the AI ​​model.

[0269] The server uses Google Cloud AI and various speech recognition services (e.g., Google Speech-to-Text) to carry out this process. The generated predicted topics are sent by the server to the terminal and presented to the user. The user can review the presented topics and edit them as needed.

[0270] During the meeting, the server converts participants' voice data into text data via a speech recognition system and monitors the progress according to the agenda. It supports efficient progress by providing reminders such as "Let's return to the current agenda" as needed.

[0271] After the meeting concludes, the server automatically organizes the analyzed data and generates a record containing decisions and action plans. This record is shared with all participants, enabling effective post-meeting follow-up.

[0272] In this way, by integrating multiple functions, this system can support the entire process of a meeting, achieving a higher level of effectiveness and efficiency.

[0273] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0274] Step 1:

[0275] The user enters meeting objectives into a terminal. This input is text information that linguistically expresses specific goals and topics. The terminal sends the entered information as digital data to the server. This step provides the basic data needed to initially set the focus of the meeting.

[0276] Step 2:

[0277] The server activates an AI algorithm based on the objective information received from the terminal. During this process, a prompt (e.g., "Please suggest the agenda items necessary to achieve the objectives of this meeting") is input to the generating AI model. The server analyzes past databases and related information to generate predicted agenda items. This data processing generates a list of important topics to be discussed in the meeting. This list is output as an agenda item tailored to the user's needs.

[0278] Step 3:

[0279] The server sends the generated predicted topics to the terminal, and the terminal presents the content to the user. The user can check the presented topics and make edits if necessary. The edited topics are output as application information for the user to adjust the direction of the discussion.

[0280] Step 4:

[0281] When the meeting starts, the server obtains the voice data of the participants through the voice recognition system. This voice data is converted into text data through services such as Google Speech-to-Text. The server analyzes this and monitors whether the progress is along the topic. Through this analysis, a consistency check between the speech and the topic is performed, and if necessary, a notification such as "Let's return to the current topic" is provided.

[0282] Step 5:

[0283] After the meeting ends, the server organizes the data collected during the meeting and generates a record including the decisions and action plans. This includes the extraction of important information based on the real-time converted text data. The generated record is automatically shared with the participants through the terminal. This step enables efficient follow-up after the meeting.

[0284] (Application Example 1)

[0285] Next, Application Example 1 will be described. In the following description, the data processing device 12 is referred to as the "server", and the smart glasses 214 are referred to as the "terminal".

[0286] In meetings where various stakeholders participate in fields such as urban development and public transportation, efficient information sharing and decision-making are required. However, since discussions may lack efficiency over a long period of time, it is necessary to optimize the progress of the meeting. Also, a system for quickly and accurately following up on the decisions is required.

[0287] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0288] This invention includes a server that provides information by aggregating smart environmental information to improve the efficiency of meetings related to urban planning and social management, and presenting relevant information; a voice recognition means that converts voice data into text data and monitors speech during meetings; and a means that cooperates with an urban management system to provide support information for the implementation of plans and activities decided at meetings. This improves the efficiency of meetings and enables rapid and accurate decision-making and support for the implementation of those decisions.

[0289] "Meeting objective information" refers to information that describes the specific goals and topics to be achieved in the meeting.

[0290] A "predicted agenda" is an agenda generated by an AI algorithm based on past data and objective information, indicating important topics for the meeting.

[0291] "Presentation means" refers to methods or devices for showing information to participants visually or audibly.

[0292] "Speech recognition means" refers to technology that converts participants' voices into text data and analyzes the progress of the meeting and the content of their statements.

[0293] "Organizational methods" refer to methods or processes for automatically organizing decisions and minutes after a meeting and compiling them into a report.

[0294] A "sharing method" refers to a system for distributing generated meeting minutes and related information to participants.

[0295] "Information provision methods" refer to methods of gathering smart environment information related to urban planning and social management, and presenting participants with the information necessary to support the efficiency of meetings.

[0296] A "city management system" is a management system that uses information technology to support the implementation of planning and activities in a city.

[0297] To implement this invention, it is necessary to build a system that aims to improve the efficiency and effectiveness of meetings. This system is centered around a server, terminals, and users, each playing a specific role.

[0298] First, the user inputs meeting objectives using their device. This defines the specific goals and topics to be addressed in the meeting. Based on this objective information, the server uses an AI algorithm to generate a predicted agenda. The predicted agenda is a list of important topics for the meeting, compiled by analyzing past data and related information.

[0299] The server sends the generated predicted agenda to the terminal and presents it to the user. The user can review this agenda and edit it as needed, thereby clarifying the direction of the meeting.

[0300] During the meeting, the server converts participants' voice data into text in real time via a speech recognition system (e.g., Google Cloud Speech-to-Text API). This process allows for recording participants' statements and monitoring the progress of the meeting. Furthermore, information provision tools are used to present relevant information as needed, providing timely information necessary for urban planning and social management.

[0301] After the meeting ends, the server automatically organizes the decisions made based on the data recorded during the meeting and generates meeting minutes. The generated minutes, which include specific action plans, responsible parties, and deadlines, are shared with participants via their devices. As a result, post-meeting follow-up can be conducted more effectively.

[0302] As a specific example, in a meeting regarding the improvement of the urban transportation system, instructions such as "Please proceed to the next proposed point" are possible. As an example of a prompt sentence for the generative AI model, a format like "Please present a specific action plan expected as the result of this meeting" can be considered.

[0303] The flow of the specific process in Application Example 1 will be described using FIG. 12.

[0304] Step 1:

[0305] The user inputs the purpose information of the meeting using the terminal. The input purpose information specifically indicates the goals and topics in the meeting, and this is sent to the server. The input here is the outcome and discussion topics that the user desires.

[0306] Step 2:

[0307] Based on the received purpose information, the server activates the AI algorithm. Input "Please generate topics based on the purpose of this meeting" as the prompt sentence to the AI, and analyze past data and related information to generate predicted topics. As a result, the important topics to be discussed in the meeting are listed, and this list is sent from the server to the terminal.

[0308] Step 3:

[0309] The user checks the predicted topics generated on the terminal and edits them as necessary. This editing work is to make the content of the topics more appropriate, and the newly edited topics become the final topics. This editing includes adding or deleting items of the topics.

[0310] Step 4:

[0311] After the meeting begins, the server uses a speech recognition system to convert the meeting's audio data into text data in real time. The audio input is the participants' statements, and the output is the corresponding text data. This conversion allows participants' statements to be recorded in text format, while simultaneously monitoring their alignment with the current agenda.

[0312] Step 5:

[0313] The server continues its analysis, checking whether the meeting is progressing according to the agenda. If necessary, it provides relevant information in real time to assist in the meeting's progress. At this time, it uses the prompt "Please provide relevant materials" to request relevant information.

[0314] Step 6:

[0315] At the end of the meeting, the server automatically organizes the decisions made based on the text data and generates meeting minutes. This process verifies the statements made against the agenda and clearly outlines the final action plan.

[0316] Step 7:

[0317] The generated meeting minutes are shared with participants via their devices. This allows for post-meeting follow-up and ensures that all participants understand the agreed-upon future action plan. Sending meeting minutes is typically done using electronic messaging systems.

[0318] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0319] This invention proposes a system incorporating an emotion engine to recognize and analyze participants' emotions in real time during a meeting. This system begins with the user inputting meeting objective information using a terminal, which is then transmitted to a server. Based on this objective information, the server generates predicted agenda items using an AI algorithm and presents them to the terminal.

[0320] During the meeting, the server uses speech recognition to convert participants' voices into text, while an emotion engine analyzes participants' emotions from the audio and video. This analysis is used as data to control the progress of the meeting. For example, if it determines that participants' interest is waning, the server will suggest changing the agenda or taking a break to ensure smooth progress.

[0321] Furthermore, the server organizes all decisions, including sentiment data, after the meeting and generates meeting minutes. The generated minutes are automatically shared with participants, and the sentiment data is used to prepare for the next meeting.

[0322] For example, if a user enters the objective information as "discuss the market strategy for a new product," the server will generate and present agenda items such as "analysis of competing products" and "identification of the target market" based on this information. If the emotion engine determines that participants' levels of interest have decreased during the meeting or presentation, the server will send a notification to the terminal saying, "Consider sharing relevant success stories at this point." This allows for efficient decision-making while maintaining the energy of the meeting.

[0323] The following describes the processing flow.

[0324] Step 1:

[0325] The user uses a terminal to input meeting purpose information. The user enters the specific meeting goals in the input field on the terminal and presses the submit button.

[0326] Step 2:

[0327] The terminal sends the target information entered by the user to the server. The data is sent to the server in an encrypted form for security purposes.

[0328] Step 3:

[0329] The server receives the target information and generates predicted agenda items using an AI algorithm. The server then refers to a historical database, extracts relevant topics, and lists them as agenda items.

[0330] Step 4:

[0331] The server sends the generated predicted agenda items to the terminal and presents them to the user.

[0332] Step 5:

[0333] The user reviews the predicted agenda presented on their device and edits it if necessary. Once the user confirms the final agenda, they send that information to the server.

[0334] Step 6:

[0335] Once the meeting begins, the server activates the speech recognition system and emotion engine, analyzing participants' audio and video data in real time.

[0336] Step 7:

[0337] The server references data from the emotion engine to analyze the emotional state of the participants. For example, if it detects signs that a participant is bored, the server will then present suggestions on the terminal to improve the meeting's flow.

[0338] Step 8:

[0339] Once the meeting concludes, the server organizes the decisions and next actions based on audio and sentiment data. It then generates meeting minutes summarizing the results.

[0340] Step 9:

[0341] The server automatically distributes meeting minutes to all participants. The minutes also include sentiment analysis results, which can be used to prepare for future meetings.

[0342] (Example 2)

[0343] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0344] In meetings, there are challenges in understanding participants' changing emotions and interests in real time and appropriately adjusting the meeting's progress. Furthermore, there is a problem in automatically generating detailed meeting minutes, including emotional information, after the meeting, making effective preparation for the next meeting difficult. In addition, accurately recording what is said during the meeting and managing it in a way that allows for easy later reference is also challenging.

[0345] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0346] This invention includes a server configuration that acquires participants' voices as audio data during a meeting and analyzes their emotions using natural language processing technology; a configuration that generates instructions to support the progress of the meeting and notifies participants' devices; and a configuration that automatically organizes and documents all decisions, including emotion data, after the meeting. This makes it possible to grasp changes in participants' emotions and interests in real time and adjust the meeting progress appropriately based on that data. In addition, the automatically generated detailed minutes enable effective preparation for the next meeting, and statements made during the meeting can be accurately recorded and managed.

[0347] A "device" refers to a combination of hardware and software used for information processing, particularly those that enable input and output manipulation.

[0348] "Configuration" refers to the arrangement of elements or processes combined to achieve a specific function, or the effect achieved by that arrangement.

[0349] "Audio data" refers to data that represents speech in digital format and contains fundamental information for speech recognition and analysis.

[0350] "Natural language processing technology" refers to artificial intelligence technology that enables computers to understand, appropriately analyze, and generate human language.

[0351] "Analyzing emotions" refers to the process of identifying emotional states from input data and quantifying or categorizing those states.

[0352] "Instructions" are pieces of information or recommendations provided to encourage specific behaviors, and they involve suggestions or requests for action.

[0353] "Documenting" refers to the process of systematically recording information in document form, making it easy to refer to and analyze later.

[0354] "Real-time" means that processing and responses occur simultaneously with the occurrence of an event, and refers to a state in which information is transmitted or processed without delay.

[0355] This invention provides a system that analyzes the emotions and statements of participants during a meeting in real time to support the progress of the meeting, and at the same time automatically generates detailed meeting minutes that include emotions after the meeting. This system consists of three main elements: users, terminals, and servers.

[0356] The user uses a terminal to input information about the purpose of the meeting. For example, they might input information such as "Discuss the market strategy for the new product." This terminal is used to receive user input and send the entered information to the server.

[0357] The server uses a generative AI model to generate relevant agenda items based on the received target information. This AI model analyzes text data using natural language processing techniques and generates predictive agenda items based on user input. Specific software used here includes machine learning frameworks such as TensorFlow and PyTorch. As a result, agenda items such as "Analyze Competitor Products" and "Identify Target Markets" are generated and presented to the terminal.

[0358] During the meeting, the server uses speech recognition technology (e.g., Google Cloud Speech-to-Text) to convert participants' voices into text data, while simultaneously performing sentiment analysis that includes video data. An AI-based sentiment engine is used for the analysis. This allows for real-time monitoring of participants' emotional states and generates instructions appropriate for the meeting's progress. For example, if a participant's interest wanes, the server sends a notification to their device saying, "Consider sharing relevant success stories at this point."

[0359] After the meeting, the server uses AI technology to automatically generate detailed meeting minutes based on sentiment data and decisions made during the meeting. These minutes are shared with participants via email or cloud services, allowing them to accurately understand the meeting content and contribute to preparing for future meetings.

[0360] An example of a prompt to a generative AI model would be: "Meeting purpose information has been entered. Please list the predicted topics. For example, if the purpose is to 'discuss the market strategy for a new product,' what topics would you suggest?" Through this prompt, the server can generate the most appropriate topics.

[0361] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0362] Step 1:

[0363] The user uses a terminal to input meeting objective information. The entered objective information is sent from the terminal to the server. Specifically, the user uses the input field on the terminal and the keyboard to type "Discuss the market strategy for the new product" and presses the send button. The input is text data, and the output is data sent to the server.

[0364] Step 2:

[0365] The server sends prompts to the generating AI model based on the received objective information, and generates relevant agenda items. Here, natural language processing technology is used to analyze text data and generate predicted agenda items. Specifically, in response to the prompt "Discuss market strategy for new product," the AI ​​model on the server generates agenda items such as "Analyze competing products" and "Identify target market." The input is the text data of the objective information, and the output is a list of generated agenda items.

[0366] Step 3:

[0367] The server sends the generated agenda items to the terminal, and the terminal displays the list on its screen. Specifically, the server transfers the generated agenda items as data packets to the terminal, and the agenda list is displayed on the terminal's display. The input is agenda data from the server, and the output is the display of agenda items on the user interface.

[0368] Step 4:

[0369] During the meeting, the server uses speech recognition technology to collect participants' voices as audio data and converts it to text. It also uses an emotion engine to analyze participants' emotions from the audio and video data. Specifically, the server processes data acquired from microphones and cameras installed in the meeting room in real time. The input is the audio and video data from the meeting, and the output is the transcribed speech and the results of the emotion analysis.

[0370] Step 5:

[0371] The server generates instructions appropriate for the meeting's progress based on the sentiment analysis results and notifies the terminals. Specifically, if the server receives an analysis result indicating "participant interest is declining," it creates an instruction such as "consider sharing relevant success stories at this point" and notifies the participants' terminals. The input is the sentiment analysis result, and the output is the notification of the instruction content.

[0372] Step 6:

[0373] After the meeting, the server organizes the meeting data and uses generative AI technology to create detailed meeting minutes. This includes sentiment data and decisions made. Specifically, all aggregated data is stored on the server, and the AI ​​model automatically creates a list and documented meeting minutes. The input is all the data from the meeting, and the output is a complete meeting minute.

[0374] Step 7:

[0375] The server shares the generated meeting minutes with the participants. Specifically, it uses email sending functions or cloud storage to distribute the minutes to participants' email addresses or designated folders. The input is the meeting minutes data, and the output is distribution to each participant.

[0376] (Application Example 2)

[0377] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the smart glasses 214 as the "terminal".

[0378] In modern community-participatory meetings, there is a need for methods to efficiently gather diverse opinions and ensure smooth proceedings while understanding participants' emotional states in real time. Furthermore, there is a demand for providing high-quality feedback after the meeting that takes into account participants' level of interest and emotions. Currently, there is a problem in that emotional awareness during meetings is insufficient, making it difficult to adjust the proceedings and provide appropriate feedback.

[0379] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0380] This invention includes a server that analyzes participants' voice and video during a meeting, recognizes and analyzes emotional data in real time to support the progress of the meeting, generates and presents relevant suggestions in real time, and automatically organizes decisions and emotional data after the meeting and generates meeting minutes. This enables accurate understanding of participants' emotions and facilitates efficient and smooth meeting management and feedback provision.

[0381] "Purpose information" refers to information about the specific goals and topics that participants aim to achieve in a meeting or gathering.

[0382] "Predicted agenda items" are themes and topics that should be discussed during the meeting, generated by an AI algorithm based on the input objective information.

[0383] "Presentation means" refers to a system or device for visually or audibly communicating generated predicted agenda items and related proposals to participants.

[0384] The "analysis means" is a component that collects participants' voice and video data in real time and performs emotion recognition and analysis of the content of their statements.

[0385] "Emotional data" refers to information representing the emotional state extracted from participants' voices and facial expressions.

[0386] "Real-time recognition and analysis" refers to a process where data collection and processing occur almost simultaneously, allowing for immediate results.

[0387] A "proposal generation tool" is a system that creates and presents specific proposals to revitalize or adjust a meeting, depending on the emotional state of the participants and the progress of the discussion.

[0388] "Organizational tools" refer to the function of organizing data collected after a meeting and creating meeting minutes in a format that is easy for participants to understand.

[0389] Meeting minutes are documents that summarize what was said, decided, and how people felt during a meeting, and are shared with the participants.

[0390] To realize this invention, the server is first connected to a system equipped with a speech recognition API and an emotion recognition engine. This allows it to acquire and analyze in real time the audio and video data transmitted from terminals during a meeting. Specifically, it uses Google Cloud Speech-to-Text API and Microsoft Azure Cognitive Services to convert the audio data into text. It also uses Amazon Rekognition and other facial recognition APIs to extract emotion data from participants' facial expressions.

[0391] The terminal functions as an interface for participants joining a meeting, providing a means for inputting purpose information. This allows participants to send the meeting topic and desired outcomes to the server in advance. Based on this information, the server uses an AI algorithm to generate predicted agenda items and presents them to participants through the terminal, thereby streamlining the meeting process.

[0392] As the meeting progresses, the server uses collected sentiment data to generate relevant suggestions in real time and notify participants' terminals if their interest wanes. This helps maintain the meeting's energy and effectively supports decision-making.

[0393] After the meeting concludes, the server organizes all the comments and sentiment data, and automatically generates meeting minutes. These minutes are shared with participants via their devices, allowing them to easily review past meeting content. Sentiment data is also used to prepare for future meetings.

[0394] For example, when a user holds a meeting to "consider proposals for renovating a local park," the system can suggest predictive agenda items such as "improving methods for gathering citizen opinions" and "the possibility of installing new playground equipment." Furthermore, if participants' interest wanes, the server can send a notification suggesting they "consider sharing successful examples from neighboring areas."

[0395] An example of a prompt might be: "In a community-participatory meeting, we are discussing park renovations. Participants are losing interest. Please suggest ways to rekindle their interest."

[0396] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0397] Step 1:

[0398] Users input meeting objectives using a terminal. This input includes specific themes such as "Considering proposals for the renovation of local parks." The terminal sends this information to the server, which receives it. This information serves as the basis for generating predicted agenda items to be discussed at the meeting.

[0399] Step 2:

[0400] Based on the received objective information, the server uses an AI algorithm to generate predicted agenda items. This process involves referencing past relevant meeting data and templates to create agenda items that are helpful in addressing the issues. The generated predicted agenda items include topics such as "improving methods for collecting citizen opinions" and "the possibility of installing new playground equipment." The server then sends these to the terminal.

[0401] Step 3:

[0402] During the meeting, the terminal continuously transmits participants' audio and video to the server. The server uses a speech recognition API to convert the audio data into text and a facial recognition API to extract emotional data from the video. Specifically, it utilizes the Google Cloud Speech-to-Text API and Amazon Rekognition to transcribe spoken content and analyze emotional states. This allows for an understanding of the participants' levels of interest and engagement.

[0403] Step 4:

[0404] The server evaluates the progress of the meeting based on the analyzed sentiment data. If it determines that participants' interest is waning, it uses a generative AI model to create relevant suggestions. For example, it might generate a message such as, "Consider sharing success stories from your local area." This suggestion is then notified to the user via their device.

[0405] Step 5:

[0406] After the meeting, the server organizes all spoken text and sentiment data and automatically generates meeting minutes. These minutes are shared on participants' devices so they can view them at any time. Specifically, a text editing algorithm is used to organize the spoken content into a summarized format, and it includes an evaluation of the importance of each agenda item based on changes in sentiment.

[0407] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0408] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0409] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0410] [Third Embodiment]

[0411] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0412] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0413] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0414] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0415] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0416] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0417] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0418] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0419] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0420] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0421] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0422] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0423] The system for implementing this invention aims to improve the efficiency and effectiveness of meetings, and achieves this objective by providing multiple functions. The system begins with the user entering meeting objective information into a terminal. This information specifically indicates the goals to be achieved at the meeting and is transmitted to the server.

[0424] The server activates an AI algorithm based on the received target information, analyzing past data and related information to generate predicted agenda items. These predicted agenda items are a list of important topics that should be discussed in the meeting. The generated agenda items are sent from the server to the terminal and presented to the user.

[0425] Users can review the presented agenda and edit it as needed. This clarifies the direction of the meeting and allows for smoother progress.

[0426] Once the meeting begins, the server analyzes the meeting's audio data through a speech recognition system and converts it into text data in real time. This analysis records participants' statements and simultaneously monitors the progress of the meeting. The server checks whether the discussion is on topic and provides participants with relevant information as needed.

[0427] After the meeting, the server automatically organizes the decisions and next actions based on the analyzed data. This data is compiled into meeting minutes, and the organized information is automatically shared with participants. These minutes include specific action plans, responsible parties, and deadlines, enabling more effective follow-up after the meeting.

[0428] For example, if a user enters the objective information as "determine the marketing strategy for a new product," the server will generate specific predictive agenda items such as "competitor analysis," "target market selection," and "advertising channel selection." If the discussion strays from the main topic during the meeting, the server will provide participants with a notification such as "Let's return to the current agenda." In this way, it is possible to consistently improve the quality and efficiency of meetings.

[0429] The following describes the processing flow.

[0430] Step 1:

[0431] The user uses their device to input meeting objectives. The user describes the specific goals of the meeting on the input screen and presses the submit button.

[0432] Step 2:

[0433] The terminal receives the target information entered by the user and sends it to the server. The data is encrypted and sent to the server via a secure communication channel.

[0434] Step 3:

[0435] The server receives the target information and uses an AI algorithm to generate predicted agenda items based on that information. It refers to past meeting data and related information to list appropriate items.

[0436] Step 4:

[0437] The server sends the generated predicted agenda items to the terminal. The generated agenda items are displayed in the user interface.

[0438] Step 5:

[0439] Users review the predicted agenda items presented on their device and edit them as needed. Once the user confirms the final agenda, it is sent to the server.

[0440] Step 6:

[0441] During the meeting, the server uses speech recognition technology to analyze participants' voices in real time. It converts their speech into text data and monitors the meeting's progress.

[0442] Step 7:

[0443] The server provides relevant data and support information to speakers as needed during the meeting. If the agenda deviates from its intended direction, it will propose corrections.

[0444] Step 8:

[0445] Once the meeting ends, the server automatically organizes the decisions and next actions based on the audio and text data.

[0446] Step 9:

[0447] The server generates meeting minutes based on the information it has compiled, and automatically distributes these minutes to all participants. Distribution is done via email or a sharing platform.

[0448] (Example 1)

[0449] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0450] Traditional meeting systems often rely on participants' skills and preparation to ensure effective use of time. Furthermore, discussions frequently veer off-topic, making progress management difficult. Additionally, decisions and action plans after meetings may not be reliably shared, leading to insufficient follow-up.

[0451] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0452] In this invention, the server includes a device for inputting meeting purpose information, a device for generating predicted agenda items based on the input purpose information, and a device for presenting the generated predicted agenda items. This makes it possible to improve the efficiency and results of meetings and make effective use of time. It also supports progress management and ensures that follow-up after meetings is carried out reliably.

[0453] A "device" is a set of hardware and software components designed to perform a specific function.

[0454] "Meeting objective information" refers to information that represents the specific goals and objectives that the meeting aims to achieve.

[0455] A "predicted agenda" is a list of important topics that should be discussed at the meeting, based on past data and relevant information.

[0456] "Presentation" refers to the act of visually displaying generated information or data to the user.

[0457] "Analysis" is the process of understanding meaning and obtaining information based on collected data.

[0458] "Audio information" refers to data that includes audio recordings made by participants during a meeting.

[0459] "Text information" refers to the information format obtained after converting audio information into text data.

[0460] A "record" is a document that details the proceedings and decisions made at a meeting.

[0461] "Sharing" refers to the act of distributing generated information or data among participants and making it accessible to them.

[0462] This system is an integrated platform designed to support more efficient and productive meetings. Users first input meeting objective information into a terminal, including specific meeting goals such as "determine the marketing strategy for the new product." The terminal then sends this objective information to a server. The server receives the information and activates an AI algorithm. This AI algorithm analyzes historical databases and relevant information sources to generate a predictive agenda of key topics to be discussed in the meeting. For example, a prompt such as "Please suggest the agenda necessary to achieve the meeting's objectives" can be entered into the AI ​​model.

[0463] The server uses Google Cloud AI and various speech recognition services (e.g., Google Speech-to-Text) to carry out this process. The generated predicted topics are sent by the server to the terminal and presented to the user. The user can review the presented topics and edit them as needed.

[0464] During the meeting, the server converts participants' voice data into text data via a speech recognition system and monitors the progress according to the agenda. It supports efficient progress by providing reminders such as "Let's return to the current agenda" as needed.

[0465] After the meeting concludes, the server automatically organizes the analyzed data and generates a record containing decisions and action plans. This record is shared with all participants, enabling effective post-meeting follow-up.

[0466] In this way, by integrating multiple functions, this system can support the entire process of a meeting, achieving a higher level of effectiveness and efficiency.

[0467] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0468] Step 1:

[0469] The user enters meeting objectives into a terminal. This input is text information that linguistically expresses specific goals and topics. The terminal sends the entered information as digital data to the server. This step provides the basic data needed to initially set the focus of the meeting.

[0470] Step 2:

[0471] The server activates an AI algorithm based on the objective information received from the terminal. During this process, a prompt (e.g., "Please suggest the agenda items necessary to achieve the objectives of this meeting") is input to the generating AI model. The server analyzes past databases and related information to generate predicted agenda items. This data processing generates a list of important topics to be discussed in the meeting. This list is output as an agenda item tailored to the user's needs.

[0472] Step 3:

[0473] The server sends the generated predicted agenda to the terminal, which then presents the content to the user. The user can review the presented agenda and edit it as needed. The edited agenda is output as application information to help the user adjust the direction of the discussion.

[0474] Step 4:

[0475] Once the meeting begins, the server retrieves participants' voice data through a speech recognition system. This voice data is converted into text data through a service like Google Speech-to-Text. The server analyzes this text and monitors whether the meeting is progressing according to the agenda. This analysis checks the consistency between what is said and the agenda, and provides notifications such as "Let's return to the current agenda" if necessary.

[0476] Step 5:

[0477] After the meeting concludes, the server organizes the data collected during the meeting and generates a record containing decisions and action plans. This includes extracting key information based on real-time converted text data. The generated record is automatically shared with participants via their devices. This step enables efficient post-meeting follow-up.

[0478] (Application Example 1)

[0479] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0480] In fields such as urban development and public transportation, meetings involving diverse stakeholders require efficient information sharing and decision-making. However, discussions can be lengthy and inefficient, necessitating optimization of meeting procedures. Furthermore, systems are needed to quickly and accurately follow up on decisions.

[0481] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0482] This invention includes a server that provides information by aggregating smart environmental information to improve the efficiency of meetings related to urban planning and social management, and presenting relevant information; a voice recognition means that converts voice data into text data and monitors speech during meetings; and a means that cooperates with an urban management system to provide support information for the implementation of plans and activities decided at meetings. This improves the efficiency of meetings and enables rapid and accurate decision-making and support for the implementation of those decisions.

[0483] "Meeting objective information" refers to information that describes the specific goals and topics to be achieved in the meeting.

[0484] A "predicted agenda" is an agenda generated by an AI algorithm based on past data and objective information, indicating important topics for the meeting.

[0485] "Presentation means" refers to methods or devices for showing information to participants visually or audibly.

[0486] "Speech recognition means" refers to technology that converts participants' voices into text data and analyzes the progress of the meeting and the content of their statements.

[0487] "Organizational methods" refer to methods or processes for automatically organizing decisions and minutes after a meeting and compiling them into a report.

[0488] A "sharing method" refers to a system for distributing generated meeting minutes and related information to participants.

[0489] "Information provision methods" refer to methods of gathering smart environment information related to urban planning and social management, and presenting participants with the information necessary to support the efficiency of meetings.

[0490] A "city management system" is a management system that uses information technology to support the implementation of planning and activities in a city.

[0491] To implement this invention, it is necessary to build a system that aims to improve the efficiency and effectiveness of meetings. This system is centered around a server, terminals, and users, each playing a specific role.

[0492] First, the user inputs meeting objectives using their device. This defines the specific goals and topics to be addressed in the meeting. Based on this objective information, the server uses an AI algorithm to generate a predicted agenda. The predicted agenda is a list of important topics for the meeting, compiled by analyzing past data and related information.

[0493] The server sends the generated predicted agenda to the terminal and presents it to the user. The user can review this agenda and edit it as needed, thereby clarifying the direction of the meeting.

[0494] During the meeting, the server converts participants' voice data into text in real time via a speech recognition system (e.g., Google Cloud Speech-to-Text API). This process allows for recording participants' statements and monitoring the progress of the meeting. Furthermore, information provision tools are used to present relevant information as needed, providing timely information necessary for urban planning and social management.

[0495] After the meeting ends, the server automatically organizes the decisions made based on the data recorded during the meeting and generates meeting minutes. The generated minutes, which include specific action plans, responsible parties, and deadlines, are shared with participants via their devices. As a result, post-meeting follow-up can be conducted more effectively.

[0496] For example, in a meeting about improving urban transportation systems, instructions such as "Please proceed to the next proposal point" could be given. An example of a prompt for a generative AI model would be, "Please present a specific action plan that is expected as an outcome of this meeting."

[0497] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0498] Step 1:

[0499] The user enters meeting objective information using a terminal. This information specifically outlines the goals and agenda of the meeting and is sent to the server. The input here includes the user's desired outcomes and discussion topics.

[0500] Step 2:

[0501] Based on the received purpose information, the server activates the AI ​​algorithm. The prompt "Generate an agenda based on the purpose of this meeting" is input to the AI, which then analyzes past data and relevant information to generate a predicted agenda. This creates a list of important topics to be discussed at the meeting, which is then sent from the server to the terminal.

[0502] Step 3:

[0503] Users review the predicted agenda generated on their devices and edit it as needed. This editing process is intended to improve the content of the agenda, and the newly edited agenda becomes the final one. This editing includes adding or deleting items from the agenda.

[0504] Step 4:

[0505] After the meeting begins, the server uses a speech recognition system to convert the meeting's audio data into text data in real time. The audio input is the participants' statements, and the output is the corresponding text data. This conversion allows participants' statements to be recorded in text format, while simultaneously monitoring their alignment with the current agenda.

[0506] Step 5:

[0507] The server continues its analysis, checking whether the meeting is progressing according to the agenda. If necessary, it provides relevant information in real time to assist in the meeting's progress. At this time, it uses the prompt "Please provide relevant materials" to request relevant information.

[0508] Step 6:

[0509] At the end of the meeting, the server automatically organizes the decisions made based on the text data and generates meeting minutes. This process verifies the statements made against the agenda and clearly outlines the final action plan.

[0510] Step 7:

[0511] The generated meeting minutes are shared with participants via their devices. This allows for post-meeting follow-up and ensures that all participants understand the agreed-upon future action plan. Sending meeting minutes is typically done using electronic messaging systems.

[0512] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0513] This invention proposes a system incorporating an emotion engine to recognize and analyze participants' emotions in real time during a meeting. This system begins with the user inputting meeting objective information using a terminal, which is then transmitted to a server. Based on this objective information, the server generates predicted agenda items using an AI algorithm and presents them to the terminal.

[0514] During the meeting, the server uses speech recognition to convert participants' voices into text, while an emotion engine analyzes participants' emotions from the audio and video. This analysis is used as data to control the progress of the meeting. For example, if it determines that participants' interest is waning, the server will suggest changing the agenda or taking a break to ensure smooth progress.

[0515] Furthermore, the server organizes all decisions, including sentiment data, after the meeting and generates meeting minutes. The generated minutes are automatically shared with participants, and the sentiment data is used to prepare for the next meeting.

[0516] For example, if a user enters the objective information as "discuss the market strategy for a new product," the server will generate and present agenda items such as "analysis of competing products" and "identification of the target market" based on this information. If the emotion engine determines that participants' levels of interest have decreased during the meeting or presentation, the server will send a notification to the terminal saying, "Consider sharing relevant success stories at this point." This allows for efficient decision-making while maintaining the energy of the meeting.

[0517] The following describes the processing flow.

[0518] Step 1:

[0519] The user uses a terminal to input meeting purpose information. The user enters the specific meeting goals in the input field on the terminal and presses the submit button.

[0520] Step 2:

[0521] The terminal sends the target information entered by the user to the server. The data is sent to the server in an encrypted form for security purposes.

[0522] Step 3:

[0523] The server receives the target information and generates predicted agenda items using an AI algorithm. The server then refers to a historical database, extracts relevant topics, and lists them as agenda items.

[0524] Step 4:

[0525] The server sends the generated predicted agenda items to the terminal and presents them to the user.

[0526] Step 5:

[0527] The user reviews the predicted agenda presented on their device and edits it if necessary. Once the user confirms the final agenda, they send that information to the server.

[0528] Step 6:

[0529] Once the meeting begins, the server activates the speech recognition system and emotion engine, analyzing participants' audio and video data in real time.

[0530] Step 7:

[0531] The server references data from the emotion engine to analyze the emotional state of the participants. For example, if it detects signs that a participant is bored, the server will then present suggestions on the terminal to improve the meeting's flow.

[0532] Step 8:

[0533] Once the meeting concludes, the server organizes the decisions and next actions based on audio and sentiment data. It then generates meeting minutes summarizing the results.

[0534] Step 9:

[0535] The server automatically distributes meeting minutes to all participants. The minutes also include sentiment analysis results, which can be used to prepare for future meetings.

[0536] (Example 2)

[0537] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0538] In meetings, there are challenges in understanding participants' changing emotions and interests in real time and appropriately adjusting the meeting's progress. Furthermore, there is a problem in automatically generating detailed meeting minutes, including emotional information, after the meeting, making effective preparation for the next meeting difficult. In addition, accurately recording what is said during the meeting and managing it in a way that allows for easy later reference is also challenging.

[0539] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0540] This invention includes a server configuration that acquires participants' voices as audio data during a meeting and analyzes their emotions using natural language processing technology; a configuration that generates instructions to support the progress of the meeting and notifies participants' devices; and a configuration that automatically organizes and documents all decisions, including emotion data, after the meeting. This makes it possible to grasp changes in participants' emotions and interests in real time and adjust the meeting progress appropriately based on that data. In addition, the automatically generated detailed minutes enable effective preparation for the next meeting, and statements made during the meeting can be accurately recorded and managed.

[0541] A "device" refers to a combination of hardware and software used for information processing, particularly those that enable input and output manipulation.

[0542] "Configuration" refers to the arrangement of elements or processes combined to achieve a specific function, or the effect achieved by that arrangement.

[0543] "Audio data" refers to data that represents speech in digital format and contains fundamental information for speech recognition and analysis.

[0544] "Natural language processing technology" refers to artificial intelligence technology that enables computers to understand, appropriately analyze, and generate human language.

[0545] "Analyzing emotions" refers to the process of identifying emotional states from input data and quantifying or categorizing those states.

[0546] "Instructions" are pieces of information or recommendations provided to encourage specific behaviors, and they involve suggestions or requests for action.

[0547] "Documenting" refers to the process of systematically recording information in document form, making it easy to refer to and analyze later.

[0548] "Real-time" means that processing and responses occur simultaneously with the occurrence of an event, and refers to a state in which information is transmitted or processed without delay.

[0549] This invention provides a system that analyzes the emotions and statements of participants during a meeting in real time to support the progress of the meeting, and at the same time automatically generates detailed meeting minutes that include emotions after the meeting. This system consists of three main elements: users, terminals, and servers.

[0550] The user uses a terminal to input information about the purpose of the meeting. For example, they might input information such as "Discuss the market strategy for the new product." This terminal is used to receive user input and send the entered information to the server.

[0551] The server uses a generative AI model to generate relevant agenda items based on the received target information. This AI model analyzes text data using natural language processing techniques and generates predictive agenda items based on user input. Specific software used here includes machine learning frameworks such as TensorFlow and PyTorch. As a result, agenda items such as "Analyze Competitor Products" and "Identify Target Markets" are generated and presented to the terminal.

[0552] During the meeting, the server uses speech recognition technology (e.g., Google Cloud Speech-to-Text) to convert participants' voices into text data, while simultaneously performing sentiment analysis that includes video data. An AI-based sentiment engine is used for the analysis. This allows for real-time monitoring of participants' emotional states and generates instructions appropriate for the meeting's progress. For example, if a participant's interest wanes, the server sends a notification to their device saying, "Consider sharing relevant success stories at this point."

[0553] After the meeting, the server uses AI technology to automatically generate detailed meeting minutes based on sentiment data and decisions made during the meeting. These minutes are shared with participants via email or cloud services, allowing them to accurately understand the meeting content and contribute to preparing for future meetings.

[0554] An example of a prompt to a generative AI model would be: "Meeting purpose information has been entered. Please list the predicted topics. For example, if the purpose is to 'discuss the market strategy for a new product,' what topics would you suggest?" Through this prompt, the server can generate the most appropriate topics.

[0555] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0556] Step 1:

[0557] The user uses a terminal to input meeting objective information. The entered objective information is sent from the terminal to the server. Specifically, the user uses the input field on the terminal and the keyboard to type "Discuss the market strategy for the new product" and presses the send button. The input is text data, and the output is data sent to the server.

[0558] Step 2:

[0559] The server sends prompts to the generating AI model based on the received objective information, and generates relevant agenda items. Here, natural language processing technology is used to analyze text data and generate predicted agenda items. Specifically, in response to the prompt "Discuss market strategy for new product," the AI ​​model on the server generates agenda items such as "Analyze competing products" and "Identify target market." The input is the text data of the objective information, and the output is a list of generated agenda items.

[0560] Step 3:

[0561] The server sends the generated agenda items to the terminal, and the terminal displays the list on its screen. Specifically, the server transfers the generated agenda items as data packets to the terminal, and the agenda list is displayed on the terminal's display. The input is agenda data from the server, and the output is the display of agenda items on the user interface.

[0562] Step 4:

[0563] During the meeting, the server uses speech recognition technology to collect participants' voices as audio data and converts it to text. It also uses an emotion engine to analyze participants' emotions from the audio and video data. Specifically, the server processes data acquired from microphones and cameras installed in the meeting room in real time. The input is the audio and video data from the meeting, and the output is the transcribed speech and the results of the emotion analysis.

[0564] Step 5:

[0565] The server generates instructions appropriate for the meeting's progress based on the sentiment analysis results and notifies the terminals. Specifically, if the server receives an analysis result indicating "participant interest is declining," it creates an instruction such as "consider sharing relevant success stories at this point" and notifies the participants' terminals. The input is the sentiment analysis result, and the output is the notification of the instruction content.

[0566] Step 6:

[0567] After the meeting, the server organizes the meeting data and uses generative AI technology to create detailed meeting minutes. This includes sentiment data and decisions made. Specifically, all aggregated data is stored on the server, and the AI ​​model automatically creates a list and documented meeting minutes. The input is all the data from the meeting, and the output is a complete meeting minute.

[0568] Step 7:

[0569] The server shares the generated meeting minutes with the participants. Specifically, it uses email sending functions or cloud storage to distribute the minutes to participants' email addresses or designated folders. The input is the meeting minutes data, and the output is distribution to each participant.

[0570] (Application Example 2)

[0571] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0572] In modern community-participatory meetings, there is a need for methods to efficiently gather diverse opinions and ensure smooth proceedings while understanding participants' emotional states in real time. Furthermore, there is a demand for providing high-quality feedback after the meeting that takes into account participants' level of interest and emotions. Currently, there is a problem in that emotional awareness during meetings is insufficient, making it difficult to adjust the proceedings and provide appropriate feedback.

[0573] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0574] This invention includes a server that analyzes participants' voice and video during a meeting, recognizes and analyzes emotional data in real time to support the progress of the meeting, generates and presents relevant suggestions in real time, and automatically organizes decisions and emotional data after the meeting and generates meeting minutes. This enables accurate understanding of participants' emotions and facilitates efficient and smooth meeting management and feedback provision.

[0575] "Purpose information" refers to information about the specific goals and topics that participants aim to achieve in a meeting or gathering.

[0576] "Predicted agenda items" are themes and topics that should be discussed during the meeting, generated by an AI algorithm based on the input objective information.

[0577] "Presentation means" refers to a system or device for visually or audibly communicating generated predicted agenda items and related proposals to participants.

[0578] The "analysis means" is a component that collects participants' voice and video data in real time and performs emotion recognition and analysis of the content of their statements.

[0579] "Emotional data" refers to information representing the emotional state extracted from participants' voices and facial expressions.

[0580] "Real-time recognition and analysis" refers to a process where data collection and processing occur almost simultaneously, allowing for immediate results.

[0581] A "proposal generation tool" is a system that creates and presents specific proposals to revitalize or adjust a meeting, depending on the emotional state of the participants and the progress of the discussion.

[0582] "Organizational tools" refer to the function of organizing data collected after a meeting and creating meeting minutes in a format that is easy for participants to understand.

[0583] Meeting minutes are documents that summarize what was said, decided, and how people felt during a meeting, and are shared with the participants.

[0584] To realize this invention, the server is first connected to a system equipped with a speech recognition API and an emotion recognition engine. This allows it to acquire and analyze in real time the audio and video data transmitted from terminals during a meeting. Specifically, it uses Google Cloud Speech-to-Text API and Microsoft Azure Cognitive Services to convert the audio data into text. It also uses Amazon Rekognition and other facial recognition APIs to extract emotion data from participants' facial expressions.

[0585] The terminal functions as an interface for participants joining a meeting, providing a means for inputting purpose information. This allows participants to send the meeting topic and desired outcomes to the server in advance. Based on this information, the server uses an AI algorithm to generate predicted agenda items and presents them to participants through the terminal, thereby streamlining the meeting process.

[0586] As the meeting progresses, the server uses collected sentiment data to generate relevant suggestions in real time and notify participants' terminals if their interest wanes. This helps maintain the meeting's energy and effectively supports decision-making.

[0587] After the meeting concludes, the server organizes all the comments and sentiment data, and automatically generates meeting minutes. These minutes are shared with participants via their devices, allowing them to easily review past meeting content. Sentiment data is also used to prepare for future meetings.

[0588] For example, when a user holds a meeting to "consider proposals for renovating a local park," the system can suggest predictive agenda items such as "improving methods for gathering citizen opinions" and "the possibility of installing new playground equipment." Furthermore, if participants' interest wanes, the server can send a notification suggesting they "consider sharing successful examples from neighboring areas."

[0589] An example of a prompt might be: "In a community-participatory meeting, we are discussing park renovations. Participants are losing interest. Please suggest ways to rekindle their interest."

[0590] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0591] Step 1:

[0592] Users input meeting objectives using a terminal. This input includes specific themes such as "Considering proposals for the renovation of local parks." The terminal sends this information to the server, which receives it. This information serves as the basis for generating predicted agenda items to be discussed at the meeting.

[0593] Step 2:

[0594] Based on the received objective information, the server uses an AI algorithm to generate predicted agenda items. This process involves referencing past relevant meeting data and templates to create agenda items that are helpful in addressing the issues. The generated predicted agenda items include topics such as "improving methods for collecting citizen opinions" and "the possibility of installing new playground equipment." The server then sends these to the terminal.

[0595] Step 3:

[0596] During the meeting, the terminal continuously transmits participants' audio and video to the server. The server uses a speech recognition API to convert the audio data into text and a facial recognition API to extract emotional data from the video. Specifically, it utilizes the Google Cloud Speech-to-Text API and Amazon Rekognition to transcribe spoken content and analyze emotional states. This allows for an understanding of the participants' levels of interest and engagement.

[0597] Step 4:

[0598] The server evaluates the progress of the meeting based on the analyzed sentiment data. If it determines that participants' interest is waning, it uses a generative AI model to create relevant suggestions. For example, it might generate a message such as, "Consider sharing success stories from your local area." This suggestion is then notified to the user via their device.

[0599] Step 5:

[0600] After the meeting, the server organizes all spoken text and sentiment data and automatically generates meeting minutes. These minutes are shared on participants' devices so they can view them at any time. Specifically, a text editing algorithm is used to organize the spoken content into a summarized format, and it includes an evaluation of the importance of each agenda item based on changes in sentiment.

[0601] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0602] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0603] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0604] [Fourth Embodiment]

[0605] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0606] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0607] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0608] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0609] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0610] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0611] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0612] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0613] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0614] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0615] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0616] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0617] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0618] The system for implementing this invention aims to improve the efficiency and effectiveness of meetings, and achieves this objective by providing multiple functions. The system begins with the user entering meeting objective information into a terminal. This information specifically indicates the goals to be achieved at the meeting and is transmitted to the server.

[0619] The server activates an AI algorithm based on the received target information, analyzing past data and related information to generate predicted agenda items. These predicted agenda items are a list of important topics that should be discussed in the meeting. The generated agenda items are sent from the server to the terminal and presented to the user.

[0620] Users can review the presented agenda and edit it as needed. This clarifies the direction of the meeting and allows for smoother progress.

[0621] Once the meeting begins, the server analyzes the meeting's audio data through a speech recognition system and converts it into text data in real time. This analysis records participants' statements and simultaneously monitors the progress of the meeting. The server checks whether the discussion is on topic and provides participants with relevant information as needed.

[0622] After the meeting, the server automatically organizes the decisions and next actions based on the analyzed data. This data is compiled into meeting minutes, and the organized information is automatically shared with participants. These minutes include specific action plans, responsible parties, and deadlines, enabling more effective follow-up after the meeting.

[0623] For example, if a user enters the objective information as "determine the marketing strategy for a new product," the server will generate specific predictive agenda items such as "competitor analysis," "target market selection," and "advertising channel selection." If the discussion strays from the main topic during the meeting, the server will provide participants with a notification such as "Let's return to the current agenda." In this way, it is possible to consistently improve the quality and efficiency of meetings.

[0624] The following describes the processing flow.

[0625] Step 1:

[0626] The user uses their device to input meeting objectives. The user describes the specific goals of the meeting on the input screen and presses the submit button.

[0627] Step 2:

[0628] The terminal receives the target information entered by the user and sends it to the server. The data is encrypted and sent to the server via a secure communication channel.

[0629] Step 3:

[0630] The server receives the target information and uses an AI algorithm to generate predicted agenda items based on that information. It refers to past meeting data and related information to list appropriate items.

[0631] Step 4:

[0632] The server sends the generated predicted agenda items to the terminal. The generated agenda items are displayed in the user interface.

[0633] Step 5:

[0634] Users review the predicted agenda items presented on their device and edit them as needed. Once the user confirms the final agenda, it is sent to the server.

[0635] Step 6:

[0636] During the meeting, the server uses speech recognition technology to analyze participants' voices in real time. It converts their speech into text data and monitors the meeting's progress.

[0637] Step 7:

[0638] The server provides relevant data and support information to speakers as needed during the meeting. If the agenda deviates from its intended direction, it will propose corrections.

[0639] Step 8:

[0640] Once the meeting ends, the server automatically organizes the decisions and next actions based on the audio and text data.

[0641] Step 9:

[0642] The server generates meeting minutes based on the information it has compiled, and automatically distributes these minutes to all participants. Distribution is done via email or a sharing platform.

[0643] (Example 1)

[0644] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0645] Traditional meeting systems often rely on participants' skills and preparation to ensure effective use of time. Furthermore, discussions frequently veer off-topic, making progress management difficult. Additionally, decisions and action plans after meetings may not be reliably shared, leading to insufficient follow-up.

[0646] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0647] In this invention, the server includes a device for inputting meeting purpose information, a device for generating predicted agenda items based on the input purpose information, and a device for presenting the generated predicted agenda items. This makes it possible to improve the efficiency and results of meetings and make effective use of time. It also supports progress management and ensures that follow-up after meetings is carried out reliably.

[0648] A "device" is a set of hardware and software components designed to perform a specific function.

[0649] "Meeting objective information" refers to information that represents the specific goals and objectives that the meeting aims to achieve.

[0650] A "predicted agenda" is a list of important topics that should be discussed at the meeting, based on past data and relevant information.

[0651] "Presentation" refers to the act of visually displaying generated information or data to the user.

[0652] "Analysis" is the process of understanding meaning and obtaining information based on collected data.

[0653] "Audio information" refers to data that includes audio recordings made by participants during a meeting.

[0654] "Text information" refers to the information format obtained after converting audio information into text data.

[0655] A "record" is a document that details the proceedings and decisions made at a meeting.

[0656] "Sharing" refers to the act of distributing generated information or data among participants and making it accessible to them.

[0657] This system is an integrated platform designed to support more efficient and productive meetings. Users first input meeting objective information into a terminal, including specific meeting goals such as "determine the marketing strategy for the new product." The terminal then sends this objective information to a server. The server receives the information and activates an AI algorithm. This AI algorithm analyzes historical databases and relevant information sources to generate a predictive agenda of key topics to be discussed in the meeting. For example, a prompt such as "Please suggest the agenda necessary to achieve the meeting's objectives" can be entered into the AI ​​model.

[0658] The server uses Google Cloud AI and various speech recognition services (e.g., Google Speech-to-Text) to carry out this process. The generated predicted topics are sent by the server to the terminal and presented to the user. The user can review the presented topics and edit them as needed.

[0659] During the meeting, the server converts participants' voice data into text data via a speech recognition system and monitors the progress according to the agenda. It supports efficient progress by providing reminders such as "Let's return to the current agenda" as needed.

[0660] After the meeting concludes, the server automatically organizes the analyzed data and generates a record containing decisions and action plans. This record is shared with all participants, enabling effective post-meeting follow-up.

[0661] In this way, by integrating multiple functions, this system can support the entire process of a meeting, achieving a higher level of effectiveness and efficiency.

[0662] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0663] Step 1:

[0664] The user enters meeting objectives into a terminal. This input is text information that linguistically expresses specific goals and topics. The terminal sends the entered information as digital data to the server. This step provides the basic data needed to initially set the focus of the meeting.

[0665] Step 2:

[0666] The server activates an AI algorithm based on the objective information received from the terminal. During this process, a prompt (e.g., "Please suggest the agenda items necessary to achieve the objectives of this meeting") is input to the generating AI model. The server analyzes past databases and related information to generate predicted agenda items. This data processing generates a list of important topics to be discussed in the meeting. This list is output as an agenda item tailored to the user's needs.

[0667] Step 3:

[0668] The server sends the generated predicted agenda to the terminal, which then presents the content to the user. The user can review the presented agenda and edit it as needed. The edited agenda is output as application information to help the user adjust the direction of the discussion.

[0669] Step 4:

[0670] Once the meeting begins, the server retrieves participants' voice data through a speech recognition system. This voice data is converted into text data through a service like Google Speech-to-Text. The server analyzes this text and monitors whether the meeting is progressing according to the agenda. This analysis checks the consistency between what is said and the agenda, and provides notifications such as "Let's return to the current agenda" if necessary.

[0671] Step 5:

[0672] After the meeting concludes, the server organizes the data collected during the meeting and generates a record containing decisions and action plans. This includes extracting key information based on real-time converted text data. The generated record is automatically shared with participants via their devices. This step enables efficient post-meeting follow-up.

[0673] (Application Example 1)

[0674] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0675] In fields such as urban development and public transportation, meetings involving diverse stakeholders require efficient information sharing and decision-making. However, discussions can be lengthy and inefficient, necessitating optimization of meeting procedures. Furthermore, systems are needed to quickly and accurately follow up on decisions.

[0676] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0677] This invention includes a server that provides information by aggregating smart environmental information to improve the efficiency of meetings related to urban planning and social management, and presenting relevant information; a voice recognition means that converts voice data into text data and monitors speech during meetings; and a means that cooperates with an urban management system to provide support information for the implementation of plans and activities decided at meetings. This improves the efficiency of meetings and enables rapid and accurate decision-making and support for the implementation of those decisions.

[0678] "Meeting objective information" refers to information that describes the specific goals and topics to be achieved in the meeting.

[0679] A "predicted agenda" is an agenda generated by an AI algorithm based on past data and objective information, indicating important topics for the meeting.

[0680] "Presentation means" refers to methods or devices for showing information to participants visually or audibly.

[0681] "Speech recognition means" refers to technology that converts participants' voices into text data and analyzes the progress of the meeting and the content of their statements.

[0682] "Organizational methods" refer to methods or processes for automatically organizing decisions and minutes after a meeting and compiling them into a report.

[0683] A "sharing method" refers to a system for distributing generated meeting minutes and related information to participants.

[0684] "Information provision methods" refer to methods of gathering smart environment information related to urban planning and social management, and presenting participants with the information necessary to support the efficiency of meetings.

[0685] A "city management system" is a management system that uses information technology to support the implementation of planning and activities in a city.

[0686] To implement this invention, it is necessary to build a system that aims to improve the efficiency and effectiveness of meetings. This system is centered around a server, terminals, and users, each playing a specific role.

[0687] First, the user inputs meeting objectives using their device. This defines the specific goals and topics to be addressed in the meeting. Based on this objective information, the server uses an AI algorithm to generate a predicted agenda. The predicted agenda is a list of important topics for the meeting, compiled by analyzing past data and related information.

[0688] The server sends the generated predicted agenda to the terminal and presents it to the user. The user can review this agenda and edit it as needed, thereby clarifying the direction of the meeting.

[0689] During the meeting, the server converts participants' voice data into text in real time via a speech recognition system (e.g., Google Cloud Speech-to-Text API). This process allows for recording participants' statements and monitoring the progress of the meeting. Furthermore, information provision tools are used to present relevant information as needed, providing timely information necessary for urban planning and social management.

[0690] After the meeting ends, the server automatically organizes the decisions made based on the data recorded during the meeting and generates meeting minutes. The generated minutes, which include specific action plans, responsible parties, and deadlines, are shared with participants via their devices. As a result, post-meeting follow-up can be conducted more effectively.

[0691] For example, in a meeting about improving urban transportation systems, instructions such as "Please proceed to the next proposal point" could be given. An example of a prompt for a generative AI model would be, "Please present a specific action plan that is expected as an outcome of this meeting."

[0692] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0693] Step 1:

[0694] The user enters meeting objective information using a terminal. This information specifically outlines the goals and agenda of the meeting and is sent to the server. The input here includes the user's desired outcomes and discussion topics.

[0695] Step 2:

[0696] Based on the received purpose information, the server activates the AI ​​algorithm. The prompt "Generate an agenda based on the purpose of this meeting" is input to the AI, which then analyzes past data and relevant information to generate a predicted agenda. This creates a list of important topics to be discussed at the meeting, which is then sent from the server to the terminal.

[0697] Step 3:

[0698] Users review the predicted agenda generated on their devices and edit it as needed. This editing process is intended to improve the content of the agenda, and the newly edited agenda becomes the final one. This editing includes adding or deleting items from the agenda.

[0699] Step 4:

[0700] After the meeting begins, the server uses a speech recognition system to convert the meeting's audio data into text data in real time. The audio input is the participants' statements, and the output is the corresponding text data. This conversion allows participants' statements to be recorded in text format, while simultaneously monitoring their alignment with the current agenda.

[0701] Step 5:

[0702] The server continues its analysis, checking whether the meeting is progressing according to the agenda. If necessary, it provides relevant information in real time to assist in the meeting's progress. At this time, it uses the prompt "Please provide relevant materials" to request relevant information.

[0703] Step 6:

[0704] At the end of the meeting, the server automatically organizes the decisions made based on the text data and generates meeting minutes. This process verifies the statements made against the agenda and clearly outlines the final action plan.

[0705] Step 7:

[0706] The generated meeting minutes are shared with participants via their devices. This allows for post-meeting follow-up and ensures that all participants understand the agreed-upon future action plan. Sending meeting minutes is typically done using electronic messaging systems.

[0707] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0708] This invention proposes a system incorporating an emotion engine to recognize and analyze participants' emotions in real time during a meeting. This system begins with the user inputting meeting objective information using a terminal, which is then transmitted to a server. Based on this objective information, the server generates predicted agenda items using an AI algorithm and presents them to the terminal.

[0709] During the meeting, the server uses speech recognition to convert participants' voices into text, while an emotion engine analyzes participants' emotions from the audio and video. This analysis is used as data to control the progress of the meeting. For example, if it determines that participants' interest is waning, the server will suggest changing the agenda or taking a break to ensure smooth progress.

[0710] Furthermore, the server organizes all decisions, including sentiment data, after the meeting and generates meeting minutes. The generated minutes are automatically shared with participants, and the sentiment data is used to prepare for the next meeting.

[0711] For example, if a user enters the objective information as "discuss the market strategy for a new product," the server will generate and present agenda items such as "analysis of competing products" and "identification of the target market" based on this information. If the emotion engine determines that participants' levels of interest have decreased during the meeting or presentation, the server will send a notification to the terminal saying, "Consider sharing relevant success stories at this point." This allows for efficient decision-making while maintaining the energy of the meeting.

[0712] The following describes the processing flow.

[0713] Step 1:

[0714] The user uses a terminal to input meeting purpose information. The user enters the specific meeting goals in the input field on the terminal and presses the submit button.

[0715] Step 2:

[0716] The terminal sends the target information entered by the user to the server. The data is sent to the server in an encrypted form for security purposes.

[0717] Step 3:

[0718] The server receives the target information and generates predicted agenda items using an AI algorithm. The server then refers to a historical database, extracts relevant topics, and lists them as agenda items.

[0719] Step 4:

[0720] The server sends the generated predicted agenda items to the terminal and presents them to the user.

[0721] Step 5:

[0722] The user reviews the predicted agenda presented on their device and edits it if necessary. Once the user confirms the final agenda, they send that information to the server.

[0723] Step 6:

[0724] Once the meeting begins, the server activates the speech recognition system and emotion engine, analyzing participants' audio and video data in real time.

[0725] Step 7:

[0726] The server references data from the emotion engine to analyze the emotional state of the participants. For example, if it detects signs that a participant is bored, the server will then present suggestions on the terminal to improve the meeting's flow.

[0727] Step 8:

[0728] Once the meeting concludes, the server organizes the decisions and next actions based on audio and sentiment data. It then generates meeting minutes summarizing the results.

[0729] Step 9:

[0730] The server automatically distributes meeting minutes to all participants. The minutes also include sentiment analysis results, which can be used to prepare for future meetings.

[0731] (Example 2)

[0732] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0733] In meetings, there are challenges in understanding participants' changing emotions and interests in real time and appropriately adjusting the meeting's progress. Furthermore, there is a problem in automatically generating detailed meeting minutes, including emotional information, after the meeting, making effective preparation for the next meeting difficult. In addition, accurately recording what is said during the meeting and managing it in a way that allows for easy later reference is also challenging.

[0734] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0735] This invention includes a server configuration that acquires participants' voices as audio data during a meeting and analyzes their emotions using natural language processing technology; a configuration that generates instructions to support the progress of the meeting and notifies participants' devices; and a configuration that automatically organizes and documents all decisions, including emotion data, after the meeting. This makes it possible to grasp changes in participants' emotions and interests in real time and adjust the meeting progress appropriately based on that data. In addition, the automatically generated detailed minutes enable effective preparation for the next meeting, and statements made during the meeting can be accurately recorded and managed.

[0736] A "device" refers to a combination of hardware and software used for information processing, particularly those that enable input and output manipulation.

[0737] "Configuration" refers to the arrangement of elements or processes combined to achieve a specific function, or the effect achieved by that arrangement.

[0738] "Audio data" refers to data that represents speech in digital format and contains fundamental information for speech recognition and analysis.

[0739] "Natural language processing technology" refers to artificial intelligence technology that enables computers to understand, appropriately analyze, and generate human language.

[0740] "Analyzing emotions" refers to the process of identifying emotional states from input data and quantifying or categorizing those states.

[0741] "Instructions" are pieces of information or recommendations provided to encourage specific behaviors, and they involve suggestions or requests for action.

[0742] "Documenting" refers to the process of systematically recording information in document form, making it easy to refer to and analyze later.

[0743] "Real-time" means that processing and responses occur simultaneously with the occurrence of an event, and refers to a state in which information is transmitted or processed without delay.

[0744] This invention provides a system that analyzes the emotions and statements of participants during a meeting in real time to support the progress of the meeting, and at the same time automatically generates detailed meeting minutes that include emotions after the meeting. This system consists of three main elements: users, terminals, and servers.

[0745] The user uses a terminal to input information about the purpose of the meeting. For example, they might input information such as "Discuss the market strategy for the new product." This terminal is used to receive user input and send the entered information to the server.

[0746] The server uses a generative AI model to generate relevant agenda items based on the received target information. This AI model analyzes text data using natural language processing techniques and generates predictive agenda items based on user input. Specific software used here includes machine learning frameworks such as TensorFlow and PyTorch. As a result, agenda items such as "Analyze Competitor Products" and "Identify Target Markets" are generated and presented to the terminal.

[0747] During the meeting, the server uses speech recognition technology (e.g., Google Cloud Speech-to-Text) to convert participants' voices into text data, while simultaneously performing sentiment analysis that includes video data. An AI-based sentiment engine is used for the analysis. This allows for real-time monitoring of participants' emotional states and generates instructions appropriate for the meeting's progress. For example, if a participant's interest wanes, the server sends a notification to their device saying, "Consider sharing relevant success stories at this point."

[0748] After the meeting, the server uses AI technology to automatically generate detailed meeting minutes based on sentiment data and decisions made during the meeting. These minutes are shared with participants via email or cloud services, allowing them to accurately understand the meeting content and contribute to preparing for future meetings.

[0749] An example of a prompt to a generative AI model would be: "Meeting purpose information has been entered. Please list the predicted topics. For example, if the purpose is to 'discuss the market strategy for a new product,' what topics would you suggest?" Through this prompt, the server can generate the most appropriate topics.

[0750] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0751] Step 1:

[0752] The user uses a terminal to input meeting objective information. The entered objective information is sent from the terminal to the server. Specifically, the user uses the input field on the terminal and the keyboard to type "Discuss the market strategy for the new product" and presses the send button. The input is text data, and the output is data sent to the server.

[0753] Step 2:

[0754] The server sends prompts to the generating AI model based on the received objective information, and generates relevant agenda items. Here, natural language processing technology is used to analyze text data and generate predicted agenda items. Specifically, in response to the prompt "Discuss market strategy for new product," the AI ​​model on the server generates agenda items such as "Analyze competing products" and "Identify target market." The input is the text data of the objective information, and the output is a list of generated agenda items.

[0755] Step 3:

[0756] The server sends the generated agenda items to the terminal, and the terminal displays the list on its screen. Specifically, the server transfers the generated agenda items as data packets to the terminal, and the agenda list is displayed on the terminal's display. The input is agenda data from the server, and the output is the display of agenda items on the user interface.

[0757] Step 4:

[0758] During the meeting, the server uses speech recognition technology to collect participants' voices as audio data and converts it to text. It also uses an emotion engine to analyze participants' emotions from the audio and video data. Specifically, the server processes data acquired from microphones and cameras installed in the meeting room in real time. The input is the audio and video data from the meeting, and the output is the transcribed speech and the results of the emotion analysis.

[0759] Step 5:

[0760] The server generates instructions appropriate for the meeting's progress based on the sentiment analysis results and notifies the terminals. Specifically, if the server receives an analysis result indicating "participant interest is declining," it creates an instruction such as "consider sharing relevant success stories at this point" and notifies the participants' terminals. The input is the sentiment analysis result, and the output is the notification of the instruction content.

[0761] Step 6:

[0762] After the meeting, the server organizes the meeting data and uses generative AI technology to create detailed meeting minutes. This includes sentiment data and decisions made. Specifically, all aggregated data is stored on the server, and the AI ​​model automatically creates a list and documented meeting minutes. The input is all the data from the meeting, and the output is a complete meeting minute.

[0763] Step 7:

[0764] The server shares the generated meeting minutes with the participants. Specifically, it uses email sending functions or cloud storage to distribute the minutes to participants' email addresses or designated folders. The input is the meeting minutes data, and the output is distribution to each participant.

[0765] (Application Example 2)

[0766] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0767] In modern community-participatory meetings, there is a need for methods to efficiently gather diverse opinions and ensure smooth proceedings while understanding participants' emotional states in real time. Furthermore, there is a demand for providing high-quality feedback after the meeting that takes into account participants' level of interest and emotions. Currently, there is a problem in that emotional awareness during meetings is insufficient, making it difficult to adjust the proceedings and provide appropriate feedback.

[0768] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0769] This invention includes a server that analyzes participants' voice and video during a meeting, recognizes and analyzes emotional data in real time to support the progress of the meeting, generates and presents relevant suggestions in real time, and automatically organizes decisions and emotional data after the meeting and generates meeting minutes. This enables accurate understanding of participants' emotions and facilitates efficient and smooth meeting management and feedback provision.

[0770] "Purpose information" refers to information about the specific goals and topics that participants aim to achieve in a meeting or gathering.

[0771] "Predicted agenda items" are themes and topics that should be discussed during the meeting, generated by an AI algorithm based on the input objective information.

[0772] "Presentation means" refers to a system or device for visually or audibly communicating generated predicted agenda items and related proposals to participants.

[0773] The "analysis means" is a component that collects participants' voice and video data in real time and performs emotion recognition and analysis of the content of their statements.

[0774] "Emotional data" refers to information representing the emotional state extracted from participants' voices and facial expressions.

[0775] "Real-time recognition and analysis" refers to a process where data collection and processing occur almost simultaneously, allowing for immediate results.

[0776] A "proposal generation tool" is a system that creates and presents specific proposals to revitalize or adjust a meeting, depending on the emotional state of the participants and the progress of the discussion.

[0777] "Organizational tools" refer to the function of organizing data collected after a meeting and creating meeting minutes in a format that is easy for participants to understand.

[0778] Meeting minutes are documents that summarize what was said, decided, and how people felt during a meeting, and are shared with the participants.

[0779] To realize this invention, the server is first connected to a system equipped with a speech recognition API and an emotion recognition engine. This allows it to acquire and analyze in real time the audio and video data transmitted from terminals during a meeting. Specifically, it uses Google Cloud Speech-to-Text API and Microsoft Azure Cognitive Services to convert the audio data into text. It also uses Amazon Rekognition and other facial recognition APIs to extract emotion data from participants' facial expressions.

[0780] The terminal functions as an interface for participants joining a meeting, providing a means for inputting purpose information. This allows participants to send the meeting topic and desired outcomes to the server in advance. Based on this information, the server uses an AI algorithm to generate predicted agenda items and presents them to participants through the terminal, thereby streamlining the meeting process.

[0781] As the meeting progresses, the server uses collected sentiment data to generate relevant suggestions in real time and notify participants' terminals if their interest wanes. This helps maintain the meeting's energy and effectively supports decision-making.

[0782] After the meeting concludes, the server organizes all the comments and sentiment data, and automatically generates meeting minutes. These minutes are shared with participants via their devices, allowing them to easily review past meeting content. Sentiment data is also used to prepare for future meetings.

[0783] For example, when a user holds a meeting to "consider proposals for renovating a local park," the system can suggest predictive agenda items such as "improving methods for gathering citizen opinions" and "the possibility of installing new playground equipment." Furthermore, if participants' interest wanes, the server can send a notification suggesting they "consider sharing successful examples from neighboring areas."

[0784] An example of a prompt might be: "In a community-participatory meeting, we are discussing park renovations. Participants are losing interest. Please suggest ways to rekindle their interest."

[0785] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0786] Step 1:

[0787] Users input meeting objectives using a terminal. This input includes specific themes such as "Considering proposals for the renovation of local parks." The terminal sends this information to the server, which receives it. This information serves as the basis for generating predicted agenda items to be discussed at the meeting.

[0788] Step 2:

[0789] Based on the received objective information, the server uses an AI algorithm to generate predicted agenda items. This process involves referencing past relevant meeting data and templates to create agenda items that are helpful in addressing the issues. The generated predicted agenda items include topics such as "improving methods for collecting citizen opinions" and "the possibility of installing new playground equipment." The server then sends these to the terminal.

[0790] Step 3:

[0791] During the meeting, the terminal continuously transmits participants' audio and video to the server. The server uses a speech recognition API to convert the audio data into text and a facial recognition API to extract emotional data from the video. Specifically, it utilizes the Google Cloud Speech-to-Text API and Amazon Rekognition to transcribe spoken content and analyze emotional states. This allows for an understanding of the participants' levels of interest and engagement.

[0792] Step 4:

[0793] The server evaluates the progress of the meeting based on the analyzed sentiment data. If it determines that participants' interest is waning, it uses a generative AI model to create relevant suggestions. For example, it might generate a message such as, "Consider sharing success stories from your local area." This suggestion is then notified to the user via their device.

[0794] Step 5:

[0795] After the meeting, the server organizes all spoken text and sentiment data and automatically generates meeting minutes. These minutes are shared on participants' devices so they can view them at any time. Specifically, a text editing algorithm is used to organize the spoken content into a summarized format, and it includes an evaluation of the importance of each agenda item based on changes in sentiment.

[0796] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0797] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0798] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0799] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0800] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. In the upper and lower directions of the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. Also, the upper side of the concentric circles is where "pleasant" emotions are located, and the lower side is where "unpleasant" emotions are located. In this way, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0801] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0802] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0803] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0804] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0805] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0806] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0807] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0808] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0809] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0810] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0811] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0812] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0813] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0814] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0815] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0816] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0817] The following is further disclosed regarding the embodiments described above.

[0818] (Claim 1)

[0819] An input method for entering meeting purpose information,

[0820] A generation means for generating predicted topics based on input objective information,

[0821] A means of presenting the generated predicted agenda,

[0822] An analytical tool that analyzes participants' voices during a meeting to support the progress of the meeting,

[0823] A means of organizing decisions made after a meeting and generating meeting minutes,

[0824] A means of sharing the generated meeting minutes with participants,

[0825] A system that includes this.

[0826] (Claim 2)

[0827] The system according to claim 1, comprising a proposal generation means for generating and presenting relevant proposals based on input objective information.

[0828] (Claim 3)

[0829] The system according to claim 1, comprising speech recognition means for converting audio data into text data and monitoring speech during a meeting.

[0830] "Example 1"

[0831] (Claim 1)

[0832] A device for inputting meeting purpose information,

[0833] A device that generates predicted agenda items based on input objective information,

[0834] A device that presents the generated predicted agenda,

[0835] A device that analyzes participants' voices during a meeting and supports the progress of the meeting in accordance with the agenda,

[0836] A device that converts collected audio information into text information,

[0837] A device that automatically organizes decisions made after a meeting and generates a record,

[0838] A device for sharing the generated records with participants,

[0839] A system that includes this.

[0840] (Claim 2)

[0841] The system according to claim 1, comprising a suggestion device that generates and displays relevant suggestions based on input objective information.

[0842] (Claim 3)

[0843] The system according to claim 1, comprising an audio device that converts audio information into text information and monitors speech during a meeting.

[0844] "Application Example 1"

[0845] (Claim 1)

[0846] An input method for entering meeting purpose information,

[0847] A generation means for generating predicted topics based on input objective information,

[0848] A means of presenting the generated predicted agenda,

[0849] An analytical tool that analyzes participants' voices during a meeting to support the progress of the meeting,

[0850] A speech recognition system that converts audio data from a meeting into text data and monitors speech,

[0851] A means of organizing decisions made after a meeting and generating meeting minutes,

[0852] A means of sharing the generated meeting minutes with participants,

[0853] A means of providing information that collects smart environment information and presents related information in order to improve the efficiency of meetings on urban planning and social management,

[0854] A system that includes this.

[0855] (Claim 2)

[0856] The system according to claim 1, comprising a proposal generation means for generating and presenting relevant proposals based on input objective information.

[0857] (Claim 3)

[0858] The system according to claim 1, comprising means for providing support information to enable the implementation of plans and activities decided at a meeting, in conjunction with an urban management system.

[0859] "Example 2 of combining an emotion engine"

[0860] (Claim 1)

[0861] A device for inputting meeting purpose information,

[0862] A configuration that generates predicted agenda items based on input objective information,

[0863] A device for displaying the generated agenda,

[0864] The system involves acquiring participants' voices as audio data during a meeting and analyzing their emotions using natural language processing technology.

[0865] A configuration that generates instructions to support the progress of the meeting and notifies participants' devices,

[0866] A system that automatically organizes and documents all decisions, including emotional data, after a meeting.

[0867] Means of providing the generated document to the participants,

[0868] A system that includes this.

[0869] (Claim 2)

[0870] The system according to claim 1, comprising a component that generates and presents relevant suggestions based on input objective information.

[0871] (Claim 3)

[0872] The system according to claim 1, comprising a function to convert audio data into text data and monitor speech during a meeting.

[0873] "Application example 2 when combining with an emotional engine"

[0874] (Claim 1)

[0875] A means of entering meeting purpose information,

[0876] A means for generating predicted topics based on input objective information,

[0877] A means of presenting the generated predicted agenda,

[0878] A method to support the progress of a meeting by analyzing participants' audio and video during the meeting, and recognizing and analyzing emotional data in real time.

[0879] A means of generating and presenting relevant suggestions in real time,

[0880] A method for automatically organizing decisions and sentiment data after a meeting and generating meeting minutes,

[0881] A means of sharing the generated meeting minutes with participants,

[0882] A system that includes this.

[0883] (Claim 2)

[0884] The system according to claim 1, comprising means for generating relevant proposals based on input objective information and supporting resident-participatory meetings.

[0885] (Claim 3)

[0886] The system according to claim 1, comprising means for converting audio data into text data and monitoring speech and emotions during a meeting. [Explanation of symbols]

[0887] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. An input method for entering meeting purpose information, A generation means for generating predicted topics based on input objective information, A means of presenting the generated predicted agenda, An analytical tool that analyzes participants' voices during a meeting to support the progress of the meeting, A speech recognition system that converts audio data from a meeting into text data and monitors speech, A means of organizing decisions made after a meeting and generating meeting minutes, A means of sharing the generated meeting minutes with participants, A means of providing information that collects smart environment information and presents related information in order to improve the efficiency of meetings on urban planning and social management, A system that includes this.

2. The system according to claim 1, comprising a proposal generation means for generating and presenting relevant proposals based on input objective information.

3. The system according to claim 1, comprising means for providing support information to enable the implementation of plans and activities decided at a meeting, in conjunction with an urban management system.

Images