System

Abstract

A system is provided.SOLUTION: A means for receiving conference voice data in real time and converting the voice content into text, a means for analyzing the text data and detecting that the conversation is interrupted for a certain period of time, a means for generating an agenda or a question to proceed to next when the interruption is detected and notifying the agenda or the question to a participant terminal, a means for recording the voice of the conference, converting the voice into text in real time, sending the text to a summarization algorithm, and extracting an important agenda and an important action item, A system comprising: means for automatically generating a story and an action list and transmitting the story and the action list to a participant terminal; means for grasping a schedule of each member and transmitting a reminder of a progress report at an optimum timing; means for automatically extracting task data from a chat tool or a work breakdown structure and notifying an administrator terminal of an important task; and means for collecting current status data of a project on a daily basis, evaluating each task, generating a response policy for a high-risk task, and notifying the administrator terminal of the response policy.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In system development projects, project manager's lack of skills and inadequate communication are factors that lead to project failure. In particular, meeting management, minutes creation, progress report coordination, task management, and risk management are often not performed smoothly. This often leads to project delays and overlooked risks, which negatively impacts the quality and delivery date of the final deliverables. The present invention aims to solve these problems and provide a method and system that effectively supports project management. [Means for solving the problem]

[0005] The present invention includes a system that receives conference audio data in real time and converts the audio content into text using natural language processing technology, analyzes the converted text data, detects pauses in the conversation for a certain period of time, and, upon detection, generates agenda items and questions to be addressed next and notifies the participant terminals. It also includes a system that records the conference audio, converts it into text in real time, sends it to a summarization algorithm to extract important agenda items and action items, automatically generates minutes and action lists, and sends them to the participant terminals. It also includes a system that analyzes calendar data to grasp each member's schedule and send progress report reminders at optimal times. It also includes a system that automatically extracts task data from chat tools and work breakdown structures and notifies the manager terminal of important tasks. The system further includes a system that collects project status data daily, evaluates each task, generates response policies for high-risk tasks, and notifies the manager terminal.

[0006] "Conference voice data" refers to digital data of voices spoken in a conference.

[0007] "Natural language processing technology" is a technology that enables computers to understand, analyze, and generate human language.

[0008] "Converting to text" refers to the process of converting data such as audio into character string data.

[0009] The "certain period of time" is within a time range preset by the system.

[0010] "Detecting a pause in conversation" means that the system recognizes that the flow of conversation has been interrupted for a certain period of time or longer.

[0011] An "agenda" is a major topic or theme to be discussed or discussed at a meeting.

[0012] A question is an inquiry or question that is used to elicit information.

[0013] A "participant terminal" is a device used by an individual user participating in a conference.

[0014] "Minutes" are documents that record the discussions, decisions, and action items of a meeting.

[0015] An "action list" is a list of specific action items decided in a meeting or project.

[0016] "Calendar data" is data that records a user's plans and schedules.

[0017] A "progress report reminder" is a notification that lets you know when it's time to report on the progress of a project.

[0018] A "chat tool" is software that allows you to exchange messages in real time.

[0019] A work breakdown structure is a structure for managing a large project by dividing it into smaller tasks.

[0020] "Task data" is information about individual work items within a project.

[0021] "Administrator terminal" refers to a device used by a project manager or administrator.

[0022] "Current status data" refers to data that indicates the current progress and status of a project.

[0023] A "high-risk task" is one that is likely to cause problems or have a significant impact on progress.

[0024] A "response policy" is a specific measure or procedure for dealing with a specific problem or risk. [Brief explanation of the drawings]

[0025] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0026] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0027] First, the terms used in the following description will be explained.

[0028] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0029] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0030] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0031] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0032] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0033] [First embodiment]

[0034] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0035] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0036] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0037] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0038] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0039] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0040] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0041] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0042] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0043] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0044] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0045] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0046] The present invention is a system designed to support the role of a project manager (PM) in a system development project. The system uses a large-scale language model to provide multiple functions that optimize the PM's communication and project management tasks. The following describes an embodiment of the system.

[0047] Overall system overview

[0048] This system consists of a server, terminals, and users. The server performs the central processing, and the terminals provide an interface with users. Users are team members and PMs involved in the project.

[0049] Facilitation function

[0050] When a user starts a meeting, the device collects the meeting's audio data and sends it to the server in real time. The server receives the audio data and converts it into text using natural language processing technology. When the server detects silence for a certain period of time (for example, 10 seconds), it generates the next agenda item and questions to be asked and notifies the participants' devices. This allows the meeting to proceed smoothly and important topics to be discussed.

[0051] Examples:

[0052] If 10 seconds of silence occurs during a meeting, the server generates the message "Let's move on to the next agenda item. Next, let's talk about the budget review." and displays it on the participants' devices.

[0053] Minutes function

[0054] When the user ends the meeting, the device sends the audio recorded during the meeting to the server. The server converts the audio into text and sends the text to a summarization algorithm to extract important topics and action items. Based on the extracted information, minutes and action lists are automatically generated and sent to the participants' devices.

[0055] Examples:

[0056] After the meeting, the server generates minutes that include information such as "discussion on the timing of releasing new features" and "tasks for each member," and sends them to the devices of all participants.

[0057] Communication Features

[0058] The server analyzes each user's calendar data to understand their schedules, finds out when the user is not busy, and sends progress report reminders at the appropriate times. These reminders are displayed on the device.

[0059] Examples:

[0060] The server will send a reminder at 10:00 AM on the following Monday, such as "Please report on the progress of this week's tasks."

[0061] PMO Functions

[0062] Task data is extracted from chat tools and work breakdown structures (WBS) and sent to the server. The server evaluates the extracted task data and identifies important tasks. It then notifies the PM's device of important tasks and manages the task progress. Task progress is periodically checked and reminders are sent as necessary.

[0063] Examples:

[0064] The server extracts important tasks such as "preparing for weekly review" and "reporting to client" and notifies the PM's terminal.

[0065] Risk Management Function

[0066] The server collects project status data daily and evaluates the progress of each task. It identifies high-risk tasks and tasks that are expected to be delayed, and generates a response plan for them. The generated response plan is then notified to the administrator's terminal.

[0067] Examples:

[0068] The server presents specific response guidelines, such as "Progress on Task A is behind schedule, so please consider adding resources or extending the delivery date," and notifies the PM's terminal.

[0069] With the above functions, this system can significantly reduce the burden on project managers and increase the success rate of projects.

[0070] The processing flow will be explained below.

[0071] (Facilitate function)

[0072] Step 1:

[0073] The terminal uses the microphone of the terminal to collect voice data during the conference and transmits it to the server in real time.

[0074] Step 2:

[0075] The server converts the received voice data into text using natural language processing technology.

[0076] Step 3:

[0077] The server analyzes the text data and determines whether a certain period of silence (for example, 10 seconds) is detected.

[0078] Step 4:

[0079] If silence is detected, the server generates the next topic to be discussed and appropriate questions.

[0080] Step 5:

[0081] The server transmits the generated agenda and questions to the terminal.

[0082] Step 6:

[0083] The terminal displays the received agenda and questions to the user.

[0084] (Meeting minutes function)

[0085] Step 1:

[0086] The terminal records audio data during the conference and transmits the data to the server after the conference ends.

[0087] Step 2:

[0088] The server converts the received voice data into text using natural language processing technology.

[0089] Step 3:

[0090] The server feeds the generated text data into a summarization algorithm to extract important topics and action items.

[0091] Step 4:

[0092] The server automatically generates minutes and action lists based on the extracted information.

[0093] Step 5:

[0094] The server transmits the generated minutes and action list to the terminal.

[0095] Step 6:

[0096] The terminal displays the received minutes and action list to the user.

[0097] (Communication function)

[0098] Step 1:

[0099] The server periodically collects and analyzes the user's calendar data.

[0100] Step 2:

[0101] The server takes into account the user's schedule and generates progress report reminders at optimal times.

[0102] Step 3:

[0103] The server transmits the generated reminder to the terminal.

[0104] Step 4:

[0105] The terminal notifies the user of the received reminder.

[0106] (PMO function)

[0107] Step 1:

[0108] The server automatically extracts task data from chat tools and work breakdown structures (WBS) on a regular basis.

[0109] Step 2:

[0110] The server evaluates the extracted task data and identifies important tasks.

[0111] Step 3:

[0112] The server notifies the administrator terminal of the identified important task.

[0113] Step 4:

[0114] The server periodically monitors the progress of the task and sends reminders to the administrator terminal as needed.

[0115] (Risk management function)

[0116] Step 1:

[0117] The server collects project status data daily.

[0118] Step 2:

[0119] The server evaluates tasks based on the collected data and identifies high-risk tasks.

[0120] Step 3:

[0121] The server generates a response policy for the identified risks.

[0122] Step 4:

[0123] The server notifies the administrator terminal of the generated response policy.

[0124] Step 5:

[0125] The terminal displays the received response policy to the administrator.

[0126] Example 1

[0127] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0128] Conventional systems require project managers (PMs) to manually manage a wide range of tasks, conduct meetings, create minutes, and monitor the progress of each member, requiring a great deal of time and effort. This can slow down project progress and delay the early detection and response of risks. Other issues include overlooking important agenda items during meetings and insufficient follow-up afterward. There is a need for a system that can solve these problems, reduce the burden on PMs, and improve the success rate of projects.

[0129] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0130] In this invention, the server includes: means for receiving conference audio data in real time and converting the audio content into text using natural language processing technology; means for analyzing the converted text data and detecting when the conversation has stopped for a certain period of time; means for generating next agenda items and questions and notifying participant terminals when detected; means for recording the conference audio, converting it into text in real time, sending it to a summarization algorithm to extract important agenda items and action items, automatically generating minutes and action lists and sending them to participant terminals; means for analyzing calendar data to understand each member's schedule and sending progress report reminders at optimal times; means for automatically extracting task data from a chat tool or a work breakdown structure and notifying a manager terminal of important tasks; means for collecting project status data daily, evaluating each task, generating response policies for high-risk tasks and notifying the manager terminal; and means for proposing project management actions based on the generated response policies. This automates the manual work of project managers and enables efficient project management.

[0131] "Conference audio data" refers to audio information generated during a conference, including statements and discussions.

[0132] "Natural language processing technology" is a technology that allows computers to understand and process the language that humans use on a daily basis.

[0133] "Text data" refers to text information generated from audio data using natural language processing technology.

[0134] "Conversion" is the process of switching data from one format to another, in this case changing audio data to text data.

[0135] The "certain period of time" refers to the length of time that serves as a reference in the detection process, and in this invention specifically means 10 seconds.

[0136] A "lull in conversation" refers to a state in which no one speaks for a certain period of time or longer.

[0137] An "agenda" refers to the subject or topic of a meeting or discussion.

[0138] A "question" is a question posed to obtain specific information or opinions.

[0139] "Participant terminal" refers to an electronic device such as a computer or mobile device used by a user participating in a conference.

[0140] A "summarization algorithm" is a computational procedure for extracting the important parts of long text data and presenting them in a shortened form.

[0141] "Agenda and action items" refers to the main topics covered during the meeting and any specific action items that will follow.

[0142] "Minutes" refers to a document that records the contents of a meeting.

[0143] An "action list" is a document that lists specific actions to be taken after a meeting.

[0144] "Calendar data" refers to schedule information written on an electronic calendar used by a user.

[0145] A "progress report" is a report on the progress of a project, usually done on a regular basis.

[0146] A "reminder" is information that notifies a user to remind them of a particular action or task.

[0147] A "chat tool" is software that users use to communicate instantly through text.

[0148] A work breakdown structure (WBS) is a diagram or list that breaks down project tasks hierarchically into manageable sections.

[0149] "Task data" refers to information about specific work items to be done as part of a project.

[0150] "Project status data" refers to data that indicates the current progress of a project and the status of tasks.

[0151] "Evaluation" refers to the process of analyzing data and determining its status or value based on specific criteria.

[0152] A "high-risk task" is a work item that is likely not to proceed as planned.

[0153] A "response policy" refers to a countermeasure or action plan for a specific problem or situation.

[0154] "Server" refers to a computing device that performs central processing and exchanges data with other terminals.

[0155] "Means of suggestion" refers to the method by which the system notifies the user of actions and countermeasures and supports their progress.

[0156] MODE FOR CARRYING OUT THE INVENTION

[0157] This invention is a system designed to support the role of project managers (PMs) in system development projects. This system consists of three main components: a server, a terminal, and a user, and provides multiple functions to optimize the PM's communication and project management tasks.

[0158] Hardware and software used

[0159] Server: A computer device that performs central processing and exchanges data with other devices

[0160] Terminal: An electronic device (e.g., computer, mobile device) that provides an interface with a user.

[0161] Natural language processing technology: Google Cloud Speech-to-Text API

[0162] Summarization Algorithm: BERT-Based Summarization Model

[0163] Calendar Data Analysis: Google Calendar API

[0164] Task data extraction: Slack API, JIRA API

[0165] Collecting project status data: JIRA API, GitLab API

[0166] Features and specific operation examples

[0167] Facilitation function

[0168] When a user starts a meeting, the device collects the audio data of the meeting and sends it to the server in real time. When the server receives the audio data, it converts it into text using natural language processing technology. If the server detects silence for a certain period of time (for example, 10 seconds), it generates the next agenda item and questions and notifies the participants' devices.

[0169] Example: If 10 seconds of silence occurs during a meeting, the server generates the message "Let's move on to the next agenda item. Let's discuss the budget review next." and displays it on the participants' devices.

[0170] Minutes function

[0171] When the user ends the meeting, the device sends the audio data recorded during the meeting to the server, which converts the audio data into text using natural language processing technology and sends the text data to a summarization algorithm to extract important topics and action items. Based on this, minutes and action lists are automatically generated and sent to the participants' devices.

[0172] Example: After the meeting, the server generates minutes that include "discussions about the timing of releasing new features" and "tasks for each member," and sends them to the participants' devices.

[0173] Communication Features

[0174] The server analyzes each user's calendar data to understand their schedule, finds times when the user is not busy, and generates progress report reminders at those times and sends them to each device.

[0175] Example: The server sends a reminder to participants' devices at 10:00 a.m. on the following Monday, such as "Please report on the progress of this week's tasks."

[0176] PMO Functions

[0177] Task data is extracted from chat tools and work breakdown structures (WBS) and sent to the server. The server evaluates the extracted task data and identifies important tasks. It then notifies the PM's device of important tasks and manages the task progress. Task progress is periodically checked and reminders are sent as necessary.

[0178] Example: The server extracts important tasks such as "preparing for weekly review" and "reporting to client" and notifies the PM's terminal.

[0179] Risk Management Function

[0180] The server collects project status data daily and evaluates the progress of each task. It identifies high-risk tasks and tasks that are expected to be delayed, and generates a response plan for them. The generated response plan is then notified to the administrator's terminal.

[0181] Example: The server presents a specific response policy such as "Progress on Task A is behind schedule, so please consider adding resources or extending the deadline," and notifies the PM's terminal.

[0182] Examples of prompt statements

[0183] To use this system, the following is an example of a prompt sentence to input into the generative AI model:

[0184] Meeting Facilitation Functions:

[0185] Prompt: "Detect 10 seconds of silence during a meeting. Generate next agenda items or questions."

[0186] Minutes feature:

[0187] Prompt: "Convert the audio data from a meeting recording into text and generate a summary."

[0188] Communication features:

[0189] Prompt: "Analyze the user's calendar data to find the appropriate time for task progress reports and send reminders."

[0190] PMO Functions:

[0191] Prompt: "Extract tasks from the chat tool or WBS, identify important tasks, and notify the PM."

[0192] Risk Management Functions:

[0193] Prompt: "Evaluate the progress of tasks and generate and communicate specific action plans for high-risk tasks."

[0194] In this way, by using the system of the present invention, it is possible to significantly reduce the burden on project managers and improve the success rate of projects.

[0195] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0196] Facilitation function

[0197] Processing Steps

[0198] Step 1:

[0199] A user starts a conference using a terminal. The user inputs the conference start time, participant list, etc. into the system. This information becomes the initial data for the system.

[0200] Specific behavior:

[0201] The user enters the details of the meeting on the meeting setting screen and presses the "Start" button.

[0202] Step 2:

[0203] The device collects audio data during the meeting in real time using a built-in microphone or an external microphone, and stores the collected audio data in a buffer.

[0204] Specific behavior:

[0205] The device activates the microphone and begins collecting audio.

[0206] Input: Audio during the meeting

[0207] Output: Collected audio data

[0208] Step 3:

[0209] The device transmits the collected audio data to the server in real time using a secure protocol such as HTTPS.

[0210] Specific behavior:

[0211] The terminal uploads the voice data to the server at regular intervals.

[0212] Input: Collected audio data

[0213] Output: Audio data sent to the server

[0214] Step 4:

[0215] The server converts the received voice data into text using natural language processing technology (Google Cloud Speech-to-Text API).

[0216] Specific behavior:

[0217] The server calls the API and converts the audio data into text data.

[0218] Input: Audio data sent to the server

[0219] Output: Converted text data

[0220] Step 5:

[0221] The server analyzes the text data and detects silence for a certain period of time (10 seconds).

[0222] Specific behavior:

[0223] The server analyzes the time information in the text data and flags any silence of 10 seconds or more.

[0224] Input: Converted text data

[0225] Output: Silence detection flag

[0226] Step 6:

[0227] When the server detects silence, it generates the next agenda item and question to be asked and notifies the participants' terminals.

[0228] Specific behavior:

[0229] The server uses a generative AI model to generate the next agenda item or question and create a notification message.

[0230] Input: silence detection flag

[0231] Output: Generated agenda and questions

[0232] Specific behavior:

[0233] The generated agenda and questions are displayed on the participants' devices.

[0234] Minutes function

[0235] Processing Steps

[0236] Step 1:

[0237] The user ends the conference using the terminal. The end operation is performed through the terminal interface.

[0238] Specific behavior:

[0239] The user presses the "Finish" button.

[0240] Step 2:

[0241] The terminal transmits the audio data recorded during the meeting to the server.

[0242] Specific behavior:

[0243] The device uploads the recording data to the server.

[0244] Input: Recorded audio data

[0245] Output: Audio data sent to the server

[0246] Step 3:

[0247] The server converts the voice data into text using natural language processing technology.

[0248] Specific behavior:

[0249] The server calls the API and converts the audio data into text data.

[0250] Input: Audio data sent to the server

[0251] Output: Converted text data

[0252] Step 4:

[0253] The server sends the text data to a summarization algorithm (a BERT-based summarization model) to extract important topics and action items.

[0254] Specific behavior:

[0255] The server sends the data to a summary model to extract the important information.

[0256] Input: Converted text data

[0257] Output: Extracted important topics and action items

[0258] Step 5:

[0259] The server automatically generates minutes and action lists based on the extracted information.

[0260] Specific behavior:

[0261] The server automatically generates meeting minutes and action list templates by filling them with data.

[0262] Input: Extracted important agenda items and action items

[0263] Output: Auto-generated meeting minutes and action list

[0264] Step 6:

[0265] The server sends the generated minutes and action list to the participants' terminals.

[0266] Specific behavior:

[0267] The server sends the generated document to the terminal.

[0268] Input: Auto-generated meeting minutes and action list

[0269] Output: Document sent to participant's device

[0270] Communication Features

[0271] Processing Steps

[0272] Step 1:

[0273] The server analyzes each user's calendar data and keeps track of their schedules.

[0274] Specific behavior:

[0275] The server uses the Google Calendar API to retrieve and parse the user's calendar data.

[0276] Input: User's calendar data

[0277] Output: Analysis results

[0278] Step 2:

[0279] Based on the analysis results, the server identifies times when the user is not busy.

[0280] Specific behavior:

[0281] The server extracts available time slots from the analysis results.

[0282] Input: Analysis results

[0283] Output: Available time slots

[0284] Step 3:

[0285] The server generates appropriate reminders when the user is not busy and sends them to each device.

[0286] Specific behavior:

[0287] The server uses the generative AI model to create reminders and send them to the device.

[0288] Input: Available time slots

[0289] Output: The generated reminder

[0290] Specific behavior:

[0291] The reminder will be displayed on the user's device.

[0292] PMO Functions

[0293] Processing Steps

[0294] Step 1:

[0295] Task data is extracted from chat tools and work breakdown structures and sent to a server.

[0296] Specific behavior:

[0297] The server retrieves task data using the Slack API or JIRA API.

[0298] Input: Task data generated by chat tools and WBS

[0299] Output: Extracted task data

[0300] Step 2:

[0301] The server evaluates the extracted task data and identifies important tasks.

[0302] Specific behavior:

[0303] The server analyzes the task data and ranks it based on importance.

[0304] Input: Extracted task data

[0305] Output: Identified important tasks

[0306] Step 3:

[0307] The server notifies the PM's terminal of important tasks.

[0308] Specific behavior:

[0309] The server creates a notification message and sends it to the PM's terminal.

[0310] Input: Identified critical tasks

[0311] Output: Tasks notified to the PM's terminal

[0312] Step 4:

[0313] The server periodically checks the progress of each task.

[0314] Specific behavior:

[0315] The server collects task progress data daily and evaluates the current status.

[0316] Input: Task progress data

[0317] Output: Progress evaluation results

[0318] Step 5:

[0319] The server generates reminders as needed and sends them to the PM's terminal.

[0320] Specific behavior:

[0321] The server creates a reminder and sends it to the PM's device.

[0322] Input: Progress evaluation results

[0323] Output: The generated reminder

[0324] Risk Management Function

[0325] Processing Steps

[0326] Step 1:

[0327] The server collects project status data daily.

[0328] Specific behavior:

[0329] The server retrieves the latest data using the JIRA API or GitLab API.

[0330] Input: Project status data

[0331] Output: Collected data

[0332] Step 2:

[0333] The server evaluates the progress of the task based on the collected data.

[0334] Specific behavior:

[0335] The server analyzes the collected data and evaluates the progress of each task.

[0336] Input: Collected data

[0337] Output: Progress evaluation results

[0338] Step 3:

[0339] Based on the progress, the server identifies high-risk tasks and tasks that are expected to be delayed.

[0340] Specific behavior:

[0341] The server identifies risk tasks based on the progress evaluation results.

[0342] Input: Progress evaluation results

[0343] Output: Identified risk tasks

[0344] Step 4:

[0345] Once a risk is identified, the server generates a response policy.

[0346] Specific behavior:

[0347] The server uses the generative AI model to create a specific response policy.

[0348] Input: Identified Risk Tasks

[0349] Output: Generated response policy

[0350] Step 5:

[0351] The generated response policy is notified to the administrator terminal.

[0352] Specific behavior:

[0353] The server sends the response policy as a notification message to the administrator terminal.

[0354] Input: Generated response policy

[0355] Output: Response policy notified to administrator terminal

[0356] (Application example 1)

[0357] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0358] Conventional project management systems place a heavy burden on project managers, requiring a lot of time and effort, especially for running meetings, taking minutes, managing progress, and managing risks. As a result, it is difficult to send timely reminders and manage progress, especially when it comes to robot maintenance work and task management in factories, which can lead to issues that reduce project efficiency.

[0359] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0360] In this invention, the server has means for receiving conference audio data in real time and converting the audio content into text using natural language processing technology, means for analyzing the converted text data and detecting when the conversation has stopped for a certain period of time, means for generating the next agenda item and questions to be asked when this is detected and notifying the participant terminals, means for recording the conference audio, converting it into text in real time, sending it to a summarization algorithm to extract important agenda items and action items, automatically generating minutes and action lists and sending them to the participant terminals, and means for analyzing calendar data to grasp the schedule of each member and optimally The system includes a means for automatically extracting task data from chat tools and work breakdown structures and notifying the manager's terminal of important tasks, a means for collecting current project status data daily, evaluating each task, generating response policies for high-risk tasks and notifying the manager's terminal, a means for managing the progress of maintenance work using a smartphone and sending reminders, a means for supporting the progress of meetings using a facilitation function, and a means for collecting current status data within the factory, generating response policies for high-risk work and notifying the manager's terminal. This makes it possible to smoothly progress meetings, automate the creation of meeting minutes, and send timely reminders for maintenance work, significantly reducing the burden on project managers and improving the efficiency and success rate of projects.

[0361] "Conference audio data" refers to all audio data recorded during a conference.

[0362] "Natural language processing technology" refers to technology that enables computers to understand and process the natural language that humans use on a daily basis.

[0363] "Text data" refers to data obtained by converting voice data into character information.

[0364] "Detecting a pause in conversation for a certain period of time" refers to detecting that no speech has occurred within a specified period of time.

[0365] "Generating agendas and questions" refers to automatically creating next topics and questions to help keep the meeting moving.

[0366] The term "participant terminal" refers to an electronic terminal used by a user participating in a conference.

[0367] "Key Agenda and Action Items" refers to the key issues discussed at the meeting and the action items related to those issues.

[0368] "Minutes" refers to a document summarizing the matters discussed at a meeting.

[0369] An "action list" is a list of action plans and task items decided at a meeting.

[0370] "Calendar data" refers to data that records a user's schedule and plans.

[0371] "Progress report reminder" refers to a message that notifies you not to forget to report your progress.

[0372] "Chat tool" refers to an application for text-based communication.

[0373] A "work breakdown structure" refers to a structure in which a project is divided into task units.

[0374] "Task Data" refers to information related to each task in a project.

[0375] "Administrator terminal" refers to the electronic terminal used by the project manager or administrator.

[0376] "Project Status Data" means data regarding the current progress or status of a Project.

[0377] A "high-risk task" is one that is likely to be slow or have problems.

[0378] "Response policy" refers to a specific action plan or response measures to resolve the problem.

[0379] "Maintenance work progress" refers to the current progress of maintenance work on robots and other equipment being carried out within the factory.

[0380] "Facilitation function" refers to the function of supporting the progress of meetings and promoting smooth discussions.

[0381] "Factory current status data" refers to data related to the overall operational operations and equipment status within the factory.

[0382] MODE FOR CARRYING OUT THE INVENTION

[0383] System Overview

[0384] This invention is designed to support project management and communication within factories, with a particular focus on optimizing the role of the project manager. The system consists of a server, terminals (such as smartphones), and users (factory workers and managers). The system provides the following main functions:

[0385] Meeting progress facilitator function

[0386] The server receives audio from the meeting in real time via smartphones. This audio data is converted into text using a speech recognition API (e.g., Google Cloud Speech-to-Text). Next, when a certain period of silence (e.g., 10 seconds) is detected, the server uses a generative AI model (e.g., GPT-3) to generate the next agenda item and questions, and notifies the participants' smartphones. This ensures the meeting proceeds smoothly and prevents delays in work.

[0387] Specific examples

[0388] An example of a prompt generated by the server when silence occurs:

[0389] "Let's move on to the next topic. Let's talk about the progress on robot maintenance."

[0390] Automatic maintenance log summary function

[0391] When a user ends a meeting, their smartphone sends the audio recorded during the meeting to a server. The server converts the audio into text and sends the text to a summarization algorithm (e.g., BERT) to extract important topics and action items. Based on the extracted information, meeting minutes and action lists are automatically generated and sent to the participants' smartphones.

[0392] Specific examples

[0393] An example of a prompt generated by the server after a conference ends:

[0394] "Automatically generate minutes for maintenance meetings."

[0395] Maintenance work reminder function

[0396] The server analyzes each user's calendar data to understand their schedule, finds out when the user is not busy, and sends progress report reminders at the appropriate times. These reminders are displayed on the user's smartphone.

[0397] Specific examples

[0398] Example of a prompt when sending a reminder:

[0399] "Don't forget to perform maintenance on your robot this week."

[0400] In-factory risk management function

[0401] The server collects current status data from within the factory daily and evaluates the progress of each task. It identifies high-risk tasks and tasks that are expected to be delayed, and generates countermeasures for them. The generated countermeasures are then sent to the manager's smartphone.

[0402] Specific examples

[0403] An example of a prompt generated by the server in case of an increased risk:

[0404] "Progress on Task A is behind schedule, so please consider adding more resources or extending the deadline."

[0405] Hardware and software used

[0406] The main hardware and software used to implement this system are as follows:

[0407] Smartphones (Apple iPhone, Android devices, etc.)

[0408] Server (Cloud-based server)

[0409] Speech Recognition API (Google Cloud Speech-to-Text)

[0410] Natural language processing engine (GPT-3)

[0411] Summarization Algorithm (BERT)

[0412] Calendar API (Google Calendar)

[0413] The combination of these elements can significantly improve project management and communication within the factory, making it more efficient and effective.

[0414] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0415] Program processing flow

[0416] Step 1:

[0417] The server receives the audio data of the meeting in real time via smartphones. The input is the audio data spoken during the meeting, and the output is that audio data. The server sends this audio data to a speech recognition API (Google Cloud Speech-to-Text) and converts it into text.

[0418] Step 2:

[0419] The terminal receives text data sent from the server. The input is converted text data, and the output is text data for necessary analysis. The server analyzes this text data and detects when the conversation has stopped for a certain period of time.

[0420] Step 3:

[0421] When the server detects silence for a certain period of time (for example, 10 seconds), it uses a generative AI model (GPT-3) to generate the next agenda item and questions to proceed with. The input is the detected silence information and the current meeting situation, and the output is text data of the generated agenda item and questions. The generated agenda item and questions are notified to the participants' smartphones.

[0422] Step 4:

[0423] When a user ends a meeting, the device sends the recorded meeting audio data to the server. The input is the recording data during the meeting, and the output is the audio data. The server converts the audio back into text and sends the text data to a summarization algorithm (BERT) to extract important topics and action items.

[0424] Step 5:

[0425] The server automatically generates meeting minutes and action lists based on the extracted important agenda items and action items. The input is the extracted agenda items and action items, and the output is text data of the meeting minutes and action list summarizing the entire meeting. The generated meeting minutes and action list are sent to the participants' smartphones.

[0426] Step 6:

[0427] The server analyzes each user's calendar data. The input is the user's calendar data, and the output is the analysis result. The server finds out when the user is not busy and generates progress report reminders.

[0428] Step 7:

[0429] The server sends the generated reminder to the user's smartphone at the appropriate time. The input is the generated reminder, and the output is the notification message sent to the user.

[0430] Step 8:

[0431] The server collects current status data from within the factory daily and evaluates the progress of tasks. The input is various data from within the factory (sensor information, work logs, etc.), and the output is the evaluation result of the progress of each task.

[0432] Step 9:

[0433] Based on the progress evaluation results, the server identifies high-risk tasks and tasks showing signs of delay and generates a response policy. The input is the evaluation results, and the output is text data containing specific response policies. The generated response policy is notified to the administrator's terminal.

[0434] Through these steps, this system can reduce the burden on project managers and effectively support project management and communication within the factory.

[0435] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0436] The present invention is a system for supporting the role of a project manager (PM) in a system development project, and in particular incorporates a user emotion recognition function. The system utilizes a large-scale language model and an emotion engine to provide multiple functions that optimize the PM's communication and project management tasks. The following describes an embodiment of the system.

[0437] Overall system overview

[0438] This system consists of a server, terminals, and users. The server performs the central processing, and the terminals provide the interface with users. Users are team members and PMs involved in the project. The emotion engine has the function of analyzing user emotions in real time from conference audio data.

[0439] Facilitation function

[0440] When a user starts a meeting, the device collects the meeting's audio data and sends it to the server in real time. The server receives the audio data and converts it into text using natural language processing technology. An emotion engine then analyzes the user's emotions from the audio data and adjusts the facilitation content based on their emotional state. When the server detects silence for a certain period of time (for example, 10 seconds), it generates the next agenda item and questions to be asked and notifies the participants' devices.

[0441] Examples:

[0442] If there is a 10-second silence during a meeting and the emotion engine detects the user's impatience, the server will generate the message "Let's take a short break. Next, let's talk about budget review," and display it on the participant's device.

[0443] Minutes function

[0444] When a user ends a meeting, the device sends the audio data recorded during the meeting to the server. The server converts the audio into text and sends the text to a summarization algorithm to extract important topics and action items. The emotion engine also records the user's emotional state during the meeting and reflects this information in the minutes. The server then automatically generates the final minutes and action list and sends them to the participants' devices.

[0445] Examples:

[0446] After the meeting ends, the server generates minutes containing "information about the main agenda items and action items, as well as the user's emotional state (e.g., anxiety or relief)" and sends them to all participants' devices.

[0447] Communication Features

[0448] The server periodically collects and analyzes the user's calendar data. It generates progress report reminders at optimal times, taking into account the user's emotional state. The server then sends the reminders to the device, which then notifies the user.

[0449] Examples:

[0450] The server sends a reminder to the user during a non-busy time and when the user's emotional state is calm, saying, "Please report on the progress of this week's tasks."

[0451] PMO Functions

[0452] The server extracts and evaluates task data from chat tools and work breakdown structures (WBS). It also evaluates the user's emotional state using an emotion engine to identify important tasks. The server notifies the administrator's device of identified important tasks and manages the progress of the tasks. It also periodically checks the progress of tasks and sends reminders as necessary.

[0453] Examples:

[0454] The server extracts important tasks such as "preparing for weekly review" and "reporting to client" and notifies the PM's terminal after taking into consideration the user's emotional state.

[0455] Risk Management Function

[0456] The server collects project status data daily and evaluates the progress of each task. The emotion engine performs risk assessments, taking into account the user's emotional state, and identifies high-risk tasks and tasks that are expected to be delayed. The server then generates a response policy for those risks and notifies the administrator's terminal.

[0457] Examples:

[0458] The server presents a specific response policy, such as "Progress on Task A is behind schedule and the user is showing signs of impatience, so please consider adding resources or extending the deadline," and notifies the administrator's terminal.

[0459] With the above functions, this system can significantly reduce the burden on project managers and further increase the success rate of projects by taking users' emotions into consideration.

[0460] The processing flow will be explained below.

[0461] (Facilitate function)

[0462] Step 1:

[0463] The terminal uses the microphone of the terminal to collect voice data during the conference and transmits it to the server in real time.

[0464] Step 2:

[0465] The server converts the received voice data into text using natural language processing technology.

[0466] Step 3:

[0467] The server analyzes the text data and determines whether a certain period of silence (for example, 10 seconds) is detected.

[0468] Step 4:

[0469] At the same time, the server uses an emotion engine to analyze the user's emotions from the voice data in real time.

[0470] Step 5:

[0471] The server adjusts and generates next steps and questions when silence is detected and the user's emotions indicate impatience or anxiety.

[0472] Step 6:

[0473] The server transmits the generated agenda and questions to the terminal.

[0474] Step 7:

[0475] The terminal displays the received agenda and questions to the user.

[0476] (Meeting minutes function)

[0477] Step 1:

[0478] The terminal records audio data during the conference and transmits the data to the server after the conference ends.

[0479] Step 2:

[0480] The server converts the received voice data into text using natural language processing technology.

[0481] Step 3:

[0482] The server then feeds the generated text data into a summarization algorithm to extract important topics and action items.The server also analyzes the user's emotional state recorded during the meeting using an emotion engine.

[0483] Step 4:

[0484] The server automatically generates meeting minutes and action lists that include the extracted important agenda items and action items, as well as the user's emotional state.

[0485] Step 5:

[0486] The server transmits the generated minutes and action list to the terminal.

[0487] Step 6:

[0488] The terminal displays the received minutes and action list to the user.

[0489] (Communication function)

[0490] Step 1:

[0491] The server periodically collects and analyzes the user's calendar data.

[0492] Step 2:

[0493] The server uses an emotion engine to analyze the user's emotional state and generates optimally timed progress report reminders based on calendar data.

[0494] Step 3:

[0495] The server transmits the generated reminder to the terminal.

[0496] Step 4:

[0497] The terminal notifies the user of the received reminder.

[0498] (PMO function)

[0499] Step 1:

[0500] The server automatically extracts task data from chat tools and work breakdown structures (WBS) on a regular basis.

[0501] Step 2:

[0502] The server evaluates the extracted task data and identifies important tasks.

[0503] Step 3:

[0504] The server uses an emotion engine to analyze the emotional state of users involved in tasks and adjusts task priorities and allocations accordingly.

[0505] Step 4:

[0506] The server notifies the administrator terminal of important tasks and the results of their adjustments.

[0507] Step 5:

[0508] The server periodically monitors the progress of the task and sends reminders to the administrator terminal as needed.

[0509] (Risk management function)

[0510] Step 1:

[0511] The server collects project status data daily.

[0512] Step 2:

[0513] The server evaluates tasks based on the collected data and identifies high-risk tasks.

[0514] Step 3:

[0515] The server uses an emotion engine to make a risk assessment that also takes into account the user's emotional state.

[0516] Step 4:

[0517] The server generates a response policy for high-risk tasks.

[0518] Step 5:

[0519] The server notifies the administrator terminal of the generated response policy.

[0520] Step 6:

[0521] The terminal displays the received response policy to the administrator.

[0522] Example 2

[0523] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0524] Conventional project management systems were unable to take into account the user's emotional state when managing meetings or tasks, resulting in frequent stalled conversations and unclear task priorities, which could hinder smooth project progress. In terms of risk management, it was also difficult to grasp the user's emotional state and take appropriate action. As a result, project progress was often delayed and friction in communication occurred.

[0525] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for receiving conference audio data in real time and converting the audio content into text using natural language processing technology; means for analyzing user emotions from the converted text data and audio data; means for adjusting facilitation content based on the analyzed emotional data, and when it detects that the conversation has been interrupted for a certain period of time, generating the next agenda item and questions to proceed with and notifying the participant terminals; means for recording the conference audio, converting it into text in real time, sending it to a summarization algorithm to extract important agenda items and action items, automatically generating minutes and action lists including the user's emotional state, and sending them to the participant terminals; means for analyzing calendar data to grasp the schedule and emotional state of each member and sending progress report reminders at the optimal time; means for automatically extracting task data from a chat tool or work breakdown structure, and notifying the administrator terminal of important tasks while also including the user's emotional state in the evaluation; and means for collecting current project status data daily, evaluating each task, generating response policies for high-risk tasks taking the user's emotional state into consideration, and notifying the administrator terminal. This allows project management to take into account the user's emotional state, resulting in smoother meetings, clearer task priorities, and better risk management.

[0526] "Conference voice data" refers to data that digitally represents the voices of speakers collected during a conference.

[0527] "Natural language processing technology" is a computer technology for analyzing voice or text data and understanding its meaning.

[0528] A "means for converting audio content to text" is a method or device for analyzing audio data and converting the content into corresponding text data.

[0529] "Converted text data" refers to data in which voice data is converted into text information using natural language processing technology.

[0530] The "means for analyzing user emotions" refers to a technique or device for analyzing a user's emotional state (for example, joy, anger, sadness) from text data or voice data.

[0531] "Emotion data" is information that indicates the emotional state of the user.

[0532] "Means for adjusting facilitation content" refers to techniques or methods for optimizing the progress of a meeting and the content of the agenda based on emotional data.

[0533] A "means for detecting a pause in conversation" is a technology or device for detecting silence or a pause in conversation for a certain period of time or more.

[0534] A "participant terminal" is a device (for example, a PC, a smartphone, or a tablet) used by a user participating in a conference.

[0535] The "means for generating next topics and questions" is a technology or device for automatically generating next topics and questions based on the current conversation situation.

[0536] A "real-time text conversion means" is a technique or method for collecting voice data and simultaneously converting it into text form.

[0537] A "summarization algorithm" is a technology that automatically extracts important information from long text data and summarizes it in a concise manner.

[0538] A "means for extracting agenda items and action items" is a technique or method for identifying important agenda items and upcoming actions from text data.

[0539] "Means for automatically generating meeting minutes and action lists" refers to a technology or method for automatically creating meeting minutes and action lists based on agendas and action items.

[0540] "Calendar data" is data that includes the user's schedule information.

[0541] The "means for sending a progress report reminder at an optimal time" is a method or technology for sending a notification prompting a progress report at an appropriate time, taking into consideration the user's schedule and emotional state.

[0542] A "chat tool" is software or related services that support communication between project members.

[0543] A "work breakdown structure" is a framework or method for breaking down project tasks into detailed sections and hierarchically organizing them.

[0544] The "means for automatically extracting task data" is a technology or method for automatically acquiring information about tasks from a chat tool or a work breakdown structure.

[0545] "Current status data" is data that indicates the progress and status of a project.

[0546] A "daily collection means" is a technique or method for collecting data on a regular daily basis.

[0547] The "means for generating a response policy for a high-risk task" is a technique or method for formulating specific response measures or action plans for a high-risk task.

[0548] The present invention is a system for supporting the role of a project manager (PM) in a system development project, and in particular incorporates a function for recognizing user emotions. Specific embodiments for carrying out the present invention will be described below.

[0549] Hardware and Software Configuration

[0550] This system consists of a server, terminals, and users. The server performs the central processing, and the terminals provide an interface with users. Users are team members and PMs involved in the project.

[0551] Hardware used

[0552] Server: A high-performance server system (e.g., cloud service) that processes and stores data.

[0553] Terminal: A device that provides an interface with the user (e.g., PC, smartphone, tablet)

[0554] Software used

[0555] Natural language processing technology: Libraries that convert voice data into text (e.g., Google Cloud Speech-to-Text)

[0556] Emotion engine: A library that analyzes user emotions from voice data (e.g., Microsoft Azure Emotion API)

[0557] Summarization algorithms: techniques for extracting important information from meeting texts (e.g., BERT)

[0558] Chat tool: Software for managing communication within a project (e.g., Slack)

[0559] Work Breakdown Structure (WBS): A framework for breaking down and managing tasks into detailed sections

[0560] Overall system overview

[0561] The server receives the conference audio data in real time and converts the audio content into text using natural language processing technology. It then analyzes the user's emotions from the converted text and audio data. The analyzed emotional data is used to generate the next agenda item or question to move on to if the conversation stops. The generated agenda item or question is then sent to the participants' devices.

[0562] The server also records the audio of the meeting and converts it into text in real time. The text data is sent to a summarization algorithm to extract important topics and action items. The user's emotional state is also reflected in the minutes, and the final minutes and action list are automatically generated and sent to the participants' devices.

[0563] The server analyzes calendar data and understands each member's schedule and emotional state to send progress report reminders at the optimal time. Furthermore, it automatically extracts task data from chat tools and work breakdown structures, and notifies the administrator's device of important tasks, including the user's emotional state in its evaluation.

[0564] The server also collects project status data daily and evaluates the progress of each task. For high-risk tasks, a response policy is generated that takes into account the user's emotional state and is notified to the administrator's terminal.

[0565] Specific examples

[0566] Facilitation function

[0567] Examples:

[0568] If there is a 10-second silence during a meeting and the emotion engine detects the user's impatience, the server will generate the message "Let's take a short break. Next, let's talk about budget review," and display it on the participant's device.

[0569] Minutes function

[0570] Examples:

[0571] After the meeting ends, the server generates minutes containing "information about the main agenda items and action items, as well as the user's emotional state (e.g., anxiety or relief)" and sends them to all participants' devices.

[0572] Communication Features

[0573] Examples:

[0574] The server sends a reminder to the user during a non-busy time and when the user's emotional state is calm, saying, "Please report on the progress of this week's tasks."

[0575] PMO Functions

[0576] Examples:

[0577] The server extracts important tasks such as "preparing for weekly review" and "reporting to client" and notifies the PM's terminal after taking into consideration the user's emotional state.

[0578] Risk Management Function

[0579] Examples:

[0580] The server presents a specific response policy, such as "Progress on Task A is behind schedule and the user is showing signs of impatience, so please consider adding resources or extending the deadline," and notifies the administrator's terminal.

[0581] Example of input prompt for generative AI model

[0582] "When a meeting reaches a silence of more than 10 seconds, generate appropriate facilitation suggestions taking into account the user's emotions. If impatience is detected, suggest a response such as, 'Let's take a short break. Let's talk about the budget review next.'"

[0583] This system significantly reduces the burden on project managers and increases the success rate of projects.

[0584] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0585] Program processing steps

[0586] Step 1:

[0587] A user starts a conference. The terminal collects the user's voice in real time and generates voice data. This voice data becomes input to the server.

[0588] Step 2:

[0589] The device transmits the generated voice data to a server in real time, and the server receives the voice data and passes it as input data to natural language processing technology.

[0590] Step 3:

[0591] The server uses natural language processing techniques to convert the received voice data into text data, which then becomes the input for the next processing step.

[0592] Step 4:

[0593] The server inputs the converted text and voice data into an emotion engine to analyze the user's emotions. The resulting emotion data is output and used for further processing.

[0594] Step 5:

[0595] The server adjusts the facilitation content based on the emotional data. The server starts a timer to detect silence for a certain period of time (for example, 10 seconds). This timer is used as input, and the next step is executed only if silence is detected.

[0596] Step 6:

[0597] When silence is detected, the server generates the next agenda and questions based on the results of the sentiment analysis. These generated agenda and questions are the output and are sent to the participants' devices.

[0598] Step 7:

[0599] The device notifies the user of the agenda or questions it receives. For example, during a meeting, it displays a message saying, "Let's take a short break. Next, let's discuss the budget review."

[0600] Step 8:

[0601] The user ends the conference. The terminal sends all audio data recorded during the conference to the server. This audio data becomes the input for the next processing step.

[0602] Step 9:

[0603] The server converts the received audio data back into text data, which is then used as input for the summarization algorithm.

[0604] Step 10:

[0605] The server feeds the text data into a summarization algorithm to extract key topics and action items, and this extracted data is the output.

[0606] Step 11:

[0607] The server automatically generates meeting minutes and action lists based on the extracted important topics and action items, as well as the user's emotional data. These minutes and action lists are the output and are sent to the participants' devices.

[0608] Step 12:

[0609] The device notifies the user of the minutes and action list it has received. For example, after the meeting, the user can view the minutes, including the main agenda items, action items, and the user's emotional state.

[0610] Step 13:

[0611] The server periodically collects the user's calendar data, which serves as input for the next processing step.

[0612] Step 14:

[0613] The server analyzes the calendar data to determine the user's schedule, and the resulting analysis data becomes the input for the next processing step.

[0614] Step 15:

[0615] The server uses an emotion engine to check the user's emotional state. Based on the emotional data and calendar data, it generates a progress report reminder at the optimal time. This reminder is the output.

[0616] Step 16:

[0617] The server sends the generated reminder to the device, and the device notifies the user of the reminder, for example, by displaying a message such as "Please report on the progress of this week's tasks."

[0618] Step 17:

[0619] The server automatically extracts task data from chat tools and work breakdown structures (WBS), and this task data becomes the input.

[0620] Step 18:

[0621] The server evaluates the extracted task data and the user's emotional state using an emotion engine to identify important tasks. The identified important tasks are the output.

[0622] Step 19:

[0623] The server notifies the administrator of identified important tasks, for example by displaying messages such as "Prepare for weekly review" and "Report to client."

[0624] Step 20:

[0625] The server collects project status data daily, and this status data is used as input.

[0626] Step 21:

[0627] The server evaluates the progress of each task and takes into account the user's emotional state using an emotion engine, thereby identifying high-risk tasks. The identified high-risk tasks are the output.

[0628] Step 22:

[0629] The server generates a response policy for high-risk tasks and notifies the administrator's terminal, for example, by displaying a message such as "Progress on Task A is behind schedule, so please consider adding resources or extending the deadline."

[0630] (Application example 2)

[0631] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0632] Modern factories require efficient work management and communication, but achieving this requires advanced project management and team motivation management. In particular, there is no system in place that utilizes emotion recognition technology to provide support according to the emotional state of workers. Conventional management systems are unable to adjust the timing of appropriate advice or reminders that take into account the emotions of workers. This can lead to stress building up and reduced work efficiency.

[0633] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0634] In this invention, the server has a means for receiving conference audio data in real time and converting the audio content into text using natural language processing technology, a means for analyzing the converted text data and detecting when the conversation has stopped for a certain period of time, a means for generating the next agenda item and questions to be asked when this is detected and notifying the participant terminals, a means for recording the conference audio, converting it into text in real time, sending it to a summarization algorithm to extract important agenda items and action items, automatically generating minutes and action lists and sending them to the participant terminals, and a means for analyzing calendar data to grasp the schedule of each member and select the optimal time. The system includes a means for sending progress report reminders at the most appropriate timing, a means for automatically extracting task data from chat tools and work breakdown structures and notifying a manager's terminal of important tasks, a means for collecting current project status data daily, evaluating each task, generating a response policy for high-risk tasks and notifying the manager's terminal, a means for supporting in-factory worker meetings and providing advice based on the worker's emotional state, a means for automatically generating memos summarizing work content and important points after a shift, and a means for reminding workers of work progress and important contact information at the most appropriate timing based on the worker's emotional state. This enables efficient communication and project management based on the worker's emotional state.

[0635] - "Conference audio data" refers to audio information spoken during a conference, and is data collected in digital form.

[0636] "Natural language processing technology" is a computer science technology for understanding and analyzing the content of speech and text.

[0637] "Text" refers to information in the form of a written language, which is audio data converted into written information.

[0638] A "certain period of time" refers to a specific continuous period of time, such as 10 seconds.

[0639] An "agenda" refers to a specific topic or issue that will be discussed or considered at a conference or meeting.

[0640] A "question" refers to an inquiry or question posed to another person in order to obtain information.

[0641] "Recording" refers to the act of recording sound as digital data.

[0642] A "summarization algorithm" is a computational method for extracting the main content from text data and summarizing it concisely.

[0643] An "agenda" refers to a specific issue or topic that will be the subject of a meeting or discussion.

[0644] An "action item" refers to an item based on an agenda that requires specific action or measures.

[0645] A "minutes" is a document that records the contents of discussions at a conference or meeting.

[0646] "Calendar Data" refers to data containing information about appointments and schedules.

[0647] "Progress report" refers to a report on the progress of a project or task.

[0648] A "reminder" is a notification or alert that reminds you of important appointments or tasks.

[0649] A "chat tool" refers to a tool used for online text and voice communication.

[0650] "Work breakdown structure" refers to a method of breaking down projects and tasks into smaller parts and managing them in a hierarchical structure.

[0651] "Task data" is data that contains information about a particular task or job.

[0652] "Administrator terminal" refers to a computer or device used by a system administrator.

[0653] "Current status data" refers to data that indicates the current status of a project or task.

[0654] A "high-risk task" is one that has a high chance of being delayed or failing.

[0655] "Response policy" refers to solutions or measures to address specific problems or risks.

[0656] "Factory workers" refers to all employees working in a factory.

[0657] "Emotional state" refers to the psychological state or mood of an individual worker.

[0658] "Advice" refers to advice or suggestions given in response to a particular problem or situation.

[0659] "Post-shift" refers to the time after a worker's work shift ends.

[0660] "Work content" refers to the specific activities and actions associated with a particular job or task.

[0661] "Points" refers to important points or main points.

[0662] The "optimal timing" refers to the most effective and appropriate time.

[0663] "Announcements" refer to important information or messages that need to be shared.

[0664] The system for implementing this invention includes multiple means for providing project management and support based on the emotional state of the worker. Specifically, the system uses the following hardware and software to process and calculate data.

[0665] The core hardware of the system consists of a server, a user device (PC, smartphone, tablet, etc.), and a microphone for voice input. The server performs the core processing, and the device provides the interface with the user.

[0666] 1. Real-time processing of conference audio data

[0667] The server converts the meeting audio data into text in real time using a speech recognition tool (e.g., Google Speech-to-Text API) and a natural language processing library (e.g., NLTK or spaCy). The converted text data is analyzed to detect silences over a certain period of time (e.g., 10 seconds).

[0668] 2. Question and agenda generation

[0669] The server uses a large-scale language model (e.g., GPT-3) to generate the next agenda item or question to be asked when silence occurs, and this generated content is notified to the participants' devices.

[0670] 3. Automatic generation of meeting minutes

[0671] The server records the audio of the meeting and transcribes it into text in real time. It then uses a summarization algorithm (such as TextRank) to extract important topics and action items, automatically generating minutes and action lists. These minutes are then sent to the participants' devices.

[0672] 4. Analyzing calendar data and sending reminders

[0673] The server analyzes the user's calendar data to understand each member's schedule, evaluates their emotional state using an emotion engine (e.g., IBM Watson Tone Analyzer), and sends progress report reminders at the optimal time.

[0674] 5. Automatic extraction and management of task data

[0675] The server automatically extracts task data from chat tools (e.g., Slack) and work breakdown structures (WBS) to identify important tasks. These important tasks are notified to the administrator's terminal. It also collects current project status data daily and evaluates each task. It generates response policies for high-risk tasks and notifies the administrator's terminal.

[0676] 6. In-factory support

[0677] It also includes a function to support meetings between workers in a factory and provide advice based on their emotional state. For example, if a worker is feeling impatient, advice such as "Take a short break. Relax before proceeding to the next task" is generated and displayed on the device.

[0678] 7. Automatic generation of work notes

[0679] After each shift, a memo summarizing the work and key points is automatically generated and sent to the worker's device, including key tasks and points requiring attention.

[0680] Examples and prompts

[0681] As a specific example of usage, to give voice instructions at the end of a shift in a factory:

[0682] "The work is progressing slowly today, and everyone seems to be getting impatient."

[0683] In this way, the system supports workers through emotion recognition, enabling efficient project management and communication.

[0684] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0685] Step 1:

[0686] The server receives the conference audio data in real time and converts the audio content into text using natural language processing technology. Specifically, it uses a speech recognition tool (such as the Google Speech-to-Text API) to convert the conference audio into text data. The input of this step is the conference audio data, and the output is the converted text data.

[0687] Step 2:

[0688] The server analyzes the converted text data and detects silences over a certain period of time (e.g., 10 seconds). It analyzes the text using a natural language processing library (e.g., NLTK or spaCy). The input of this step is the text data, and the output is the silence detection result.

[0689] Step 3:

[0690] When the server detects silence, it generates the next agenda and question to be asked and notifies the participant's device. It uses a large-scale language model (such as GPT-3) to generate appropriate agenda and questions. The input of this step is the silence detection result, and the output is the generated agenda and question.

[0691] Step 4:

[0692] The server records the audio of the conference and converts it into text in real time. The recorded audio data is converted into text data. The input of this step is the conference audio data, and the output is the converted audio data.

[0693] Step 5:

[0694] The server sends the text data to a summarization algorithm to extract important topics and action items. We use a summarization algorithm such as TextRank. The input of this step is the text data, and the output is summarized topics and action items.

[0695] Step 6:

[0696] The server automatically generates minutes and action lists based on the summarized data and sends them to the participants' terminals. The input of this step is the summarized agenda and action items, and the output is the minutes and action lists.

[0697] Step 7:

[0698] The server analyzes the user's calendar data to understand each member's schedule. It uses an emotion engine (such as IBM Watson Tone Analyzer) to evaluate the member's emotional state and sends a progress report reminder at the optimal time. The input of this step is the calendar data and emotion data, and the output is a reminder at the optimal time.

[0699] Step 8:

[0700] The server automatically extracts task data from chat tools and work breakdown structures and notifies the manager's device of important tasks. It uses chat tools (such as Slack) and WBS. The input of this step is task data, and the output is a notification of important tasks.

[0701] Step 9:

[0702] The server collects project status data daily and evaluates each task. It generates a response policy for high-risk tasks and notifies the administrator's terminal. The input of this step is the status data, and the output is the risk response policy.

[0703] Step 10:

[0704] The server supports meetings between workers in the factory and provides advice based on their emotional state. It combines speech recognition and emotion analysis models to generate appropriate advice. The input of this step is the worker's emotional data, and the output is an advice message.

[0705] Step 11:

[0706] After each shift, the server automatically generates a memo summarizing the work content and important points and sends it to the worker's device. The input for this step is the shift data and work content data, and the output is a summary memo.

[0707] Step 12:

[0708] The server reminds users about work progress and important communication matters at the optimal timing based on their emotional state. The appropriate timing is determined using an emotion engine. The inputs of this step are work progress data and emotional data, and the output is a reminder notification.

[0709] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0710] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0711] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0712] [Second embodiment]

[0713] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0714] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0715] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0716] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0717] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0718] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0719] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0720] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0721] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0722] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0723] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0724] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0725] The present invention is a system designed to support the role of a project manager (PM) in a system development project. The system uses a large-scale language model to provide multiple functions that optimize the PM's communication and project management tasks. The following describes an embodiment of the system.

[0726] Overall system overview

[0727] This system consists of a server, terminals, and users. The server performs the central processing, and the terminals provide an interface with users. Users are team members and PMs involved in the project.

[0728] Facilitation function

[0729] When a user starts a meeting, the device collects the meeting's audio data and sends it to the server in real time. The server receives the audio data and converts it into text using natural language processing technology. When the server detects silence for a certain period of time (for example, 10 seconds), it generates the next agenda item and questions to be asked and notifies the participants' devices. This allows the meeting to proceed smoothly and important topics to be discussed.

[0730] Examples:

[0731] If 10 seconds of silence occurs during a meeting, the server generates the message "Let's move on to the next agenda item. Next, let's talk about the budget review." and displays it on the participants' devices.

[0732] Minutes function

[0733] When the user ends the meeting, the device sends the audio recorded during the meeting to the server. The server converts the audio into text and sends the text to a summarization algorithm to extract important topics and action items. Based on the extracted information, minutes and action lists are automatically generated and sent to the participants' devices.

[0734] Examples:

[0735] After the meeting, the server generates minutes that include information such as "discussion on the timing of releasing new features" and "tasks for each member," and sends them to the devices of all participants.

[0736] Communication Features

[0737] The server analyzes each user's calendar data to understand their schedules, finds out when the user is not busy, and sends progress report reminders at the appropriate times. These reminders are displayed on the device.

[0738] Examples:

[0739] The server will send a reminder at 10:00 AM on the following Monday, such as "Please report on the progress of this week's tasks."

[0740] PMO Functions

[0741] Task data is extracted from chat tools and work breakdown structures (WBS) and sent to the server. The server evaluates the extracted task data and identifies important tasks. It then notifies the PM's device of important tasks and manages the task progress. Task progress is periodically checked and reminders are sent as necessary.

[0742] Examples:

[0743] The server extracts important tasks such as "preparing for weekly review" and "reporting to client" and notifies the PM's terminal.

[0744] Risk Management Function

[0745] The server collects project status data daily and evaluates the progress of each task. It identifies high-risk tasks and tasks that are expected to be delayed, and generates a response plan for them. The generated response plan is then notified to the administrator's terminal.

[0746] Examples:

[0747] The server presents specific response guidelines, such as "Progress on Task A is behind schedule, so please consider adding resources or extending the delivery date," and notifies the PM's terminal.

[0748] With the above functions, this system can significantly reduce the burden on project managers and increase the success rate of projects.

[0749] The processing flow will be explained below.

[0750] (Facilitate function)

[0751] Step 1:

[0752] The terminal uses the microphone of the terminal to collect voice data during the conference and transmits it to the server in real time.

[0753] Step 2:

[0754] The server converts the received voice data into text using natural language processing technology.

[0755] Step 3:

[0756] The server analyzes the text data and determines whether a certain period of silence (for example, 10 seconds) is detected.

[0757] Step 4:

[0758] If silence is detected, the server generates the next topic to be discussed and appropriate questions.

[0759] Step 5:

[0760] The server transmits the generated agenda and questions to the terminal.

[0761] Step 6:

[0762] The terminal displays the received agenda and questions to the user.

[0763] (Meeting minutes function)

[0764] Step 1:

[0765] The terminal records audio data during the conference and transmits the data to the server after the conference ends.

[0766] Step 2:

[0767] The server converts the received voice data into text using natural language processing technology.

[0768] Step 3:

[0769] The server feeds the generated text data into a summarization algorithm to extract important topics and action items.

[0770] Step 4:

[0771] The server automatically generates minutes and action lists based on the extracted information.

[0772] Step 5:

[0773] The server transmits the generated minutes and action list to the terminal.

[0774] Step 6:

[0775] The terminal displays the received minutes and action list to the user.

[0776] (Communication function)

[0777] Step 1:

[0778] The server periodically collects and analyzes the user's calendar data.

[0779] Step 2:

[0780] The server takes into account the user's schedule and generates progress report reminders at optimal times.

[0781] Step 3:

[0782] The server transmits the generated reminder to the terminal.

[0783] Step 4:

[0784] The terminal notifies the user of the received reminder.

[0785] (PMO function)

[0786] Step 1:

[0787] The server automatically extracts task data from chat tools and work breakdown structures (WBS) on a regular basis.

[0788] Step 2:

[0789] The server evaluates the extracted task data and identifies important tasks.

[0790] Step 3:

[0791] The server notifies the administrator terminal of the identified important task.

[0792] Step 4:

[0793] The server periodically monitors the progress of the task and sends reminders to the administrator terminal as needed.

[0794] (Risk management function)

[0795] Step 1:

[0796] The server collects project status data daily.

[0797] Step 2:

[0798] The server evaluates tasks based on the collected data and identifies high-risk tasks.

[0799] Step 3:

[0800] The server generates a response policy for the identified risks.

[0801] Step 4:

[0802] The server notifies the administrator terminal of the generated response policy.

[0803] Step 5:

[0804] The terminal displays the received response policy to the administrator.

[0805] Example 1

[0806] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0807] Conventional systems require project managers (PMs) to manually manage a wide range of tasks, conduct meetings, create minutes, and monitor the progress of each member, requiring a great deal of time and effort. This can slow down project progress and delay the early detection and response of risks. Other issues include overlooking important agenda items during meetings and insufficient follow-up afterward. There is a need for a system that can solve these problems, reduce the burden on PMs, and improve the success rate of projects.

[0808] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0809] In this invention, the server includes: means for receiving conference audio data in real time and converting the audio content into text using natural language processing technology; means for analyzing the converted text data and detecting when the conversation has stopped for a certain period of time; means for generating next agenda items and questions and notifying participant terminals when detected; means for recording the conference audio, converting it into text in real time, sending it to a summarization algorithm to extract important agenda items and action items, automatically generating minutes and action lists and sending them to participant terminals; means for analyzing calendar data to understand each member's schedule and sending progress report reminders at optimal times; means for automatically extracting task data from a chat tool or a work breakdown structure and notifying a manager terminal of important tasks; means for collecting project status data daily, evaluating each task, generating response policies for high-risk tasks and notifying the manager terminal; and means for proposing project management actions based on the generated response policies. This automates the manual work of project managers and enables efficient project management.

[0810] "Conference audio data" refers to audio information generated during a conference, including statements and discussions.

[0811] "Natural language processing technology" is a technology that allows computers to understand and process the language that humans use on a daily basis.

[0812] "Text data" refers to text information generated from audio data using natural language processing technology.

[0813] "Conversion" is the process of switching data from one format to another, in this case changing audio data to text data.

[0814] The "certain period of time" refers to the length of time that serves as a reference in the detection process, and in this invention specifically means 10 seconds.

[0815] A "lull in conversation" refers to a state in which no one speaks for a certain period of time or longer.

[0816] An "agenda" refers to the subject or topic of a meeting or discussion.

[0817] A "question" is a question posed to obtain specific information or opinions.

[0818] "Participant terminal" refers to an electronic device such as a computer or mobile device used by a user participating in a conference.

[0819] A "summarization algorithm" is a computational procedure for extracting the important parts of long text data and presenting them in a shortened form.

[0820] "Agenda and action items" refers to the main topics covered during the meeting and any specific action items that will follow.

[0821] "Minutes" refers to a document that records the contents of a meeting.

[0822] An "action list" is a document that lists specific actions to be taken after a meeting.

[0823] "Calendar data" refers to schedule information written on an electronic calendar used by a user.

[0824] A "progress report" is a report on the progress of a project, usually done on a regular basis.

[0825] A "reminder" is information that notifies a user to remind them of a particular action or task.

[0826] A "chat tool" is software that users use to communicate instantly through text.

[0827] A work breakdown structure (WBS) is a diagram or list that breaks down project tasks hierarchically into manageable sections.

[0828] "Task data" refers to information about specific work items to be done as part of a project.

[0829] "Project status data" refers to data that indicates the current progress of a project and the status of tasks.

[0830] "Evaluation" refers to the process of analyzing data and determining its status or value based on specific criteria.

[0831] A "high-risk task" is a work item that is likely not to proceed as planned.

[0832] A "response policy" refers to a countermeasure or action plan for a specific problem or situation.

[0833] "Server" refers to a computing device that performs central processing and exchanges data with other terminals.

[0834] "Means of suggestion" refers to the method by which the system notifies the user of actions and countermeasures and supports their progress.

[0835] MODE FOR CARRYING OUT THE INVENTION

[0836] This invention is a system designed to support the role of project managers (PMs) in system development projects. This system consists of three main components: a server, a terminal, and a user, and provides multiple functions to optimize the PM's communication and project management tasks.

[0837] Hardware and software used

[0838] Server: A computer device that performs central processing and exchanges data with other devices

[0839] Terminal: An electronic device (e.g., computer, mobile device) that provides an interface with a user.

[0840] Natural language processing technology: Google Cloud Speech-to-Text API

[0841] Summarization Algorithm: BERT-Based Summarization Model

[0842] Calendar Data Analysis: Google Calendar API

[0843] Task data extraction: Slack API, JIRA API

[0844] Collecting project status data: JIRA API, GitLab API

[0845] Features and specific operation examples

[0846] Facilitation function

[0847] When a user starts a meeting, the device collects the audio data of the meeting and sends it to the server in real time. When the server receives the audio data, it converts it into text using natural language processing technology. If the server detects silence for a certain period of time (for example, 10 seconds), it generates the next agenda item and questions and notifies the participants' devices.

[0848] Example: If 10 seconds of silence occurs during a meeting, the server generates the message "Let's move on to the next agenda item. Let's discuss the budget review next." and displays it on the participants' devices.

[0849] Minutes function

[0850] When the user ends the meeting, the device sends the audio data recorded during the meeting to the server, which converts the audio data into text using natural language processing technology and sends the text data to a summarization algorithm to extract important topics and action items. Based on this, minutes and action lists are automatically generated and sent to the participants' devices.

[0851] Example: After the meeting, the server generates minutes that include "discussions about the timing of releasing new features" and "tasks for each member," and sends them to the participants' devices.

[0852] Communication Features

[0853] The server analyzes each user's calendar data to understand their schedule, finds times when the user is not busy, and generates progress report reminders at those times and sends them to each device.

[0854] Example: The server sends a reminder to participants' devices at 10:00 a.m. on the following Monday, such as "Please report on the progress of this week's tasks."

[0855] PMO Functions

[0856] Task data is extracted from chat tools and work breakdown structures (WBS) and sent to the server. The server evaluates the extracted task data and identifies important tasks. It then notifies the PM's device of important tasks and manages the task progress. Task progress is periodically checked and reminders are sent as necessary.

[0857] Example: The server extracts important tasks such as "preparing for weekly review" and "reporting to client" and notifies the PM's terminal.

[0858] Risk Management Function

[0859] The server collects project status data daily and evaluates the progress of each task. It identifies high-risk tasks and tasks that are expected to be delayed, and generates a response plan for them. The generated response plan is then notified to the administrator's terminal.

[0860] Example: The server presents a specific response policy such as "Progress on Task A is behind schedule, so please consider adding resources or extending the deadline," and notifies the PM's terminal.

[0861] Examples of prompt statements

[0862] To use this system, the following is an example of a prompt sentence to input into the generative AI model:

[0863] Meeting Facilitation Functions:

[0864] Prompt: "Detect 10 seconds of silence during a meeting. Generate next agenda items or questions."

[0865] Minutes feature:

[0866] Prompt: "Convert the audio data from a meeting recording into text and generate a summary."

[0867] Communication features:

[0868] Prompt: "Analyze the user's calendar data to find the appropriate time for task progress reports and send reminders."

[0869] PMO Functions:

[0870] Prompt: "Extract tasks from the chat tool or WBS, identify important tasks, and notify the PM."

[0871] Risk Management Functions:

[0872] Prompt: "Evaluate the progress of tasks and generate and communicate specific action plans for high-risk tasks."

[0873] In this way, by using the system of the present invention, it is possible to significantly reduce the burden on project managers and improve the success rate of projects.

[0874] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0875] Facilitation function

[0876] Processing Steps

[0877] Step 1:

[0878] A user starts a conference using a terminal. The user inputs the conference start time, participant list, etc. into the system. This information becomes the initial data for the system.

[0879] Specific behavior:

[0880] The user enters the details of the meeting on the meeting setting screen and presses the "Start" button.

[0881] Step 2:

[0882] The device collects audio data during the meeting in real time using a built-in microphone or an external microphone, and stores the collected audio data in a buffer.

[0883] Specific behavior:

[0884] The device activates the microphone and begins collecting audio.

[0885] Input: Audio during the meeting

[0886] Output: Collected audio data

[0887] Step 3:

[0888] The device transmits the collected audio data to the server in real time using a secure protocol such as HTTPS.

[0889] Specific behavior:

[0890] The terminal uploads the voice data to the server at regular intervals.

[0891] Input: Collected audio data

[0892] Output: Audio data sent to the server

[0893] Step 4:

[0894] The server converts the received voice data into text using natural language processing technology (Google Cloud Speech-to-Text API).

[0895] Specific behavior:

[0896] The server calls the API and converts the audio data into text data.

[0897] Input: Audio data sent to the server

[0898] Output: Converted text data

[0899] Step 5:

[0900] The server analyzes the text data and detects silence for a certain period of time (10 seconds).

[0901] Specific behavior:

[0902] The server analyzes the time information in the text data and flags any silence of 10 seconds or more.

[0903] Input: Converted text data

[0904] Output: Silence detection flag

[0905] Step 6:

[0906] When the server detects silence, it generates the next agenda item and question to be asked and notifies the participants' terminals.

[0907] Specific behavior:

[0908] The server uses a generative AI model to generate the next agenda item or question and create a notification message.

[0909] Input: silence detection flag

[0910] Output: Generated agenda and questions

[0911] Specific behavior:

[0912] The generated agenda and questions are displayed on the participants' devices.

[0913] Minutes function

[0914] Processing Steps

[0915] Step 1:

[0916] The user ends the conference using the terminal. The end operation is performed through the terminal interface.

[0917] Specific behavior:

[0918] The user presses the "Finish" button.

[0919] Step 2:

[0920] The terminal transmits the audio data recorded during the meeting to the server.

[0921] Specific behavior:

[0922] The device uploads the recording data to the server.

[0923] Input: Recorded audio data

[0924] Output: Audio data sent to the server

[0925] Step 3:

[0926] The server converts the voice data into text using natural language processing technology.

[0927] Specific behavior:

[0928] The server calls the API and converts the audio data into text data.

[0929] Input: Audio data sent to the server

[0930] Output: Converted text data

[0931] Step 4:

[0932] The server sends the text data to a summarization algorithm (a BERT-based summarization model) to extract important topics and action items.

[0933] Specific behavior:

[0934] The server sends the data to a summary model to extract the important information.

[0935] Input: Converted text data

[0936] Output: Extracted important topics and action items

[0937] Step 5:

[0938] The server automatically generates minutes and action lists based on the extracted information.

[0939] Specific behavior:

[0940] The server automatically generates meeting minutes and action list templates by filling them with data.

[0941] Input: Extracted important agenda items and action items

[0942] Output: Auto-generated meeting minutes and action list

[0943] Step 6:

[0944] The server sends the generated minutes and action list to the participants' terminals.

[0945] Specific behavior:

[0946] The server sends the generated document to the terminal.

[0947] Input: Auto-generated meeting minutes and action list

[0948] Output: Document sent to participant's device

[0949] Communication Features

[0950] Processing Steps

[0951] Step 1:

[0952] The server analyzes each user's calendar data and keeps track of their schedules.

[0953] Specific behavior:

[0954] The server uses the Google Calendar API to retrieve and parse the user's calendar data.

[0955] Input: User's calendar data

[0956] Output: Analysis results

[0957] Step 2:

[0958] Based on the analysis results, the server identifies times when the user is not busy.

[0959] Specific behavior:

[0960] The server extracts available time slots from the analysis results.

[0961] Input: Analysis results

[0962] Output: Available time slots

[0963] Step 3:

[0964] The server generates appropriate reminders when the user is not busy and sends them to each device.

[0965] Specific behavior:

[0966] The server uses the generative AI model to create reminders and send them to the device.

[0967] Input: Available time slots

[0968] Output: The generated reminder

[0969] Specific behavior:

[0970] The reminder will be displayed on the user's device.

[0971] PMO Functions

[0972] Processing Steps

[0973] Step 1:

[0974] Task data is extracted from chat tools and work breakdown structures and sent to a server.

[0975] Specific behavior:

[0976] The server retrieves task data using the Slack API or JIRA API.

[0977] Input: Task data generated by chat tools and WBS

[0978] Output: Extracted task data

[0979] Step 2:

[0980] The server evaluates the extracted task data and identifies important tasks.

[0981] Specific behavior:

[0982] The server analyzes the task data and ranks it based on importance.

[0983] Input: Extracted task data

[0984] Output: Identified important tasks

[0985] Step 3:

[0986] The server notifies the PM's terminal of important tasks.

[0987] Specific behavior:

[0988] The server creates a notification message and sends it to the PM's terminal.

[0989] Input: Identified critical tasks

[0990] Output: Tasks notified to the PM's terminal

[0991] Step 4:

[0992] The server periodically checks the progress of each task.

[0993] Specific behavior:

[0994] The server collects task progress data daily and evaluates the current status.

[0995] Input: Task progress data

[0996] Output: Progress evaluation results

[0997] Step 5:

[0998] The server generates reminders as needed and sends them to the PM's terminal.

[0999] Specific behavior:

[1000] The server creates a reminder and sends it to the PM's device.

[1001] Input: Progress evaluation results

[1002] Output: The generated reminder

[1003] Risk Management Function

[1004] Processing Steps

[1005] Step 1:

[1006] The server collects project status data daily.

[1007] Specific behavior:

[1008] The server retrieves the latest data using the JIRA API or GitLab API.

[1009] Input: Project status data

[1010] Output: Collected data

[1011] Step 2:

[1012] The server evaluates the progress of the task based on the collected data.

[1013] Specific behavior:

[1014] The server analyzes the collected data and evaluates the progress of each task.

[1015] Input: Collected data

[1016] Output: Progress evaluation results

[1017] Step 3:

[1018] Based on the progress, the server identifies high-risk tasks and tasks that are expected to be delayed.

[1019] Specific behavior:

[1020] The server identifies risk tasks based on the progress evaluation results.

[1021] Input: Progress evaluation results

[1022] Output: Identified risk tasks

[1023] Step 4:

[1024] Once a risk is identified, the server generates a response policy.

[1025] Specific behavior:

[1026] The server uses the generative AI model to create a specific response policy.

[1027] Input: Identified Risk Tasks

[1028] Output: Generated response policy

[1029] Step 5:

[1030] The generated response policy is notified to the administrator terminal.

[1031] Specific behavior:

[1032] The server sends the response policy as a notification message to the administrator terminal.

[1033] Input: Generated response policy

[1034] Output: Response policy notified to administrator terminal

[1035] (Application example 1)

[1036] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1037] Conventional project management systems place a heavy burden on project managers, requiring a lot of time and effort, especially for running meetings, taking minutes, managing progress, and managing risks. As a result, it is difficult to send timely reminders and manage progress, especially when it comes to robot maintenance work and task management in factories, which can lead to issues that reduce project efficiency.

[1038] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1039] In this invention, the server has means for receiving conference audio data in real time and converting the audio content into text using natural language processing technology, means for analyzing the converted text data and detecting when the conversation has stopped for a certain period of time, means for generating the next agenda item and questions to be asked when this is detected and notifying the participant terminals, means for recording the conference audio, converting it into text in real time, sending it to a summarization algorithm to extract important agenda items and action items, automatically generating minutes and action lists and sending them to the participant terminals, and means for analyzing calendar data to grasp the schedule of each member and optimally The system includes a means for automatically extracting task data from chat tools and work breakdown structures and notifying the manager's terminal of important tasks, a means for collecting current project status data daily, evaluating each task, generating response policies for high-risk tasks and notifying the manager's terminal, a means for managing the progress of maintenance work using a smartphone and sending reminders, a means for supporting the progress of meetings using a facilitation function, and a means for collecting current status data within the factory, generating response policies for high-risk work and notifying the manager's terminal. This makes it possible to smoothly progress meetings, automate the creation of meeting minutes, and send timely reminders for maintenance work, significantly reducing the burden on project managers and improving the efficiency and success rate of projects.

[1040] "Conference audio data" refers to all audio data recorded during a conference.

[1041] "Natural language processing technology" refers to technology that enables computers to understand and process the natural language that humans use on a daily basis.

[1042] "Text data" refers to data obtained by converting voice data into character information.

[1043] "Detecting a pause in conversation for a certain period of time" refers to detecting that no speech has occurred within a specified period of time.

[1044] "Generating agendas and questions" refers to automatically creating next topics and questions to help keep the meeting moving.

[1045] The term "participant terminal" refers to an electronic terminal used by a user participating in a conference.

[1046] "Key Agenda and Action Items" refers to the key issues discussed at the meeting and the action items related to those issues.

[1047] "Minutes" refers to a document summarizing the matters discussed at a meeting.

[1048] An "action list" is a list of action plans and task items decided at a meeting.

[1049] "Calendar data" refers to data that records a user's schedule and plans.

[1050] "Progress report reminder" refers to a message that notifies you not to forget to report your progress.

[1051] "Chat tool" refers to an application for text-based communication.

[1052] A "work breakdown structure" refers to a structure in which a project is divided into task units.

[1053] "Task Data" refers to information related to each task in a project.

[1054] "Administrator terminal" refers to the electronic terminal used by the project manager or administrator.

[1055] "Project Status Data" means data regarding the current progress or status of a Project.

[1056] A "high-risk task" is one that is likely to be slow or have problems.

[1057] "Response policy" refers to a specific action plan or response measures to resolve the problem.

[1058] "Maintenance work progress" refers to the current progress of maintenance work on robots and other equipment being carried out within the factory.

[1059] "Facilitation function" refers to the function of supporting the progress of meetings and promoting smooth discussions.

[1060] "Factory current status data" refers to data related to the overall operational operations and equipment status within the factory.

[1061] MODE FOR CARRYING OUT THE INVENTION

[1062] System Overview

[1063] This invention is designed to support project management and communication within factories, with a particular focus on optimizing the role of the project manager. The system consists of a server, terminals (such as smartphones), and users (factory workers and managers). The system provides the following main functions:

[1064] Meeting progress facilitator function

[1065] The server receives audio from the meeting in real time via smartphones. This audio data is converted into text using a speech recognition API (e.g., Google Cloud Speech-to-Text). Next, when a certain period of silence (e.g., 10 seconds) is detected, the server uses a generative AI model (e.g., GPT-3) to generate the next agenda item and questions, and notifies the participants' smartphones. This ensures the meeting proceeds smoothly and prevents delays in work.

[1066] Specific examples

[1067] An example of a prompt generated by the server when silence occurs:

[1068] "Let's move on to the next topic. Let's talk about the progress on robot maintenance."

[1069] Automatic maintenance log summary function

[1070] When a user ends a meeting, their smartphone sends the audio recorded during the meeting to a server. The server converts the audio into text and sends the text to a summarization algorithm (e.g., BERT) to extract important topics and action items. Based on the extracted information, meeting minutes and action lists are automatically generated and sent to the participants' smartphones.

[1071] Specific examples

[1072] An example of a prompt generated by the server after a conference ends:

[1073] "Automatically generate minutes for maintenance meetings."

[1074] Maintenance work reminder function

[1075] The server analyzes each user's calendar data to understand their schedule, finds out when the user is not busy, and sends progress report reminders at the appropriate times. These reminders are displayed on the user's smartphone.

[1076] Specific examples

[1077] Example of a prompt when sending a reminder:

[1078] "Don't forget to perform maintenance on your robot this week."

[1079] In-factory risk management function

[1080] The server collects current status data from within the factory daily and evaluates the progress of each task. It identifies high-risk tasks and tasks that are expected to be delayed, and generates countermeasures for them. The generated countermeasures are then sent to the manager's smartphone.

[1081] Specific examples

[1082] An example of a prompt generated by the server in case of an increased risk:

[1083] "Progress on Task A is behind schedule, so please consider adding more resources or extending the deadline."

[1084] Hardware and software used

[1085] The main hardware and software used to implement this system are as follows:

[1086] Smartphones (Apple iPhone, Android devices, etc.)

[1087] Server (Cloud-based server)

[1088] Speech Recognition API (Google Cloud Speech-to-Text)

[1089] Natural language processing engine (GPT-3)

[1090] Summarization Algorithm (BERT)

[1091] Calendar API (Google Calendar)

[1092] The combination of these elements can significantly improve project management and communication within the factory, making it more efficient and effective.

[1093] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1094] Program processing flow

[1095] Step 1:

[1096] The server receives the audio data of the meeting in real time via smartphones. The input is the audio data spoken during the meeting, and the output is that audio data. The server sends this audio data to a speech recognition API (Google Cloud Speech-to-Text) and converts it into text.

[1097] Step 2:

[1098] The terminal receives text data sent from the server. The input is converted text data, and the output is text data for necessary analysis. The server analyzes this text data and detects when the conversation has stopped for a certain period of time.

[1099] Step 3:

[1100] When the server detects silence for a certain period of time (for example, 10 seconds), it uses a generative AI model (GPT-3) to generate the next agenda item and questions to proceed with. The input is the detected silence information and the current meeting situation, and the output is text data of the generated agenda item and questions. The generated agenda item and questions are notified to the participants' smartphones.

[1101] Step 4:

[1102] When a user ends a meeting, the device sends the recorded meeting audio data to the server. The input is the recording data during the meeting, and the output is the audio data. The server converts the audio back into text and sends the text data to a summarization algorithm (BERT) to extract important topics and action items.

[1103] Step 5:

[1104] The server automatically generates meeting minutes and action lists based on the extracted important agenda items and action items. The input is the extracted agenda items and action items, and the output is text data of the meeting minutes and action list summarizing the entire meeting. The generated meeting minutes and action list are sent to the participants' smartphones.

[1105] Step 6:

[1106] The server analyzes each user's calendar data. The input is the user's calendar data, and the output is the analysis result. The server finds out when the user is not busy and generates progress report reminders.

[1107] Step 7:

[1108] The server sends the generated reminder to the user's smartphone at the appropriate time. The input is the generated reminder, and the output is the notification message sent to the user.

[1109] Step 8:

[1110] The server collects current status data from within the factory daily and evaluates the progress of tasks. The input is various data from within the factory (sensor information, work logs, etc.), and the output is the evaluation result of the progress of each task.

[1111] Step 9:

[1112] Based on the progress evaluation results, the server identifies high-risk tasks and tasks showing signs of delay and generates a response policy. The input is the evaluation results, and the output is text data containing specific response policies. The generated response policy is notified to the administrator's terminal.

[1113] Through these steps, this system can reduce the burden on project managers and effectively support project management and communication within the factory.

[1114] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1115] The present invention is a system for supporting the role of a project manager (PM) in a system development project, and in particular incorporates a user emotion recognition function. The system utilizes a large-scale language model and an emotion engine to provide multiple functions that optimize the PM's communication and project management tasks. The following describes an embodiment of the system.

[1116] Overall system overview

[1117] This system consists of a server, terminals, and users. The server performs the central processing, and the terminals provide the interface with users. Users are team members and PMs involved in the project. The emotion engine has the function of analyzing user emotions in real time from conference audio data.

[1118] Facilitation function

[1119] When a user starts a meeting, the device collects the meeting's audio data and sends it to the server in real time. The server receives the audio data and converts it into text using natural language processing technology. An emotion engine then analyzes the user's emotions from the audio data and adjusts the facilitation content based on their emotional state. When the server detects silence for a certain period of time (for example, 10 seconds), it generates the next agenda item and questions to be asked and notifies the participants' devices.

[1120] Examples:

[1121] If there is a 10-second silence during a meeting and the emotion engine detects the user's impatience, the server will generate the message "Let's take a short break. Next, let's talk about budget review," and display it on the participant's device.

[1122] Minutes function

[1123] When a user ends a meeting, the device sends the audio data recorded during the meeting to the server. The server converts the audio into text and sends the text to a summarization algorithm to extract important topics and action items. The emotion engine also records the user's emotional state during the meeting and reflects this information in the minutes. The server then automatically generates the final minutes and action list and sends them to the participants' devices.

[1124] Examples:

[1125] After the meeting ends, the server generates minutes containing "information about the main agenda items and action items, as well as the user's emotional state (e.g., anxiety or relief)" and sends them to all participants' devices.

[1126] Communication Features

[1127] The server periodically collects and analyzes the user's calendar data. It generates progress report reminders at optimal times, taking into account the user's emotional state. The server then sends the reminders to the device, which then notifies the user.

[1128] Examples:

[1129] The server sends a reminder to the user during a non-busy time and when the user's emotional state is calm, saying, "Please report on the progress of this week's tasks."

[1130] PMO Functions

[1131] The server extracts and evaluates task data from chat tools and work breakdown structures (WBS). It also evaluates the user's emotional state using an emotion engine to identify important tasks. The server notifies the administrator's device of identified important tasks and manages the progress of the tasks. It also periodically checks the progress of tasks and sends reminders as necessary.

[1132] Examples:

[1133] The server extracts important tasks such as "preparing for weekly review" and "reporting to client" and notifies the PM's terminal after taking into consideration the user's emotional state.

[1134] Risk Management Function

[1135] The server collects project status data daily and evaluates the progress of each task. The emotion engine performs risk assessments, taking into account the user's emotional state, and identifies high-risk tasks and tasks that are expected to be delayed. The server then generates a response policy for those risks and notifies the administrator's terminal.

[1136] Examples:

[1137] The server presents a specific response policy, such as "Progress on Task A is behind schedule and the user is showing signs of impatience, so please consider adding resources or extending the deadline," and notifies the administrator's terminal.

[1138] With the above functions, this system can significantly reduce the burden on project managers and further increase the success rate of projects by taking users' emotions into consideration.

[1139] The processing flow will be explained below.

[1140] (Facilitate function)

[1141] Step 1:

[1142] The terminal uses the microphone of the terminal to collect voice data during the conference and transmits it to the server in real time.

[1143] Step 2:

[1144] The server converts the received voice data into text using natural language processing technology.

[1145] Step 3:

[1146] The server analyzes the text data and determines whether a certain period of silence (for example, 10 seconds) is detected.

[1147] Step 4:

[1148] At the same time, the server uses an emotion engine to analyze the user's emotions from the voice data in real time.

[1149] Step 5:

[1150] The server adjusts and generates next steps and questions when silence is detected and the user's emotions indicate impatience or anxiety.

[1151] Step 6:

[1152] The server transmits the generated agenda and questions to the terminal.

[1153] Step 7:

[1154] The terminal displays the received agenda and questions to the user.

[1155] (Meeting minutes function)

[1156] Step 1:

[1157] The terminal records audio data during the conference and transmits the data to the server after the conference ends.

[1158] Step 2:

[1159] The server converts the received voice data into text using natural language processing technology.

[1160] Step 3:

[1161] The server then feeds the generated text data into a summarization algorithm to extract important topics and action items.The server also analyzes the user's emotional state recorded during the meeting using an emotion engine.

[1162] Step 4:

[1163] The server automatically generates meeting minutes and action lists that include the extracted important agenda items and action items, as well as the user's emotional state.

[1164] Step 5:

[1165] The server transmits the generated minutes and action list to the terminal.

[1166] Step 6:

[1167] The terminal displays the received minutes and action list to the user.

[1168] (Communication function)

[1169] Step 1:

[1170] The server periodically collects and analyzes the user's calendar data.

[1171] Step 2:

[1172] The server uses an emotion engine to analyze the user's emotional state and generates optimally timed progress report reminders based on calendar data.

[1173] Step 3:

[1174] The server transmits the generated reminder to the terminal.

[1175] Step 4:

[1176] The terminal notifies the user of the received reminder.

[1177] (PMO function)

[1178] Step 1:

[1179] The server automatically extracts task data from chat tools and work breakdown structures (WBS) on a regular basis.

[1180] Step 2:

[1181] The server evaluates the extracted task data and identifies important tasks.

[1182] Step 3:

[1183] The server uses an emotion engine to analyze the emotional state of users involved in tasks and adjusts task priorities and allocations accordingly.

[1184] Step 4:

[1185] The server notifies the administrator terminal of important tasks and the results of their adjustments.

[1186] Step 5:

[1187] The server periodically monitors the progress of the task and sends reminders to the administrator terminal as needed.

[1188] (Risk management function)

[1189] Step 1:

[1190] The server collects project status data daily.

[1191] Step 2:

[1192] The server evaluates tasks based on the collected data and identifies high-risk tasks.

[1193] Step 3:

[1194] The server uses an emotion engine to make a risk assessment that also takes into account the user's emotional state.

[1195] Step 4:

[1196] The server generates a response policy for high-risk tasks.

[1197] Step 5:

[1198] The server notifies the administrator terminal of the generated response policy.

[1199] Step 6:

[1200] The terminal displays the received response policy to the administrator.

[1201] Example 2

[1202] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1203] Conventional project management systems were unable to take into account the user's emotional state when managing meetings or tasks, resulting in frequent stalled conversations and unclear task priorities, which could hinder smooth project progress. In terms of risk management, it was also difficult to grasp the user's emotional state and take appropriate action. As a result, project progress was often delayed and friction in communication occurred.

[1204] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for receiving conference audio data in real time and converting the audio content into text using natural language processing technology; means for analyzing user emotions from the converted text data and audio data; means for adjusting facilitation content based on the analyzed emotional data, and when it detects that the conversation has been interrupted for a certain period of time, generating the next agenda item and questions to proceed with and notifying the participant terminals; means for recording the conference audio, converting it into text in real time, sending it to a summarization algorithm to extract important agenda items and action items, automatically generating minutes and action lists including the user's emotional state, and sending them to the participant terminals; means for analyzing calendar data to grasp the schedule and emotional state of each member and sending progress report reminders at the optimal time; means for automatically extracting task data from a chat tool or work breakdown structure, and notifying the administrator terminal of important tasks while also including the user's emotional state in the evaluation; and means for collecting current project status data daily, evaluating each task, generating response policies for high-risk tasks taking the user's emotional state into consideration, and notifying the administrator terminal. This allows project management to take into account the user's emotional state, resulting in smoother meetings, clearer task priorities, and better risk management.

[1205] "Conference voice data" refers to data that digitally represents the voices of speakers collected during a conference.

[1206] "Natural language processing technology" is a computer technology for analyzing voice or text data and understanding its meaning.

[1207] A "means for converting audio content to text" is a method or device for analyzing audio data and converting the content into corresponding text data.

[1208] "Converted text data" refers to data in which voice data is converted into text information using natural language processing technology.

[1209] The "means for analyzing user emotions" refers to a technique or device for analyzing a user's emotional state (for example, joy, anger, sadness) from text data or voice data.

[1210] "Emotion data" is information that indicates the emotional state of the user.

[1211] "Means for adjusting facilitation content" refers to techniques or methods for optimizing the progress of a meeting and the content of the agenda based on emotional data.

[1212] A "means for detecting a pause in conversation" is a technology or device for detecting silence or a pause in conversation for a certain period of time or more.

[1213] A "participant terminal" is a device (for example, a PC, a smartphone, or a tablet) used by a user participating in a conference.

[1214] The "means for generating next topics and questions" is a technology or device for automatically generating next topics and questions based on the current conversation situation.

[1215] A "real-time text conversion means" is a technique or method for collecting voice data and simultaneously converting it into text form.

[1216] A "summarization algorithm" is a technology that automatically extracts important information from long text data and summarizes it in a concise manner.

[1217] A "means for extracting agenda items and action items" is a technique or method for identifying important agenda items and upcoming actions from text data.

[1218] "Means for automatically generating meeting minutes and action lists" refers to a technology or method for automatically creating meeting minutes and action lists based on agendas and action items.

[1219] "Calendar data" is data that includes the user's schedule information.

[1220] The "means for sending a progress report reminder at an optimal time" is a method or technology for sending a notification prompting a progress report at an appropriate time, taking into consideration the user's schedule and emotional state.

[1221] A "chat tool" is software or related services that support communication between project members.

[1222] A "work breakdown structure" is a framework or method for breaking down project tasks into detailed sections and hierarchically organizing them.

[1223] The "means for automatically extracting task data" is a technology or method for automatically acquiring information about tasks from a chat tool or a work breakdown structure.

[1224] "Current status data" is data that indicates the progress and status of a project.

[1225] A "daily collection means" is a technique or method for collecting data on a regular daily basis.

[1226] The "means for generating a response policy for a high-risk task" is a technique or method for formulating specific response measures or action plans for a high-risk task.

[1227] The present invention is a system for supporting the role of a project manager (PM) in a system development project, and in particular incorporates a function for recognizing user emotions. Specific embodiments for carrying out the present invention will be described below.

[1228] Hardware and Software Configuration

[1229] This system consists of a server, terminals, and users. The server performs the central processing, and the terminals provide an interface with users. Users are team members and PMs involved in the project.

[1230] Hardware used

[1231] Server: A high-performance server system (e.g., cloud service) that processes and stores data.

[1232] Terminal: A device that provides an interface with the user (e.g., PC, smartphone, tablet)

[1233] Software used

[1234] Natural language processing technology: Libraries that convert voice data into text (e.g., Google Cloud Speech-to-Text)

[1235] Emotion engine: A library that analyzes user emotions from voice data (e.g., Microsoft Azure Emotion API)

[1236] Summarization algorithms: techniques for extracting important information from meeting texts (e.g., BERT)

[1237] Chat tool: Software for managing communication within a project (e.g., Slack)

[1238] Work Breakdown Structure (WBS): A framework for breaking down and managing tasks into detailed sections

[1239] Overall system overview

[1240] The server receives the conference audio data in real time and converts the audio content into text using natural language processing technology. It then analyzes the user's emotions from the converted text and audio data. The analyzed emotional data is used to generate the next agenda item or question to move on to if the conversation stops. The generated agenda item or question is then sent to the participants' devices.

[1241] The server also records the audio of the meeting and converts it into text in real time. The text data is sent to a summarization algorithm to extract important topics and action items. The user's emotional state is also reflected in the minutes, and the final minutes and action list are automatically generated and sent to the participants' devices.

[1242] The server analyzes calendar data and understands each member's schedule and emotional state to send progress report reminders at the optimal time. Furthermore, it automatically extracts task data from chat tools and work breakdown structures, and notifies the administrator's device of important tasks, including the user's emotional state in its evaluation.

[1243] The server also collects project status data daily and evaluates the progress of each task. For high-risk tasks, a response policy is generated that takes into account the user's emotional state and is notified to the administrator's terminal.

[1244] Specific examples

[1245] Facilitation function

[1246] Examples:

[1247] If there is a 10-second silence during a meeting and the emotion engine detects the user's impatience, the server will generate the message "Let's take a short break. Next, let's talk about budget review," and display it on the participant's device.

[1248] Minutes function

[1249] Examples:

[1250] After the meeting ends, the server generates minutes containing "information about the main agenda items and action items, as well as the user's emotional state (e.g., anxiety or relief)" and sends them to all participants' devices.

[1251] Communication Features

[1252] Examples:

[1253] The server sends a reminder to the user during a non-busy time and when the user's emotional state is calm, saying, "Please report on the progress of this week's tasks."

[1254] PMO Functions

[1255] Examples:

[1256] The server extracts important tasks such as "preparing for weekly review" and "reporting to client" and notifies the PM's terminal after taking into consideration the user's emotional state.

[1257] Risk Management Function

[1258] Examples:

[1259] The server presents a specific response policy, such as "Progress on Task A is behind schedule and the user is showing signs of impatience, so please consider adding resources or extending the deadline," and notifies the administrator's terminal.

[1260] Example of input prompt for generative AI model

[1261] "When a meeting reaches a silence of more than 10 seconds, generate appropriate facilitation suggestions taking into account the user's emotions. If impatience is detected, suggest a response such as, 'Let's take a short break. Let's talk about the budget review next.'"

[1262] This system significantly reduces the burden on project managers and increases the success rate of projects.

[1263] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1264] Program processing steps

[1265] Step 1:

[1266] A user starts a conference. The terminal collects the user's voice in real time and generates voice data. This voice data becomes input to the server.

[1267] Step 2:

[1268] The device transmits the generated voice data to a server in real time, and the server receives the voice data and passes it as input data to natural language processing technology.

[1269] Step 3:

[1270] The server uses natural language processing techniques to convert the received voice data into text data, which then becomes the input for the next processing step.

[1271] Step 4:

[1272] The server inputs the converted text and voice data into an emotion engine to analyze the user's emotions. The resulting emotion data is output and used for further processing.

[1273] Step 5:

[1274] The server adjusts the facilitation content based on the emotional data. The server starts a timer to detect silence for a certain period of time (for example, 10 seconds). This timer is used as input, and the next step is executed only if silence is detected.

[1275] Step 6:

[1276] When silence is detected, the server generates the next agenda and questions based on the results of the sentiment analysis. These generated agenda and questions are the output and are sent to the participants' devices.

[1277] Step 7:

[1278] The device notifies the user of the agenda or questions it receives. For example, during a meeting, it displays a message saying, "Let's take a short break. Next, let's discuss the budget review."

[1279] Step 8:

[1280] The user ends the conference. The terminal sends all audio data recorded during the conference to the server. This audio data becomes the input for the next processing step.

[1281] Step 9:

[1282] The server converts the received audio data back into text data, which is then used as input for the summarization algorithm.

[1283] Step 10:

[1284] The server feeds the text data into a summarization algorithm to extract key topics and action items, and this extracted data is the output.

[1285] Step 11:

[1286] The server automatically generates meeting minutes and action lists based on the extracted important topics and action items, as well as the user's emotional data. These minutes and action lists are the output and are sent to the participants' devices.

[1287] Step 12:

[1288] The device notifies the user of the minutes and action list it has received. For example, after the meeting, the user can view the minutes, including the main agenda items, action items, and the user's emotional state.

[1289] Step 13:

[1290] The server periodically collects the user's calendar data, which serves as input for the next processing step.

[1291] Step 14:

[1292] The server analyzes the calendar data to determine the user's schedule, and the resulting analysis data becomes the input for the next processing step.

[1293] Step 15:

[1294] The server uses an emotion engine to check the user's emotional state. Based on the emotional data and calendar data, it generates a progress report reminder at the optimal time. This reminder is the output.

[1295] Step 16:

[1296] The server sends the generated reminder to the device, and the device notifies the user of the reminder, for example, by displaying a message such as "Please report on the progress of this week's tasks."

[1297] Step 17:

[1298] The server automatically extracts task data from chat tools and work breakdown structures (WBS), and this task data becomes the input.

[1299] Step 18:

[1300] The server evaluates the extracted task data and the user's emotional state using an emotion engine to identify important tasks. The identified important tasks are the output.

[1301] Step 19:

[1302] The server notifies the administrator of identified important tasks, for example by displaying messages such as "Prepare for weekly review" and "Report to client."

[1303] Step 20:

[1304] The server collects project status data daily, and this status data is used as input.

[1305] Step 21:

[1306] The server evaluates the progress of each task and takes into account the user's emotional state using an emotion engine, thereby identifying high-risk tasks. The identified high-risk tasks are the output.

[1307] Step 22:

[1308] The server generates a response policy for high-risk tasks and notifies the administrator's terminal, for example, by displaying a message such as "Progress on Task A is behind schedule, so please consider adding resources or extending the deadline."

[1309] (Application example 2)

[1310] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1311] Modern factories require efficient work management and communication, but achieving this requires advanced project management and team motivation management. In particular, there is no system in place that utilizes emotion recognition technology to provide support according to the emotional state of workers. Conventional management systems are unable to adjust the timing of appropriate advice or reminders that take into account the emotions of workers. This can lead to stress building up and reduced work efficiency.

[1312] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1313] In this invention, the server has a means for receiving conference audio data in real time and converting the audio content into text using natural language processing technology, a means for analyzing the converted text data and detecting when the conversation has stopped for a certain period of time, a means for generating the next agenda item and questions to be asked when this is detected and notifying the participant terminals, a means for recording the conference audio, converting it into text in real time, sending it to a summarization algorithm to extract important agenda items and action items, automatically generating minutes and action lists and sending them to the participant terminals, and a means for analyzing calendar data to grasp the schedule of each member and select the optimal time. The system includes a means for sending progress report reminders at the most appropriate timing, a means for automatically extracting task data from chat tools and work breakdown structures and notifying a manager's terminal of important tasks, a means for collecting current project status data daily, evaluating each task, generating a response policy for high-risk tasks and notifying the manager's terminal, a means for supporting in-factory worker meetings and providing advice based on the worker's emotional state, a means for automatically generating memos summarizing work content and important points after a shift, and a means for reminding workers of work progress and important contact information at the most appropriate timing based on the worker's emotional state. This enables efficient communication and project management based on the worker's emotional state.

[1314] - "Conference audio data" refers to audio information spoken during a conference, and is data collected in digital form.

[1315] "Natural language processing technology" is a computer science technology for understanding and analyzing the content of speech and text.

[1316] "Text" refers to information in the form of a written language, which is audio data converted into written information.

[1317] A "certain period of time" refers to a specific continuous period of time, such as 10 seconds.

[1318] An "agenda" refers to a specific topic or issue that will be discussed or considered at a conference or meeting.

[1319] A "question" refers to an inquiry or question posed to another person in order to obtain information.

[1320] "Recording" refers to the act of recording sound as digital data.

[1321] A "summarization algorithm" is a computational method for extracting the main content from text data and summarizing it concisely.

[1322] An "agenda" refers to a specific issue or topic that will be the subject of a meeting or discussion.

[1323] An "action item" refers to an item based on an agenda that requires specific action or measures.

[1324] A "minutes" is a document that records the contents of discussions at a conference or meeting.

[1325] "Calendar Data" refers to data containing information about appointments and schedules.

[1326] "Progress report" refers to a report on the progress of a project or task.

[1327] A "reminder" is a notification or alert that reminds you of important appointments or tasks.

[1328] A "chat tool" refers to a tool used for online text and voice communication.

[1329] "Work breakdown structure" refers to a method of breaking down projects and tasks into smaller parts and managing them in a hierarchical structure.

[1330] "Task data" is data that contains information about a particular task or job.

[1331] "Administrator terminal" refers to a computer or device used by a system administrator.

[1332] "Current status data" refers to data that indicates the current status of a project or task.

[1333] A "high-risk task" is one that has a high chance of being delayed or failing.

[1334] "Response policy" refers to solutions or measures to address specific problems or risks.

[1335] "Factory workers" refers to all employees working in a factory.

[1336] "Emotional state" refers to the psychological state or mood of an individual worker.

[1337] "Advice" refers to advice or suggestions given in response to a particular problem or situation.

[1338] "Post-shift" refers to the time after a worker's work shift ends.

[1339] "Work content" refers to the specific activities and actions associated with a particular job or task.

[1340] "Points" refers to important points or main points.

[1341] The "optimal timing" refers to the most effective and appropriate time.

[1342] "Announcements" refer to important information or messages that need to be shared.

[1343] The system for implementing this invention includes multiple means for providing project management and support based on the emotional state of the worker. Specifically, the system uses the following hardware and software to process and calculate data.

[1344] The core hardware of the system consists of a server, a user device (PC, smartphone, tablet, etc.), and a microphone for voice input. The server performs the core processing, and the device provides the interface with the user.

[1345] 1. Real-time processing of conference audio data

[1346] The server converts the meeting audio data into text in real time using a speech recognition tool (e.g., Google Speech-to-Text API) and a natural language processing library (e.g., NLTK or spaCy). The converted text data is analyzed to detect silences over a certain period of time (e.g., 10 seconds).

[1347] 2. Question and agenda generation

[1348] The server uses a large-scale language model (e.g., GPT-3) to generate the next agenda item or question to be asked when silence occurs, and this generated content is notified to the participants' devices.

[1349] 3. Automatic generation of meeting minutes

[1350] The server records the audio of the meeting and transcribes it into text in real time. It then uses a summarization algorithm (such as TextRank) to extract important topics and action items, automatically generating minutes and action lists. These minutes are then sent to the participants' devices.

[1351] 4. Analyzing calendar data and sending reminders

[1352] The server analyzes the user's calendar data to understand each member's schedule, evaluates their emotional state using an emotion engine (e.g., IBM Watson Tone Analyzer), and sends progress report reminders at the optimal time.

[1353] 5. Automatic extraction and management of task data

[1354] The server automatically extracts task data from chat tools (e.g., Slack) and work breakdown structures (WBS) to identify important tasks. These important tasks are notified to the administrator's terminal. It also collects current project status data daily and evaluates each task. It generates response policies for high-risk tasks and notifies the administrator's terminal.

[1355] 6. In-factory support

[1356] It also includes a function to support meetings between workers in a factory and provide advice based on their emotional state. For example, if a worker is feeling impatient, advice such as "Take a short break. Relax before proceeding to the next task" is generated and displayed on the device.

[1357] 7. Automatic generation of work notes

[1358] After each shift, a memo summarizing the work and key points is automatically generated and sent to the worker's device, including key tasks and points requiring attention.

[1359] Examples and prompts

[1360] As a specific example of usage, to give voice instructions at the end of a shift in a factory:

[1361] "The work is progressing slowly today, and everyone seems to be getting impatient."

[1362] In this way, the system supports workers through emotion recognition, enabling efficient project management and communication.

[1363] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1364] Step 1:

[1365] The server receives the conference audio data in real time and converts the audio content into text using natural language processing technology. Specifically, it uses a speech recognition tool (such as the Google Speech-to-Text API) to convert the conference audio into text data. The input of this step is the conference audio data, and the output is the converted text data.

[1366] Step 2:

[1367] The server analyzes the converted text data and detects silences over a certain period of time (e.g., 10 seconds). It analyzes the text using a natural language processing library (e.g., NLTK or spaCy). The input of this step is the text data, and the output is the silence detection result.

[1368] Step 3:

[1369] When the server detects silence, it generates the next agenda and question to be asked and notifies the participant's device. It uses a large-scale language model (such as GPT-3) to generate appropriate agenda and questions. The input of this step is the silence detection result, and the output is the generated agenda and question.

[1370] Step 4:

[1371] The server records the audio of the conference and converts it into text in real time. The recorded audio data is converted into text data. The input of this step is the conference audio data, and the output is the converted audio data.

[1372] Step 5:

[1373] The server sends the text data to a summarization algorithm to extract important topics and action items. We use a summarization algorithm such as TextRank. The input of this step is the text data, and the output is summarized topics and action items.

[1374] Step 6:

[1375] The server automatically generates minutes and action lists based on the summarized data and sends them to the participants' terminals. The input of this step is the summarized agenda and action items, and the output is the minutes and action lists.

[1376] Step 7:

[1377] The server analyzes the user's calendar data to understand each member's schedule. It uses an emotion engine (such as IBM Watson Tone Analyzer) to evaluate the member's emotional state and sends a progress report reminder at the optimal time. The input of this step is the calendar data and emotion data, and the output is a reminder at the optimal time.

[1378] Step 8:

[1379] The server automatically extracts task data from chat tools and work breakdown structures and notifies the manager's device of important tasks. It uses chat tools (such as Slack) and WBS. The input of this step is task data, and the output is a notification of important tasks.

[1380] Step 9:

[1381] The server collects project status data daily and evaluates each task. It generates a response policy for high-risk tasks and notifies the administrator's terminal. The input of this step is the status data, and the output is the risk response policy.

[1382] Step 10:

[1383] The server supports meetings between workers in the factory and provides advice based on their emotional state. It combines speech recognition and emotion analysis models to generate appropriate advice. The input of this step is the worker's emotional data, and the output is an advice message.

[1384] Step 11:

[1385] After each shift, the server automatically generates a memo summarizing the work content and important points and sends it to the worker's device. The input for this step is the shift data and work content data, and the output is a summary memo.

[1386] Step 12:

[1387] The server reminds users about work progress and important communication matters at the optimal timing based on their emotional state. The appropriate timing is determined using an emotion engine. The inputs of this step are work progress data and emotional data, and the output is a reminder notification.

[1388] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1389] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1390] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1391] [Third embodiment]

[1392] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1393] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1394] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1395] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1396] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1397] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1398] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1399] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1400] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1401] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1402] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1403] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1404] The present invention is a system designed to support the role of a project manager (PM) in a system development project. The system uses a large-scale language model to provide multiple functions that optimize the PM's communication and project management tasks. The following describes an embodiment of the system.

[1405] Overall system overview

[1406] This system consists of a server, terminals, and users. The server performs the central processing, and the terminals provide an interface with users. Users are team members and PMs involved in the project.

[1407] Facilitation function

[1408] When a user starts a meeting, the device collects the meeting's audio data and sends it to the server in real time. The server receives the audio data and converts it into text using natural language processing technology. When the server detects silence for a certain period of time (for example, 10 seconds), it generates the next agenda item and questions to be asked and notifies the participants' devices. This allows the meeting to proceed smoothly and important topics to be discussed.

[1409] Examples:

[1410] If 10 seconds of silence occurs during a meeting, the server generates the message "Let's move on to the next agenda item. Next, let's talk about the budget review." and displays it on the participants' devices.

[1411] Minutes function

[1412] When the user ends the meeting, the device sends the audio recorded during the meeting to the server. The server converts the audio into text and sends the text to a summarization algorithm to extract important topics and action items. Based on the extracted information, minutes and action lists are automatically generated and sent to the participants' devices.

[1413] Examples:

[1414] After the meeting, the server generates minutes that include information such as "discussion on the timing of releasing new features" and "tasks for each member," and sends them to the devices of all participants.

[1415] Communication Features

[1416] The server analyzes each user's calendar data to understand their schedules, finds out when the user is not busy, and sends progress report reminders at the appropriate times. These reminders are displayed on the device.

[1417] Examples:

[1418] The server will send a reminder at 10:00 AM on the following Monday, such as "Please report on the progress of this week's tasks."

[1419] PMO Functions

[1420] Task data is extracted from chat tools and work breakdown structures (WBS) and sent to the server. The server evaluates the extracted task data and identifies important tasks. It then notifies the PM's device of important tasks and manages the task progress. Task progress is periodically checked and reminders are sent as necessary.

[1421] Examples:

[1422] The server extracts important tasks such as "preparing for weekly review" and "reporting to client" and notifies the PM's terminal.

[1423] Risk Management Function

[1424] The server collects project status data daily and evaluates the progress of each task. It identifies high-risk tasks and tasks that are expected to be delayed, and generates a response plan for them. The generated response plan is then notified to the administrator's terminal.

[1425] Examples:

[1426] The server presents specific response guidelines, such as "Progress on Task A is behind schedule, so please consider adding resources or extending the delivery date," and notifies the PM's terminal.

[1427] With the above functions, this system can significantly reduce the burden on project managers and increase the success rate of projects.

[1428] The processing flow will be explained below.

[1429] (Facilitate function)

[1430] Step 1:

[1431] The terminal uses the microphone of the terminal to collect voice data during the conference and transmits it to the server in real time.

[1432] Step 2:

[1433] The server converts the received voice data into text using natural language processing technology.

[1434] Step 3:

[1435] The server analyzes the text data and determines whether a certain period of silence (for example, 10 seconds) is detected.

[1436] Step 4:

[1437] If silence is detected, the server generates the next topic to be discussed and appropriate questions.

[1438] Step 5:

[1439] The server transmits the generated agenda and questions to the terminal.

[1440] Step 6:

[1441] The terminal displays the received agenda and questions to the user.

[1442] (Meeting minutes function)

[1443] Step 1:

[1444] The terminal records audio data during the conference and transmits the data to the server after the conference ends.

[1445] Step 2:

[1446] The server converts the received voice data into text using natural language processing technology.

[1447] Step 3:

[1448] The server feeds the generated text data into a summarization algorithm to extract important topics and action items.

[1449] Step 4:

[1450] The server automatically generates minutes and action lists based on the extracted information.

[1451] Step 5:

[1452] The server transmits the generated minutes and action list to the terminal.

[1453] Step 6:

[1454] The terminal displays the received minutes and action list to the user.

[1455] (Communication function)

[1456] Step 1:

[1457] The server periodically collects and analyzes the user's calendar data.

[1458] Step 2:

[1459] The server takes into account the user's schedule and generates progress report reminders at optimal times.

[1460] Step 3:

[1461] The server transmits the generated reminder to the terminal.

[1462] Step 4:

[1463] The terminal notifies the user of the received reminder.

[1464] (PMO function)

[1465] Step 1:

[1466] The server automatically extracts task data from chat tools and work breakdown structures (WBS) on a regular basis.

[1467] Step 2:

[1468] The server evaluates the extracted task data and identifies important tasks.

[1469] Step 3:

[1470] The server notifies the administrator terminal of the identified important task.

[1471] Step 4:

[1472] The server periodically monitors the progress of the task and sends reminders to the administrator terminal as needed.

[1473] (Risk management function)

[1474] Step 1:

[1475] The server collects project status data daily.

[1476] Step 2:

[1477] The server evaluates tasks based on the collected data and identifies high-risk tasks.

[1478] Step 3:

[1479] The server generates a response policy for the identified risks.

[1480] Step 4:

[1481] The server notifies the administrator terminal of the generated response policy.

[1482] Step 5:

[1483] The terminal displays the received response policy to the administrator.

[1484] Example 1

[1485] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1486] Conventional systems require project managers (PMs) to manually manage a wide range of tasks, conduct meetings, create minutes, and monitor the progress of each member, requiring a great deal of time and effort. This can slow down project progress and delay the early detection and response of risks. Other issues include overlooking important agenda items during meetings and insufficient follow-up afterward. There is a need for a system that can solve these problems, reduce the burden on PMs, and improve the success rate of projects.

[1487] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1488] In this invention, the server includes: means for receiving conference audio data in real time and converting the audio content into text using natural language processing technology; means for analyzing the converted text data and detecting when the conversation has stopped for a certain period of time; means for generating next agenda items and questions and notifying participant terminals when detected; means for recording the conference audio, converting it into text in real time, sending it to a summarization algorithm to extract important agenda items and action items, automatically generating minutes and action lists and sending them to participant terminals; means for analyzing calendar data to understand each member's schedule and sending progress report reminders at optimal times; means for automatically extracting task data from a chat tool or a work breakdown structure and notifying a manager terminal of important tasks; means for collecting project status data daily, evaluating each task, generating response policies for high-risk tasks and notifying the manager terminal; and means for proposing project management actions based on the generated response policies. This automates the manual work of project managers and enables efficient project management.

[1489] "Conference audio data" refers to audio information generated during a conference, including statements and discussions.

[1490] "Natural language processing technology" is a technology that allows computers to understand and process the language that humans use on a daily basis.

[1491] "Text data" refers to text information generated from audio data using natural language processing technology.

[1492] "Conversion" is the process of switching data from one format to another, in this case changing audio data to text data.

[1493] The "certain period of time" refers to the length of time that serves as a reference in the detection process, and in this invention specifically means 10 seconds.

[1494] A "lull in conversation" refers to a state in which no one speaks for a certain period of time or longer.

[1495] An "agenda" refers to the subject or topic of a meeting or discussion.

[1496] A "question" is a question posed to obtain specific information or opinions.

[1497] "Participant terminal" refers to an electronic device such as a computer or mobile device used by a user participating in a conference.

[1498] A "summarization algorithm" is a computational procedure for extracting the important parts of long text data and presenting them in a shortened form.

[1499] "Agenda and action items" refers to the main topics covered during the meeting and any specific action items that will follow.

[1500] "Minutes" refers to a document that records the contents of a meeting.

[1501] An "action list" is a document that lists specific actions to be taken after a meeting.

[1502] "Calendar data" refers to schedule information written on an electronic calendar used by a user.

[1503] A "progress report" is a report on the progress of a project, usually done on a regular basis.

[1504] A "reminder" is information that notifies a user to remind them of a particular action or task.

[1505] A "chat tool" is software that users use to communicate instantly through text.

[1506] A work breakdown structure (WBS) is a diagram or list that breaks down project tasks hierarchically into manageable sections.

[1507] "Task data" refers to information about specific work items to be done as part of a project.

[1508] "Project status data" refers to data that indicates the current progress of a project and the status of tasks.

[1509] "Evaluation" refers to the process of analyzing data and determining its status or value based on specific criteria.

[1510] A "high-risk task" is a work item that is likely not to proceed as planned.

[1511] A "response policy" refers to a countermeasure or action plan for a specific problem or situation.

[1512] "Server" refers to a computing device that performs central processing and exchanges data with other terminals.

[1513] "Means of suggestion" refers to the method by which the system notifies the user of actions and countermeasures and supports their progress.

[1514] MODE FOR CARRYING OUT THE INVENTION

[1515] This invention is a system designed to support the role of project managers (PMs) in system development projects. This system consists of three main components: a server, a terminal, and a user, and provides multiple functions to optimize the PM's communication and project management tasks.

[1516] Hardware and software used

[1517] Server: A computer device that performs central processing and exchanges data with other devices

[1518] Terminal: An electronic device (e.g., computer, mobile device) that provides an interface with a user.

[1519] Natural language processing technology: Google Cloud Speech-to-Text API

[1520] Summarization Algorithm: BERT-Based Summarization Model

[1521] Calendar Data Analysis: Google Calendar API

[1522] Task data extraction: Slack API, JIRA API

[1523] Collecting project status data: JIRA API, GitLab API

[1524] Features and specific operation examples

[1525] Facilitation function

[1526] When a user starts a meeting, the device collects the audio data of the meeting and sends it to the server in real time. When the server receives the audio data, it converts it into text using natural language processing technology. If the server detects silence for a certain period of time (for example, 10 seconds), it generates the next agenda item and questions and notifies the participants' devices.

[1527] Example: If 10 seconds of silence occurs during a meeting, the server generates the message "Let's move on to the next agenda item. Let's discuss the budget review next." and displays it on the participants' devices.

[1528] Minutes function

[1529] When the user ends the meeting, the device sends the audio data recorded during the meeting to the server, which converts the audio data into text using natural language processing technology and sends the text data to a summarization algorithm to extract important topics and action items. Based on this, minutes and action lists are automatically generated and sent to the participants' devices.

[1530] Example: After the meeting, the server generates minutes that include "discussions about the timing of releasing new features" and "tasks for each member," and sends them to the participants' devices.

[1531] Communication Features

[1532] The server analyzes each user's calendar data to understand their schedule, finds times when the user is not busy, and generates progress report reminders at those times and sends them to each device.

[1533] Example: The server sends a reminder to participants' devices at 10:00 a.m. on the following Monday, such as "Please report on the progress of this week's tasks."

[1534] PMO Functions

[1535] Task data is extracted from chat tools and work breakdown structures (WBS) and sent to the server. The server evaluates the extracted task data and identifies important tasks. It then notifies the PM's device of important tasks and manages the task progress. Task progress is periodically checked and reminders are sent as necessary.

[1536] Example: The server extracts important tasks such as "preparing for weekly review" and "reporting to client" and notifies the PM's terminal.

[1537] Risk Management Function

[1538] The server collects project status data daily and evaluates the progress of each task. It identifies high-risk tasks and tasks that are expected to be delayed, and generates a response plan for them. The generated response plan is then notified to the administrator's terminal.

[1539] Example: The server presents a specific response policy such as "Progress on Task A is behind schedule, so please consider adding resources or extending the deadline," and notifies the PM's terminal.

[1540] Examples of prompt statements

[1541] To use this system, the following is an example of a prompt sentence to input into the generative AI model:

[1542] Meeting Facilitation Functions:

[1543] Prompt: "Detect 10 seconds of silence during a meeting. Generate next agenda items or questions."

[1544] Minutes feature:

[1545] Prompt: "Convert the audio data from a meeting recording into text and generate a summary."

[1546] Communication features:

[1547] Prompt: "Analyze the user's calendar data to find the appropriate time for task progress reports and send reminders."

[1548] PMO Functions:

[1549] Prompt: "Extract tasks from the chat tool or WBS, identify important tasks, and notify the PM."

[1550] Risk Management Functions:

[1551] Prompt: "Evaluate the progress of tasks and generate and communicate specific action plans for high-risk tasks."

[1552] In this way, by using the system of the present invention, it is possible to significantly reduce the burden on project managers and improve the success rate of projects.

[1553] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1554] Facilitation function

[1555] Processing Steps

[1556] Step 1:

[1557] A user starts a conference using a terminal. The user inputs the conference start time, participant list, etc. into the system. This information becomes the initial data for the system.

[1558] Specific behavior:

[1559] The user enters the details of the meeting on the meeting setting screen and presses the "Start" button.

[1560] Step 2:

[1561] The device collects audio data during the meeting in real time using a built-in microphone or an external microphone, and stores the collected audio data in a buffer.

[1562] Specific behavior:

[1563] The device activates the microphone and begins collecting audio.

[1564] Input: Audio during the meeting

[1565] Output: Collected audio data

[1566] Step 3:

[1567] The device transmits the collected audio data to the server in real time using a secure protocol such as HTTPS.

[1568] Specific behavior:

[1569] The terminal uploads the voice data to the server at regular intervals.

[1570] Input: Collected audio data

[1571] Output: Audio data sent to the server

[1572] Step 4:

[1573] The server converts the received voice data into text using natural language processing technology (Google Cloud Speech-to-Text API).

[1574] Specific behavior:

[1575] The server calls the API and converts the audio data into text data.

[1576] Input: Audio data sent to the server

[1577] Output: Converted text data

[1578] Step 5:

[1579] The server analyzes the text data and detects silence for a certain period of time (10 seconds).

[1580] Specific behavior:

[1581] The server analyzes the time information in the text data and flags any silence of 10 seconds or more.

[1582] Input: Converted text data

[1583] Output: Silence detection flag

[1584] Step 6:

[1585] When the server detects silence, it generates the next agenda item and question to be asked and notifies the participants' terminals.

[1586] Specific behavior:

[1587] The server uses a generative AI model to generate the next agenda item or question and create a notification message.

[1588] Input: silence detection flag

[1589] Output: Generated agenda and questions

[1590] Specific behavior:

[1591] The generated agenda and questions are displayed on the participants' devices.

[1592] Minutes function

[1593] Processing Steps

[1594] Step 1:

[1595] The user ends the conference using the terminal. The end operation is performed through the terminal interface.

[1596] Specific behavior:

[1597] The user presses the "Finish" button.

[1598] Step 2:

[1599] The terminal transmits the audio data recorded during the meeting to the server.

[1600] Specific behavior:

[1601] The device uploads the recording data to the server.

[1602] Input: Recorded audio data

[1603] Output: Audio data sent to the server

[1604] Step 3:

[1605] The server converts the voice data into text using natural language processing technology.

[1606] Specific behavior:

[1607] The server calls the API and converts the audio data into text data.

[1608] Input: Audio data sent to the server

[1609] Output: Converted text data

[1610] Step 4:

[1611] The server sends the text data to a summarization algorithm (a BERT-based summarization model) to extract important topics and action items.

[1612] Specific behavior:

[1613] The server sends the data to a summary model to extract the important information.

[1614] Input: Converted text data

[1615] Output: Extracted important topics and action items

[1616] Step 5:

[1617] The server automatically generates minutes and action lists based on the extracted information.

[1618] Specific behavior:

[1619] The server automatically generates meeting minutes and action list templates by filling them with data.

[1620] Input: Extracted important agenda items and action items

[1621] Output: Auto-generated meeting minutes and action list

[1622] Step 6:

[1623] The server sends the generated minutes and action list to the participants' terminals.

[1624] Specific behavior:

[1625] The server sends the generated document to the terminal.

[1626] Input: Auto-generated meeting minutes and action list

[1627] Output: Document sent to participant's device

[1628] Communication Features

[1629] Processing Steps

[1630] Step 1:

[1631] The server analyzes each user's calendar data and keeps track of their schedules.

[1632] Specific behavior:

[1633] The server uses the Google Calendar API to retrieve and parse the user's calendar data.

[1634] Input: User's calendar data

[1635] Output: Analysis results

[1636] Step 2:

[1637] Based on the analysis results, the server identifies times when the user is not busy.

[1638] Specific behavior:

[1639] The server extracts available time slots from the analysis results.

[1640] Input: Analysis results

[1641] Output: Available time slots

[1642] Step 3:

[1643] The server generates appropriate reminders when the user is not busy and sends them to each device.

[1644] Specific behavior:

[1645] The server uses the generative AI model to create reminders and send them to the device.

[1646] Input: Available time slots

[1647] Output: The generated reminder

[1648] Specific behavior:

[1649] The reminder will be displayed on the user's device.

[1650] PMO Functions

[1651] Processing Steps

[1652] Step 1:

[1653] Task data is extracted from chat tools and work breakdown structures and sent to a server.

[1654] Specific behavior:

[1655] The server retrieves task data using the Slack API or JIRA API.

[1656] Input: Task data generated by chat tools and WBS

[1657] Output: Extracted task data

[1658] Step 2:

[1659] The server evaluates the extracted task data and identifies important tasks.

[1660] Specific behavior:

[1661] The server analyzes the task data and ranks it based on importance.

[1662] Input: Extracted task data

[1663] Output: Identified important tasks

[1664] Step 3:

[1665] The server notifies the PM's terminal of important tasks.

[1666] Specific behavior:

[1667] The server creates a notification message and sends it to the PM's terminal.

[1668] Input: Identified critical tasks

[1669] Output: Tasks notified to the PM's terminal

[1670] Step 4:

[1671] The server periodically checks the progress of each task.

[1672] Specific behavior:

[1673] The server collects task progress data daily and evaluates the current status.

[1674] Input: Task progress data

[1675] Output: Progress evaluation results

[1676] Step 5:

[1677] The server generates reminders as needed and sends them to the PM's terminal.

[1678] Specific behavior:

[1679] The server creates a reminder and sends it to the PM's device.

[1680] Input: Progress evaluation results

[1681] Output: The generated reminder

[1682] Risk Management Function

[1683] Processing Steps

[1684] Step 1:

[1685] The server collects project status data daily.

[1686] Specific behavior:

[1687] The server retrieves the latest data using the JIRA API or GitLab API.

[1688] Input: Project status data

[1689] Output: Collected data

[1690] Step 2:

[1691] The server evaluates the progress of the task based on the collected data.

[1692] Specific behavior:

[1693] The server analyzes the collected data and evaluates the progress of each task.

[1694] Input: Collected data

[1695] Output: Progress evaluation results

[1696] Step 3:

[1697] Based on the progress, the server identifies high-risk tasks and tasks that are expected to be delayed.

[1698] Specific behavior:

[1699] The server identifies risk tasks based on the progress evaluation results.

[1700] Input: Progress evaluation results

[1701] Output: Identified risk tasks

[1702] Step 4:

[1703] Once a risk is identified, the server generates a response policy.

[1704] Specific behavior:

[1705] The server uses the generative AI model to create a specific response policy.

[1706] Input: Identified Risk Tasks

[1707] Output: Generated response policy

[1708] Step 5:

[1709] The generated response policy is notified to the administrator terminal.

[1710] Specific behavior:

[1711] The server sends the response policy as a notification message to the administrator terminal.

[1712] Input: Generated response policy

[1713] Output: Response policy notified to administrator terminal

[1714] (Application example 1)

[1715] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1716] Conventional project management systems place a heavy burden on project managers, requiring a lot of time and effort, especially for running meetings, taking minutes, managing progress, and managing risks. As a result, it is difficult to send timely reminders and manage progress, especially when it comes to robot maintenance work and task management in factories, which can lead to issues that reduce project efficiency.

[1717] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1718] In this invention, the server has means for receiving conference audio data in real time and converting the audio content into text using natural language processing technology, means for analyzing the converted text data and detecting when the conversation has stopped for a certain period of time, means for generating the next agenda item and questions to be asked when this is detected and notifying the participant terminals, means for recording the conference audio, converting it into text in real time, sending it to a summarization algorithm to extract important agenda items and action items, automatically generating minutes and action lists and sending them to the participant terminals, and means for analyzing calendar data to grasp the schedule of each member and optimally The system includes a means for automatically extracting task data from chat tools and work breakdown structures and notifying the manager's terminal of important tasks, a means for collecting current project status data daily, evaluating each task, generating response policies for high-risk tasks and notifying the manager's terminal, a means for managing the progress of maintenance work using a smartphone and sending reminders, a means for supporting the progress of meetings using a facilitation function, and a means for collecting current status data within the factory, generating response policies for high-risk work and notifying the manager's terminal. This makes it possible to smoothly progress meetings, automate the creation of meeting minutes, and send timely reminders for maintenance work, significantly reducing the burden on project managers and improving the efficiency and success rate of projects.

[1719] "Conference audio data" refers to all audio data recorded during a conference.

[1720] "Natural language processing technology" refers to technology that enables computers to understand and process the natural language that humans use on a daily basis.

[1721] "Text data" refers to data obtained by converting voice data into character information.

[1722] "Detecting a pause in conversation for a certain period of time" refers to detecting that no speech has occurred within a specified period of time.

[1723] "Generating agendas and questions" refers to automatically creating next topics and questions to help keep the meeting moving.

[1724] The term "participant terminal" refers to an electronic terminal used by a user participating in a conference.

[1725] "Key Agenda and Action Items" refers to the key issues discussed at the meeting and the action items related to those issues.

[1726] "Minutes" refers to a document summarizing the matters discussed at a meeting.

[1727] An "action list" is a list of action plans and task items decided at a meeting.

[1728] "Calendar data" refers to data that records a user's schedule and plans.

[1729] "Progress report reminder" refers to a message that notifies you not to forget to report your progress.

[1730] "Chat tool" refers to an application for text-based communication.

[1731] A "work breakdown structure" refers to a structure in which a project is divided into task units.

[1732] "Task Data" refers to information related to each task in a project.

[1733] "Administrator terminal" refers to the electronic terminal used by the project manager or administrator.

[1734] "Project Status Data" means data regarding the current progress or status of a Project.

[1735] A "high-risk task" is one that is likely to be slow or have problems.

[1736] "Response policy" refers to a specific action plan or response measures to resolve the problem.

[1737] "Maintenance work progress" refers to the current progress of maintenance work on robots and other equipment being carried out within the factory.

[1738] "Facilitation function" refers to the function of supporting the progress of meetings and promoting smooth discussions.

[1739] "Factory current status data" refers to data related to the overall operational operations and equipment status within the factory.

[1740] MODE FOR CARRYING OUT THE INVENTION

[1741] System Overview

[1742] This invention is designed to support project management and communication within factories, with a particular focus on optimizing the role of the project manager. The system consists of a server, terminals (such as smartphones), and users (factory workers and managers). The system provides the following main functions:

[1743] Meeting progress facilitator function

[1744] The server receives audio from the meeting in real time via smartphones. This audio data is converted into text using a speech recognition API (e.g., Google Cloud Speech-to-Text). Next, when a certain period of silence (e.g., 10 seconds) is detected, the server uses a generative AI model (e.g., GPT-3) to generate the next agenda item and questions, and notifies the participants' smartphones. This ensures the meeting proceeds smoothly and prevents delays in work.

[1745] Specific examples

[1746] An example of a prompt generated by the server when silence occurs:

[1747] "Let's move on to the next topic. Let's talk about the progress on robot maintenance."

[1748] Automatic maintenance log summary function

[1749] When a user ends a meeting, their smartphone sends the audio recorded during the meeting to a server. The server converts the audio into text and sends the text to a summarization algorithm (e.g., BERT) to extract important topics and action items. Based on the extracted information, meeting minutes and action lists are automatically generated and sent to the participants' smartphones.

[1750] Specific examples

[1751] An example of a prompt generated by the server after a conference ends:

[1752] "Automatically generate minutes for maintenance meetings."

[1753] Maintenance work reminder function

[1754] The server analyzes each user's calendar data to understand their schedule, finds out when the user is not busy, and sends progress report reminders at the appropriate times. These reminders are displayed on the user's smartphone.

[1755] Specific examples

[1756] Example of a prompt when sending a reminder:

[1757] "Don't forget to perform maintenance on your robot this week."

[1758] In-factory risk management function

[1759] The server collects current status data from within the factory daily and evaluates the progress of each task. It identifies high-risk tasks and tasks that are expected to be delayed, and generates countermeasures for them. The generated countermeasures are then sent to the manager's smartphone.

[1760] Specific examples

[1761] An example of a prompt generated by the server in case of an increased risk:

[1762] "Progress on Task A is behind schedule, so please consider adding more resources or extending the deadline."

[1763] Hardware and software used

[1764] The main hardware and software used to implement this system are as follows:

[1765] Smartphones (Apple iPhone, Android devices, etc.)

[1766] Server (Cloud-based server)

[1767] Speech Recognition API (Google Cloud Speech-to-Text)

[1768] Natural language processing engine (GPT-3)

[1769] Summarization Algorithm (BERT)

[1770] Calendar API (Google Calendar)

[1771] The combination of these elements can significantly improve project management and communication within the factory, making it more efficient and effective.

[1772] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1773] Program processing flow

[1774] Step 1:

[1775] The server receives the audio data of the meeting in real time via smartphones. The input is the audio data spoken during the meeting, and the output is that audio data. The server sends this audio data to a speech recognition API (Google Cloud Speech-to-Text) and converts it into text.

[1776] Step 2:

[1777] The terminal receives text data sent from the server. The input is converted text data, and the output is text data for necessary analysis. The server analyzes this text data and detects when the conversation has stopped for a certain period of time.

[1778] Step 3:

[1779] When the server detects silence for a certain period of time (for example, 10 seconds), it uses a generative AI model (GPT-3) to generate the next agenda item and questions to proceed with. The input is the detected silence information and the current meeting situation, and the output is text data of the generated agenda item and questions. The generated agenda item and questions are notified to the participants' smartphones.

[1780] Step 4:

[1781] When a user ends a meeting, the device sends the recorded meeting audio data to the server. The input is the recording data during the meeting, and the output is the audio data. The server converts the audio back into text and sends the text data to a summarization algorithm (BERT) to extract important topics and action items.

[1782] Step 5:

[1783] The server automatically generates meeting minutes and action lists based on the extracted important agenda items and action items. The input is the extracted agenda items and action items, and the output is text data of the meeting minutes and action list summarizing the entire meeting. The generated meeting minutes and action list are sent to the participants' smartphones.

[1784] Step 6:

[1785] The server analyzes each user's calendar data. The input is the user's calendar data, and the output is the analysis result. The server finds out when the user is not busy and generates progress report reminders.

[1786] Step 7:

[1787] The server sends the generated reminder to the user's smartphone at the appropriate time. The input is the generated reminder, and the output is the notification message sent to the user.

[1788] Step 8:

[1789] The server collects current status data from within the factory daily and evaluates the progress of tasks. The input is various data from within the factory (sensor information, work logs, etc.), and the output is the evaluation result of the progress of each task.

[1790] Step 9:

[1791] Based on the progress evaluation results, the server identifies high-risk tasks and tasks showing signs of delay and generates a response policy. The input is the evaluation results, and the output is text data containing specific response policies. The generated response policy is notified to the administrator's terminal.

[1792] Through these steps, this system can reduce the burden on project managers and effectively support project management and communication within the factory.

[1793] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1794] The present invention is a system for supporting the role of a project manager (PM) in a system development project, and in particular incorporates a user emotion recognition function. The system utilizes a large-scale language model and an emotion engine to provide multiple functions that optimize the PM's communication and project management tasks. The following describes an embodiment of the system.

[1795] Overall system overview

[1796] This system consists of a server, terminals, and users. The server performs the central processing, and the terminals provide the interface with users. Users are team members and PMs involved in the project. The emotion engine has the function of analyzing user emotions in real time from conference audio data.

[1797] Facilitation function

[1798] When a user starts a meeting, the device collects the meeting's audio data and sends it to the server in real time. The server receives the audio data and converts it into text using natural language processing technology. An emotion engine then analyzes the user's emotions from the audio data and adjusts the facilitation content based on their emotional state. When the server detects silence for a certain period of time (for example, 10 seconds), it generates the next agenda item and questions to be asked and notifies the participants' devices.

[1799] Examples:

[1800] If there is a 10-second silence during a meeting and the emotion engine detects the user's impatience, the server will generate the message "Let's take a short break. Next, let's talk about budget review," and display it on the participant's device.

[1801] Minutes function

[1802] When a user ends a meeting, the device sends the audio data recorded during the meeting to the server. The server converts the audio into text and sends the text to a summarization algorithm to extract important topics and action items. The emotion engine also records the user's emotional state during the meeting and reflects this information in the minutes. The server then automatically generates the final minutes and action list and sends them to the participants' devices.

[1803] Examples:

[1804] After the meeting ends, the server generates minutes containing "information about the main agenda items and action items, as well as the user's emotional state (e.g., anxiety or relief)" and sends them to all participants' devices.

[1805] Communication Features

[1806] The server periodically collects and analyzes the user's calendar data. It generates progress report reminders at optimal times, taking into account the user's emotional state. The server then sends the reminders to the device, which then notifies the user.

[1807] Examples:

[1808] The server sends a reminder to the user during a non-busy time and when the user's emotional state is calm, saying, "Please report on the progress of this week's tasks."

[1809] PMO Functions

[1810] The server extracts and evaluates task data from chat tools and work breakdown structures (WBS). It also evaluates the user's emotional state using an emotion engine to identify important tasks. The server notifies the administrator's device of identified important tasks and manages the progress of the tasks. It also periodically checks the progress of tasks and sends reminders as necessary.

[1811] Examples:

[1812] The server extracts important tasks such as "preparing for weekly review" and "reporting to client" and notifies the PM's terminal after taking into consideration the user's emotional state.

[1813] Risk Management Function

[1814] The server collects project status data daily and evaluates the progress of each task. The emotion engine performs risk assessments, taking into account the user's emotional state, and identifies high-risk tasks and tasks that are expected to be delayed. The server then generates a response policy for those risks and notifies the administrator's terminal.

[1815] Examples:

[1816] The server presents a specific response policy, such as "Progress on Task A is behind schedule and the user is showing signs of impatience, so please consider adding resources or extending the deadline," and notifies the administrator's terminal.

[1817] With the above functions, this system can significantly reduce the burden on project managers and further increase the success rate of projects by taking users' emotions into consideration.

[1818] The processing flow will be explained below.

[1819] (Facilitate function)

[1820] Step 1:

[1821] The terminal uses the microphone of the terminal to collect voice data during the conference and transmits it to the server in real time.

[1822] Step 2:

[1823] The server converts the received voice data into text using natural language processing technology.

[1824] Step 3:

[1825] The server analyzes the text data and determines whether a certain period of silence (for example, 10 seconds) is detected.

[1826] Step 4:

[1827] At the same time, the server uses an emotion engine to analyze the user's emotions from the voice data in real time.

[1828] Step 5:

[1829] The server adjusts and generates next steps and questions when silence is detected and the user's emotions indicate impatience or anxiety.

[1830] Step 6:

[1831] The server transmits the generated agenda and questions to the terminal.

[1832] Step 7:

[1833] The terminal displays the received agenda and questions to the user.

[1834] (Meeting minutes function)

[1835] Step 1:

[1836] The terminal records audio data during the conference and transmits the data to the server after the conference ends.

[1837] Step 2:

[1838] The server converts the received voice data into text using natural language processing technology.

[1839] Step 3:

[1840] The server then feeds the generated text data into a summarization algorithm to extract important topics and action items.The server also analyzes the user's emotional state recorded during the meeting using an emotion engine.

[1841] Step 4:

[1842] The server automatically generates meeting minutes and action lists that include the extracted important agenda items and action items, as well as the user's emotional state.

[1843] Step 5:

[1844] The server transmits the generated minutes and action list to the terminal.

[1845] Step 6:

[1846] The terminal displays the received minutes and action list to the user.

[1847] (Communication function)

[1848] Step 1:

[1849] The server periodically collects and analyzes the user's calendar data.

[1850] Step 2:

[1851] The server uses an emotion engine to analyze the user's emotional state and generates optimally timed progress report reminders based on calendar data.

[1852] Step 3:

[1853] The server transmits the generated reminder to the terminal.

[1854] Step 4:

[1855] The terminal notifies the user of the received reminder.

[1856] (PMO function)

[1857] Step 1:

[1858] The server automatically extracts task data from chat tools and work breakdown structures (WBS) on a regular basis.

[1859] Step 2:

[1860] The server evaluates the extracted task data and identifies important tasks.

[1861] Step 3:

[1862] The server uses an emotion engine to analyze the emotional state of users involved in tasks and adjusts task priorities and allocations accordingly.

[1863] Step 4:

[1864] The server notifies the administrator terminal of important tasks and the results of their adjustments.

[1865] Step 5:

[1866] The server periodically monitors the progress of the task and sends reminders to the administrator terminal as needed.

[1867] (Risk management function)

[1868] Step 1:

[1869] The server collects project status data daily.

[1870] Step 2:

[1871] The server evaluates tasks based on the collected data and identifies high-risk tasks.

[1872] Step 3:

[1873] The server uses an emotion engine to make a risk assessment that also takes into account the user's emotional state.

[1874] Step 4:

[1875] The server generates a response policy for high-risk tasks.

[1876] Step 5:

[1877] The server notifies the administrator terminal of the generated response policy.

[1878] Step 6:

[1879] The terminal displays the received response policy to the administrator.

[1880] Example 2

[1881] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1882] Conventional project management systems were unable to take into account the user's emotional state when managing meetings or tasks, resulting in frequent stalled conversations and unclear task priorities, which could hinder smooth project progress. In terms of risk management, it was also difficult to grasp the user's emotional state and take appropriate action. As a result, project progress was often delayed and friction in communication occurred.

[1883] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for receiving conference audio data in real time and converting the audio content into text using natural language processing technology; means for analyzing user emotions from the converted text data and audio data; means for adjusting facilitation content based on the analyzed emotional data, and when it detects that the conversation has been interrupted for a certain period of time, generating the next agenda item and questions to proceed with and notifying the participant terminals; means for recording the conference audio, converting it into text in real time, sending it to a summarization algorithm to extract important agenda items and action items, automatically generating minutes and action lists including the user's emotional state, and sending them to the participant terminals; means for analyzing calendar data to grasp the schedule and emotional state of each member and sending progress report reminders at the optimal time; means for automatically extracting task data from a chat tool or work breakdown structure, and notifying the administrator terminal of important tasks while also including the user's emotional state in the evaluation; and means for collecting current project status data daily, evaluating each task, generating response policies for high-risk tasks taking the user's emotional state into consideration, and notifying the administrator terminal. This allows project management to take into account the user's emotional state, resulting in smoother meetings, clearer task priorities, and better risk management.

[1884] "Conference voice data" refers to data that digitally represents the voices of speakers collected during a conference.

[1885] "Natural language processing technology" is a computer technology for analyzing voice or text data and understanding its meaning.

[1886] A "means for converting audio content to text" is a method or device for analyzing audio data and converting the content into corresponding text data.

[1887] "Converted text data" refers to data in which voice data is converted into text information using natural language processing technology.

[1888] The "means for analyzing user emotions" refers to a technique or device for analyzing a user's emotional state (for example, joy, anger, sadness) from text data or voice data.

[1889] "Emotion data" is information that indicates the emotional state of the user.

[1890] "Means for adjusting facilitation content" refers to techniques or methods for optimizing the progress of a meeting and the content of the agenda based on emotional data.

[1891] A "means for detecting a pause in conversation" is a technology or device for detecting silence or a pause in conversation for a certain period of time or more.

[1892] A "participant terminal" is a device (for example, a PC, a smartphone, or a tablet) used by a user participating in a conference.

[1893] The "means for generating next topics and questions" is a technology or device for automatically generating next topics and questions based on the current conversation situation.

[1894] A "real-time text conversion means" is a technique or method for collecting voice data and simultaneously converting it into text form.

[1895] A "summarization algorithm" is a technology that automatically extracts important information from long text data and summarizes it in a concise manner.

[1896] A "means for extracting agenda items and action items" is a technique or method for identifying important agenda items and upcoming actions from text data.

[1897] "Means for automatically generating meeting minutes and action lists" refers to a technology or method for automatically creating meeting minutes and action lists based on agendas and action items.

[1898] "Calendar data" is data that includes the user's schedule information.

[1899] The "means for sending a progress report reminder at an optimal time" is a method or technology for sending a notification prompting a progress report at an appropriate time, taking into consideration the user's schedule and emotional state.

[1900] A "chat tool" is software or related services that support communication between project members.

[1901] A "work breakdown structure" is a framework or method for breaking down project tasks into detailed sections and hierarchically organizing them.

[1902] The "means for automatically extracting task data" is a technology or method for automatically acquiring information about tasks from a chat tool or a work breakdown structure.

[1903] "Current status data" is data that indicates the progress and status of a project.

[1904] A "daily collection means" is a technique or method for collecting data on a regular daily basis.

[1905] The "means for generating a response policy for a high-risk task" is a technique or method for formulating specific response measures or action plans for a high-risk task.

[1906] The present invention is a system for supporting the role of a project manager (PM) in a system development project, and in particular incorporates a function for recognizing user emotions. Specific embodiments for carrying out the present invention will be described below.

[1907] Hardware and Software Configuration

[1908] This system consists of a server, terminals, and users. The server performs the central processing, and the terminals provide an interface with users. Users are team members and PMs involved in the project.

[1909] Hardware used

[1910] Server: A high-performance server system (e.g., cloud service) that processes and stores data.

[1911] Terminal: A device that provides an interface with the user (e.g., PC, smartphone, tablet)

[1912] Software used

[1913] Natural language processing technology: Libraries that convert voice data into text (e.g., Google Cloud Speech-to-Text)

[1914] Emotion engine: A library that analyzes user emotions from voice data (e.g., Microsoft Azure Emotion API)

[1915] Summarization algorithms: techniques for extracting important information from meeting texts (e.g., BERT)

[1916] Chat tool: Software for managing communication within a project (e.g., Slack)

[1917] Work Breakdown Structure (WBS): A framework for breaking down and managing tasks into detailed sections

[1918] Overall system overview

[1919] The server receives the conference audio data in real time and converts the audio content into text using natural language processing technology. It then analyzes the user's emotions from the converted text and audio data. The analyzed emotional data is used to generate the next agenda item or question to move on to if the conversation stops. The generated agenda item or question is then sent to the participants' devices.

[1920] The server also records the audio of the meeting and converts it into text in real time. The text data is sent to a summarization algorithm to extract important topics and action items. The user's emotional state is also reflected in the minutes, and the final minutes and action list are automatically generated and sent to the participants' devices.

[1921] The server analyzes calendar data and understands each member's schedule and emotional state to send progress report reminders at the optimal time. Furthermore, it automatically extracts task data from chat tools and work breakdown structures, and notifies the administrator's device of important tasks, including the user's emotional state in its evaluation.

[1922] The server also collects project status data daily and evaluates the progress of each task. For high-risk tasks, a response policy is generated that takes into account the user's emotional state and is notified to the administrator's terminal.

[1923] Specific examples

[1924] Facilitation function

[1925] Examples:

[1926] If there is a 10-second silence during a meeting and the emotion engine detects the user's impatience, the server will generate the message "Let's take a short break. Next, let's talk about budget review," and display it on the participant's device.

[1927] Minutes function

[1928] Examples:

[1929] After the meeting ends, the server generates minutes containing "information about the main agenda items and action items, as well as the user's emotional state (e.g., anxiety or relief)" and sends them to all participants' devices.

[1930] Communication Features

[1931] Examples:

[1932] The server sends a reminder to the user during a non-busy time and when the user's emotional state is calm, saying, "Please report on the progress of this week's tasks."

[1933] PMO Functions

[1934] Examples:

[1935] The server extracts important tasks such as "preparing for weekly review" and "reporting to client" and notifies the PM's terminal after taking into consideration the user's emotional state.

[1936] Risk Management Function

[1937] Examples:

[1938] The server presents a specific response policy, such as "Progress on Task A is behind schedule and the user is showing signs of impatience, so please consider adding resources or extending the deadline," and notifies the administrator's terminal.

[1939] Example of input prompt for generative AI model

[1940] "When a meeting reaches a silence of more than 10 seconds, generate appropriate facilitation suggestions taking into account the user's emotions. If impatience is detected, suggest a response such as, 'Let's take a short break. Let's talk about the budget review next.'"

[1941] This system significantly reduces the burden on project managers and increases the success rate of projects.

[1942] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1943] Program processing steps

[1944] Step 1:

[1945] A user starts a conference. The terminal collects the user's voice in real time and generates voice data. This voice data becomes input to the server.

[1946] Step 2:

[1947] The device transmits the generated voice data to a server in real time, and the server receives the voice data and passes it as input data to natural language processing technology.

[1948] Step 3:

[1949] The server uses natural language processing techniques to convert the received voice data into text data, which then becomes the input for the next processing step.

[1950] Step 4:

[1951] The server inputs the converted text and voice data into an emotion engine to analyze the user's emotions. The resulting emotion data is output and used for further processing.

[1952] Step 5:

[1953] The server adjusts the facilitation content based on the emotional data. The server starts a timer to detect silence for a certain period of time (for example, 10 seconds). This timer is used as input, and the next step is executed only if silence is detected.

[1954] Step 6:

[1955] When silence is detected, the server generates the next agenda and questions based on the results of the sentiment analysis. These generated agenda and questions are the output and are sent to the participants' devices.

[1956] Step 7:

[1957] The device notifies the user of the agenda or questions it receives. For example, during a meeting, it displays a message saying, "Let's take a short break. Next, let's discuss the budget review."

[1958] Step 8:

[1959] The user ends the conference. The terminal sends all audio data recorded during the conference to the server. This audio data becomes the input for the next processing step.

[1960] Step 9:

[1961] The server converts the received audio data back into text data, which is then used as input for the summarization algorithm.

[1962] Step 10:

[1963] The server feeds the text data into a summarization algorithm to extract key topics and action items, and this extracted data is the output.

[1964] Step 11:

[1965] The server automatically generates meeting minutes and action lists based on the extracted important topics and action items, as well as the user's emotional data. These minutes and action lists are the output and are sent to the participants' devices.

[1966] Step 12:

[1967] The device notifies the user of the minutes and action list it has received. For example, after the meeting, the user can view the minutes, including the main agenda items, action items, and the user's emotional state.

[1968] Step 13:

[1969] The server periodically collects the user's calendar data, which serves as input for the next processing step.

[1970] Step 14:

[1971] The server analyzes the calendar data to determine the user's schedule, and the resulting analysis data becomes the input for the next processing step.

[1972] Step 15:

[1973] The server uses an emotion engine to check the user's emotional state. Based on the emotional data and calendar data, it generates a progress report reminder at the optimal time. This reminder is the output.

[1974] Step 16:

[1975] The server sends the generated reminder to the device, and the device notifies the user of the reminder, for example, by displaying a message such as "Please report on the progress of this week's tasks."

[1976] Step 17:

[1977] The server automatically extracts task data from chat tools and work breakdown structures (WBS), and this task data becomes the input.

[1978] Step 18:

[1979] The server evaluates the extracted task data and the user's emotional state using an emotion engine to identify important tasks. The identified important tasks are the output.

[1980] Step 19:

[1981] The server notifies the administrator of identified important tasks, for example by displaying messages such as "Prepare for weekly review" and "Report to client."

[1982] Step 20:

[1983] The server collects project status data daily, and this status data is used as input.

[1984] Step 21:

[1985] The server evaluates the progress of each task and takes into account the user's emotional state using an emotion engine, thereby identifying high-risk tasks. The identified high-risk tasks are the output.

[1986] Step 22:

[1987] The server generates a response policy for high-risk tasks and notifies the administrator's terminal, for example, by displaying a message such as "Progress on Task A is behind schedule, so please consider adding resources or extending the deadline."

[1988] (Application example 2)

[1989] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1990] Modern factories require efficient work management and communication, but achieving this requires advanced project management and team motivation management. In particular, there is no system in place that utilizes emotion recognition technology to provide support according to the emotional state of workers. Conventional management systems are unable to adjust the timing of appropriate advice or reminders that take into account the emotions of workers. This can lead to stress building up and reduced work efficiency.

[1991] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1992] In this invention, the server has a means for receiving conference audio data in real time and converting the audio content into text using natural language processing technology, a means for analyzing the converted text data and detecting when the conversation has stopped for a certain period of time, a means for generating the next agenda item and questions to be asked when this is detected and notifying the participant terminals, a means for recording the conference audio, converting it into text in real time, sending it to a summarization algorithm to extract important agenda items and action items, automatically generating minutes and action lists and sending them to the participant terminals, and a means for analyzing calendar data to grasp the schedule of each member and select the optimal time. The system includes a means for sending progress report reminders at the most appropriate timing, a means for automatically extracting task data from chat tools and work breakdown structures and notifying a manager's terminal of important tasks, a means for collecting current project status data daily, evaluating each task, generating a response policy for high-risk tasks and notifying the manager's terminal, a means for supporting in-factory worker meetings and providing advice based on the worker's emotional state, a means for automatically generating memos summarizing work content and important points after a shift, and a means for reminding workers of work progress and important contact information at the most appropriate timing based on the worker's emotional state. This enables efficient communication and project management based on the worker's emotional state.

[1993] - "Conference audio data" refers to audio information spoken during a conference, and is data collected in digital form.

[1994] "Natural language processing technology" is a computer science technology for understanding and analyzing the content of speech and text.

[1995] "Text" refers to information in the form of a written language, which is audio data converted into written information.

[1996] A "certain period of time" refers to a specific continuous period of time, such as 10 seconds.

[1997] An "agenda" refers to a specific topic or issue that will be discussed or considered at a conference or meeting.

[1998] A "question" refers to an inquiry or question posed to another person in order to obtain information.

[1999] "Recording" refers to the act of recording sound as digital data.

[2000] A "summarization algorithm" is a computational method for extracting the main content from text data and summarizing it concisely.

[2001] An "agenda" refers to a specific issue or topic that will be the subject of a meeting or discussion.

[2002] An "action item" refers to an item based on an agenda that requires specific action or measures.

[2003] A "minutes" is a document that records the contents of discussions at a conference or meeting.

[2004] "Calendar Data" refers to data containing information about appointments and schedules.

[2005] "Progress report" refers to a report on the progress of a project or task.

[2006] A "reminder" is a notification or alert that reminds you of important appointments or tasks.

[2007] A "chat tool" refers to a tool used for online text and voice communication.

[2008] "Work breakdown structure" refers to a method of breaking down projects and tasks into smaller parts and managing them in a hierarchical structure.

[2009] "Task data" is data that contains information about a particular task or job.

[2010] "Administrator terminal" refers to a computer or device used by a system administrator.

[2011] "Current status data" refers to data that indicates the current status of a project or task.

[2012] A "high-risk task" is one that has a high chance of being delayed or failing.

[2013] "Response policy" refers to solutions or measures to address specific problems or risks.

[2014] "Factory workers" refers to all employees working in a factory.

[2015] "Emotional state" refers to the psychological state or mood of an individual worker.

[2016] "Advice" refers to advice or suggestions given in response to a particular problem or situation.

[2017] "Post-shift" refers to the time after a worker's work shift ends.

[2018] "Work content" refers to the specific activities and actions associated with a particular job or task.

[2019] "Points" refers to important points or main points.

[2020] The "optimal timing" refers to the most effective and appropriate time.

[2021] "Announcements" refer to important information or messages that need to be shared.

[2022] The system for implementing this invention includes multiple means for providing project management and support based on the emotional state of the worker. Specifically, the system uses the following hardware and software to process and calculate data.

[2023] The core hardware of the system consists of a server, a user device (PC, smartphone, tablet, etc.), and a microphone for voice input. The server performs the core processing, and the device provides the interface with the user.

[2024] 1. Real-time processing of conference audio data

[2025] The server converts the meeting audio data into text in real time using a speech recognition tool (e.g., Google Speech-to-Text API) and a natural language processing library (e.g., NLTK or spaCy). The converted text data is analyzed to detect silences over a certain period of time (e.g., 10 seconds).

[2026] 2. Question and agenda generation

[2027] The server uses a large-scale language model (e.g., GPT-3) to generate the next agenda item or question to be asked when silence occurs, and this generated content is notified to the participants' devices.

[2028] 3. Automatic generation of meeting minutes

[2029] The server records the audio of the meeting and transcribes it into text in real time. It then uses a summarization algorithm (such as TextRank) to extract important topics and action items, automatically generating minutes and action lists. These minutes are then sent to the participants' devices.

[2030] 4. Analyzing calendar data and sending reminders

[2031] The server analyzes the user's calendar data to understand each member's schedule, evaluates their emotional state using an emotion engine (e.g., IBM Watson Tone Analyzer), and sends progress report reminders at the optimal time.

[2032] 5. Automatic extraction and management of task data

[2033] The server automatically extracts task data from chat tools (e.g., Slack) and work breakdown structures (WBS) to identify important tasks. These important tasks are notified to the administrator's terminal. It also collects current project status data daily and evaluates each task. It generates response policies for high-risk tasks and notifies the administrator's terminal.

[2034] 6. In-factory support

[2035] It also includes a function to support meetings between workers in a factory and provide advice based on their emotional state. For example, if a worker is feeling impatient, advice such as "Take a short break. Relax before proceeding to the next task" is generated and displayed on the device.

[2036] 7. Automatic generation of work notes

[2037] After each shift, a memo summarizing the work and key points is automatically generated and sent to the worker's device, including key tasks and points requiring attention.

[2038] Examples and prompts

[2039] As a specific example of usage, to give voice instructions at the end of a shift in a factory:

[2040] "The work is progressing slowly today, and everyone seems to be getting impatient."

[2041] In this way, the system supports workers through emotion recognition, enabling efficient project management and communication.

[2042] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2043] Step 1:

[2044] The server receives the conference audio data in real time and converts the audio content into text using natural language processing technology. Specifically, it uses a speech recognition tool (such as the Google Speech-to-Text API) to convert the conference audio into text data. The input of this step is the conference audio data, and the output is the converted text data.

[2045] Step 2:

[2046] The server analyzes the converted text data and detects silences over a certain period of time (e.g., 10 seconds). It analyzes the text using a natural language processing library (e.g., NLTK or spaCy). The input of this step is the text data, and the output is the silence detection result.

[2047] Step 3:

[2048] When the server detects silence, it generates the next agenda and question to be asked and notifies the participant's device. It uses a large-scale language model (such as GPT-3) to generate appropriate agenda and questions. The input of this step is the silence detection result, and the output is the generated agenda and question.

[2049] Step 4:

[2050] The server records the audio of the conference and converts it into text in real time. The recorded audio data is converted into text data. The input of this step is the conference audio data, and the output is the converted audio data.

[2051] Step 5:

[2052] The server sends the text data to a summarization algorithm to extract important topics and action items. We use a summarization algorithm such as TextRank. The input of this step is the text data, and the output is summarized topics and action items.

[2053] Step 6:

[2054] The server automatically generates minutes and action lists based on the summarized data and sends them to the participants' terminals. The input of this step is the summarized agenda and action items, and the output is the minutes and action lists.

[2055] Step 7:

[2056] The server analyzes the user's calendar data to understand each member's schedule. It uses an emotion engine (such as IBM Watson Tone Analyzer) to evaluate the member's emotional state and sends a progress report reminder at the optimal time. The input of this step is the calendar data and emotion data, and the output is a reminder at the optimal time.

[2057] Step 8:

[2058] The server automatically extracts task data from chat tools and work breakdown structures and notifies the manager's device of important tasks. It uses chat tools (such as Slack) and WBS. The input of this step is task data, and the output is a notification of important tasks.

[2059] Step 9:

[2060] The server collects project status data daily and evaluates each task. It generates a response policy for high-risk tasks and notifies the administrator's terminal. The input of this step is the status data, and the output is the risk response policy.

[2061] Step 10:

[2062] The server supports meetings between workers in the factory and provides advice based on their emotional state. It combines speech recognition and emotion analysis models to generate appropriate advice. The input of this step is the worker's emotional data, and the output is an advice message.

[2063] Step 11:

[2064] After each shift, the server automatically generates a memo summarizing the work content and important points and sends it to the worker's device. The input for this step is the shift data and work content data, and the output is a summary memo.

[2065] Step 12:

[2066] The server reminds users about work progress and important communication matters at the optimal timing based on their emotional state. The appropriate timing is determined using an emotion engine. The inputs of this step are work progress data and emotional data, and the output is a reminder notification.

[2067] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[2068] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2069] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[2070] [Fourth embodiment]

[2071] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[2072] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[2073] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[2074] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[2075] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[2076] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[2077] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[2078] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[2079] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[2080] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[2081] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[2082] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[2083] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2084] The present invention is a system designed to support the role of a project manager (PM) in a system development project. The system uses a large-scale language model to provide multiple functions that optimize the PM's communication and project management tasks. The following describes an embodiment of the system.

[2085] Overall system overview

[2086] This system consists of a server, terminals, and users. The server performs the central processing, and the terminals provide an interface with users. Users are team members and PMs involved in the project.

[2087] Facilitation function

[2088] When a user starts a meeting, the device collects the meeting's audio data and sends it to the server in real time. The server receives the audio data and converts it into text using natural language processing technology. When the server detects silence for a certain period of time (for example, 10 seconds), it generates the next agenda item and questions to be asked and notifies the participants' devices. This allows the meeting to proceed smoothly and important topics to be discussed.

[2089] Examples:

[2090] If 10 seconds of silence occurs during a meeting, the server generates the message "Let's move on to the next agenda item. Next, let's talk about the budget review." and displays it on the participants' devices.

[2091] Minutes function

[2092] When the user ends the meeting, the device sends the audio recorded during the meeting to the server. The server converts the audio into text and sends the text to a summarization algorithm to extract important topics and action items. Based on the extracted information, minutes and action lists are automatically generated and sent to the participants' devices.

[2093] Examples:

[2094] After the meeting, the server generates minutes that include information such as "discussion on the timing of releasing new features" and "tasks for each member," and sends them to the devices of all participants.

[2095] Communication Features

[2096] The server analyzes each user's calendar data to understand their schedules, finds out when the user is not busy, and sends progress report reminders at the appropriate times. These reminders are displayed on the device.

[2097] Examples:

[2098] The server will send a reminder at 10:00 AM on the following Monday, such as "Please report on the progress of this week's tasks."

[2099] PMO Functions

[2100] Task data is extracted from chat tools and work breakdown structures (WBS) and sent to the server. The server evaluates the extracted task data and identifies important tasks. It then notifies the PM's device of important tasks and manages the task progress. Task progress is periodically checked and reminders are sent as necessary.

[2101] Examples:

[2102] The server extracts important tasks such as "preparing for weekly review" and "reporting to client" and notifies the PM's terminal.

[2103] Risk Management Function

[2104] The server collects project status data daily and evaluates the progress of each task. It identifies high-risk tasks and tasks that are expected to be delayed, and generates a response plan for them. The generated response plan is then notified to the administrator's terminal.

[2105] Examples:

[2106] The server presents specific response guidelines, such as "Progress on Task A is behind schedule, so please consider adding resources or extending the delivery date," and notifies the PM's terminal.

[2107] With the above functions, this system can significantly reduce the burden on project managers and increase the success rate of projects.

[2108] The processing flow will be explained below.

[2109] (Facilitate function)

[2110] Step 1:

[2111] The terminal uses the microphone of the terminal to collect voice data during the conference and transmits it to the server in real time.

[2112] Step 2:

[2113] The server converts the received voice data into text using natural language processing technology.

[2114] Step 3:

[2115] The server analyzes the text data and determines whether a certain period of silence (for example, 10 seconds) is detected.

[2116] Step 4:

[2117] If silence is detected, the server generates the next topic to be discussed and appropriate questions.

[2118] Step 5:

[2119] The server transmits the generated agenda and questions to the terminal.

[2120] Step 6:

[2121] The terminal displays the received agenda and questions to the user.

[2122] (Meeting minutes function)

[2123] Step 1:

[2124] The terminal records audio data during the conference and transmits the data to the server after the conference ends.

[2125] Step 2:

[2126] The server converts the received voice data into text using natural language processing technology.

[2127] Step 3:

[2128] The server feeds the generated text data into a summarization algorithm to extract important topics and action items.

[2129] Step 4:

[2130] The server automatically generates minutes and action lists based on the extracted information.

[2131] Step 5:

[2132] The server transmits the generated minutes and action list to the terminal.

[2133] Step 6:

[2134] The terminal displays the received minutes and action list to the user.

[2135] (Communication function)

[2136] Step 1:

[2137] The server periodically collects and analyzes the user's calendar data.

[2138] Step 2:

[2139] The server takes into account the user's schedule and generates progress report reminders at optimal times.

[2140] Step 3:

[2141] The server transmits the generated reminder to the terminal.

[2142] Step 4:

[2143] The terminal notifies the user of the received reminder.

[2144] (PMO function)

[2145] Step 1:

[2146] The server automatically extracts task data from chat tools and work breakdown structures (WBS) on a regular basis.

[2147] Step 2:

[2148] The server evaluates the extracted task data and identifies important tasks.

[2149] Step 3:

[2150] The server notifies the administrator terminal of the identified important task.

[2151] Step 4:

[2152] The server periodically monitors the progress of the task and sends reminders to the administrator terminal as needed.

[2153] (Risk management function)

[2154] Step 1:

[2155] The server collects project status data daily.

[2156] Step 2:

[2157] The server evaluates tasks based on the collected data and identifies high-risk tasks.

[2158] Step 3:

[2159] The server generates a response policy for the identified risks.

[2160] Step 4:

[2161] The server notifies the administrator terminal of the generated response policy.

[2162] Step 5:

[2163] The terminal displays the received response policy to the administrator.

[2164] Example 1

[2165] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2166] Conventional systems require project managers (PMs) to manually manage a wide range of tasks, conduct meetings, create minutes, and monitor the progress of each member, requiring a great deal of time and effort. This can slow down project progress and delay the early detection and response of risks. Other issues include overlooking important agenda items during meetings and insufficient follow-up afterward. There is a need for a system that can solve these problems, reduce the burden on PMs, and improve the success rate of projects.

[2167] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[2168] In this invention, the server includes: means for receiving conference audio data in real time and converting the audio content into text using natural language processing technology; means for analyzing the converted text data and detecting when the conversation has stopped for a certain period of time; means for generating next agenda items and questions and notifying participant terminals when detected; means for recording the conference audio, converting it into text in real time, sending it to a summarization algorithm to extract important agenda items and action items, automatically generating minutes and action lists and sending them to participant terminals; means for analyzing calendar data to understand each member's schedule and sending progress report reminders at optimal times; means for automatically extracting task data from a chat tool or a work breakdown structure and notifying a manager terminal of important tasks; means for collecting project status data daily, evaluating each task, generating response policies for high-risk tasks and notifying the manager terminal; and means for proposing project management actions based on the generated response policies. This automates the manual work of project managers and enables efficient project management.

[2169] "Conference audio data" refers to audio information generated during a conference, including statements and discussions.

[2170] "Natural language processing technology" is a technology that allows computers to understand and process the language that humans use on a daily basis.

[2171] "Text data" refers to text information generated from audio data using natural language processing technology.

[2172] "Conversion" is the process of switching data from one format to another, in this case changing audio data to text data.

[2173] The "certain period of time" refers to the length of time that serves as a reference in the detection process, and in this invention specifically means 10 seconds.

[2174] A "lull in conversation" refers to a state in which no one speaks for a certain period of time or longer.

[2175] An "agenda" refers to the subject or topic of a meeting or discussion.

[2176] A "question" is a question posed to obtain specific information or opinions.

[2177] "Participant terminal" refers to an electronic device such as a computer or mobile device used by a user participating in a conference.

[2178] A "summarization algorithm" is a computational procedure for extracting the important parts of long text data and presenting them in a shortened form.

[2179] "Agenda and action items" refers to the main topics covered during the meeting and any specific action items that will follow.

[2180] "Minutes" refers to a document that records the contents of a meeting.

[2181] An "action list" is a document that lists specific actions to be taken after a meeting.

[2182] "Calendar data" refers to schedule information written on an electronic calendar used by a user.

[2183] A "progress report" is a report on the progress of a project, usually done on a regular basis.

[2184] A "reminder" is information that notifies a user to remind them of a particular action or task.

[2185] A "chat tool" is software that users use to communicate instantly through text.

[2186] A work breakdown structure (WBS) is a diagram or list that breaks down project tasks hierarchically into manageable sections.

[2187] "Task data" refers to information about specific work items to be done as part of a project.

[2188] "Project status data" refers to data that indicates the current progress of a project and the status of tasks.

[2189] "Evaluation" refers to the process of analyzing data and determining its status or value based on specific criteria.

[2190] A "high-risk task" is a work item that is likely not to proceed as planned.

[2191] A "response policy" refers to a countermeasure or action plan for a specific problem or situation.

[2192] "Server" refers to a computing device that performs central processing and exchanges data with other terminals.

[2193] "Means of suggestion" refers to the method by which the system notifies the user of actions and countermeasures and supports their progress.

[2194] MODE FOR CARRYING OUT THE INVENTION

[2195] This invention is a system designed to support the role of project managers (PMs) in system development projects. This system consists of three main components: a server, a terminal, and a user, and provides multiple functions to optimize the PM's communication and project management tasks.

[2196] Hardware and software used

[2197] Server: A computer device that performs central processing and exchanges data with other devices

[2198] Terminal: An electronic device (e.g., computer, mobile device) that provides an interface with a user.

[2199] Natural language processing technology: Google Cloud Speech-to-Text API

[2200] Summarization Algorithm: BERT-Based Summarization Model

[2201] Calendar Data Analysis: Google Calendar API

[2202] Task data extraction: Slack API, JIRA API

[2203] Collecting project status data: JIRA API, GitLab API

[2204] Features and specific operation examples

[2205] Facilitation function

[2206] When a user starts a meeting, the device collects the audio data of the meeting and sends it to the server in real time. When the server receives the audio data, it converts it into text using natural language processing technology. If the server detects silence for a certain period of time (for example, 10 seconds), it generates the next agenda item and questions and notifies the participants' devices.

[2207] Example: If 10 seconds of silence occurs during a meeting, the server generates the message "Let's move on to the next agenda item. Let's discuss the budget review next." and displays it on the participants' devices.

[2208] Minutes function

[2209] When the user ends the meeting, the device sends the audio data recorded during the meeting to the server, which converts the audio data into text using natural language processing technology and sends the text data to a summarization algorithm to extract important topics and action items. Based on this, minutes and action lists are automatically generated and sent to the participants' devices.

[2210] Example: After the meeting, the server generates minutes that include "discussions about the timing of releasing new features" and "tasks for each member," and sends them to the participants' devices.

[2211] Communication Features

[2212] The server analyzes each user's calendar data to understand their schedule, finds times when the user is not busy, and generates progress report reminders at those times and sends them to each device.

[2213] Example: The server sends a reminder to participants' devices at 10:00 a.m. on the following Monday, such as "Please report on the progress of this week's tasks."

[2214] PMO Functions

[2215] Task data is extracted from chat tools and work breakdown structures (WBS) and sent to the server. The server evaluates the extracted task data and identifies important tasks. It then notifies the PM's device of important tasks and manages the task progress. Task progress is periodically checked and reminders are sent as necessary.

[2216] Example: The server extracts important tasks such as "preparing for weekly review" and "reporting to client" and notifies the PM's terminal.

[2217] Risk Management Function

[2218] The server collects project status data daily and evaluates the progress of each task. It identifies high-risk tasks and tasks that are expected to be delayed, and generates a response plan for them. The generated response plan is then notified to the administrator's terminal.

[2219] Example: The server presents a specific response policy such as "Progress on Task A is behind schedule, so please consider adding resources or extending the deadline," and notifies the PM's terminal.

[2220] Examples of prompt statements

[2221] To use this system, the following is an example of a prompt sentence to input into the generative AI model:

[2222] Meeting Facilitation Functions:

[2223] Prompt: "Detect 10 seconds of silence during a meeting. Generate next agenda items or questions."

[2224] Minutes feature:

[2225] Prompt: "Convert the audio data from a meeting recording into text and generate a summary."

[2226] Communication features:

[2227] Prompt: "Analyze the user's calendar data to find the appropriate time for task progress reports and send reminders."

[2228] PMO Functions:

[2229] Prompt: "Extract tasks from the chat tool or WBS, identify important tasks, and notify the PM."

[2230] Risk Management Functions:

[2231] Prompt: "Evaluate the progress of tasks and generate and communicate specific action plans for high-risk tasks."

[2232] In this way, by using the system of the present invention, it is possible to significantly reduce the burden on project managers and improve the success rate of projects.

[2233] The flow of the identification process in the first embodiment will be described with reference to FIG.

[2234] Facilitation function

[2235] Processing Steps

[2236] Step 1:

[2237] A user starts a conference using a terminal. The user inputs the conference start time, participant list, etc. into the system. This information becomes the initial data for the system.

[2238] Specific behavior:

[2239] The user enters the details of the meeting on the meeting setting screen and presses the "Start" button.

[2240] Step 2:

[2241] The device collects audio data during the meeting in real time using a built-in microphone or an external microphone, and stores the collected audio data in a buffer.

[2242] Specific behavior:

[2243] The device activates the microphone and begins collecting audio.

[2244] Input: Audio during the meeting

[2245] Output: Collected audio data

[2246] Step 3:

[2247] The device transmits the collected audio data to the server in real time using a secure protocol such as HTTPS.

[2248] Specific behavior:

[2249] The terminal uploads the voice data to the server at regular intervals.

[2250] Input: Collected audio data

[2251] Output: Audio data sent to the server

[2252] Step 4:

[2253] The server converts the received voice data into text using natural language processing technology (Google Cloud Speech-to-Text API).

[2254] Specific behavior:

[2255] The server calls the API and converts the audio data into text data.

[2256] Input: Audio data sent to the server

[2257] Output: Converted text data

[2258] Step 5:

[2259] The server analyzes the text data and detects silence for a certain period of time (10 seconds).

[2260] Specific behavior:

[2261] The server analyzes the time information in the text data and flags any silence of 10 seconds or more.

[2262] Input: Converted text data

[2263] Output: Silence detection flag

[2264] Step 6:

[2265] When the server detects silence, it generates the next agenda item and question to be asked and notifies the participants' terminals.

[2266] Specific behavior:

[2267] The server uses a generative AI model to generate the next agenda item or question and create a notification message.

[2268] Input: silence detection flag

[2269] Output: Generated agenda and questions

[2270] Specific behavior:

[2271] The generated agenda and questions are displayed on the participants' devices.

[2272] Minutes function

[2273] Processing Steps

[2274] Step 1:

[2275] The user ends the conference using the terminal. The end operation is performed through the terminal interface.

[2276] Specific behavior:

[2277] The user presses the "Finish" button.

[2278] Step 2:

[2279] The terminal transmits the audio data recorded during the meeting to the server.

[2280] Specific behavior:

[2281] The device uploads the recording data to the server.

[2282] Input: Recorded audio data

[2283] Output: Audio data sent to the server

[2284] Step 3:

[2285] The server converts the voice data into text using natural language processing technology.

[2286] Specific behavior:

[2287] The server calls the API and converts the audio data into text data.

[2288] Input: Audio data sent to the server

[2289] Output: Converted text data

[2290] Step 4:

[2291] The server sends the text data to a summarization algorithm (a BERT-based summarization model) to extract important topics and action items.

[2292] Specific behavior:

[2293] The server sends the data to a summary model to extract the important information.

[2294] Input: Converted text data

[2295] Output: Extracted important topics and action items

[2296] Step 5:

[2297] The server automatically generates minutes and action lists based on the extracted information.

[2298] Specific behavior:

[2299] The server automatically generates meeting minutes and action list templates by filling them with data.

[2300] Input: Extracted important agenda items and action items

[2301] Output: Auto-generated meeting minutes and action list

[2302] Step 6:

[2303] The server sends the generated minutes and action list to the participants' terminals.

[2304] Specific behavior:

[2305] The server sends the generated document to the terminal.

[2306] Input: Auto-generated meeting minutes and action list

[2307] Output: Document sent to participant's device

[2308] Communication Features

[2309] Processing Steps

[2310] Step 1:

[2311] The server analyzes each user's calendar data and keeps track of their schedules.

[2312] Specific behavior:

[2313] The server uses the Google Calendar API to retrieve and parse the user's calendar data.

[2314] Input: User's calendar data

[2315] Output: Analysis results

[2316] Step 2:

[2317] Based on the analysis results, the server identifies times when the user is not busy.

[2318] Specific behavior:

[2319] The server extracts available time slots from the analysis results.

[2320] Input: Analysis results

[2321] Output: Available time slots

[2322] Step 3:

[2323] The server generates appropriate reminders when the user is not busy and sends them to each device.

[2324] Specific behavior:

[2325] The server uses the generative AI model to create reminders and send them to the device.

[2326] Input: Available time slots

[2327] Output: The generated reminder

[2328] Specific behavior:

[2329] The reminder will be displayed on the user's device.

[2330] PMO Functions

[2331] Processing Steps

[2332] Step 1:

[2333] Task data is extracted from chat tools and work breakdown structures and sent to a server.

[2334] Specific behavior:

[2335] The server retrieves task data using the Slack API or JIRA API.

[2336] Input: Task data generated by chat tools and WBS

[2337] Output: Extracted task data

[2338] Step 2:

[2339] The server evaluates the extracted task data and identifies important tasks.

[2340] Specific behavior:

[2341] The server analyzes the task data and ranks it based on importance.

[2342] Input: Extracted task data

[2343] Output: Identified important tasks

[2344] Step 3:

[2345] The server notifies the PM's terminal of important tasks.

[2346] Specific behavior:

[2347] The server creates a notification message and sends it to the PM's terminal.

[2348] Input: Identified critical tasks

[2349] Output: Tasks notified to the PM's terminal

[2350] Step 4:

[2351] The server periodically checks the progress of each task.

[2352] Specific behavior:

[2353] The server collects task progress data daily and evaluates the current status.

[2354] Input: Task progress data

[2355] Output: Progress evaluation results

[2356] Step 5:

[2357] The server generates reminders as needed and sends them to the PM's terminal.

[2358] Specific behavior:

[2359] The server creates a reminder and sends it to the PM's device.

[2360] Input: Progress evaluation results

[2361] Output: The generated reminder

[2362] Risk Management Function

[2363] Processing Steps

[2364] Step 1:

[2365] The server collects project status data daily.

[2366] Specific behavior:

[2367] The server retrieves the latest data using the JIRA API or GitLab API.

[2368] Input: Project status data

[2369] Output: Collected data

[2370] Step 2:

[2371] The server evaluates the progress of the task based on the collected data.

[2372] Specific behavior:

[2373] The server analyzes the collected data and evaluates the progress of each task.

[2374] Input: Collected data

[2375] Output: Progress evaluation results

[2376] Step 3:

[2377] Based on the progress, the server identifies high-risk tasks and tasks that are expected to be delayed.

[2378] Specific behavior:

[2379] The server identifies risk tasks based on the progress evaluation results.

[2380] Input: Progress evaluation results

[2381] Output: Identified risk tasks

[2382] Step 4:

[2383] Once a risk is identified, the server generates a response policy.

[2384] Specific behavior:

[2385] The server uses the generative AI model to create a specific response policy.

[2386] Input: Identified Risk Tasks

[2387] Output: Generated response policy

[2388] Step 5:

[2389] The generated response policy is notified to the administrator terminal.

[2390] Specific behavior:

[2391] The server sends the response policy as a notification message to the administrator terminal.

[2392] Input: Generated response policy

[2393] Output: Response policy notified to administrator terminal

[2394] (Application example 1)

[2395] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2396] Conventional project management systems place a heavy burden on project managers, requiring a lot of time and effort, especially for running meetings, taking minutes, managing progress, and managing risks. As a result, it is difficult to send timely reminders and manage progress, especially when it comes to robot maintenance work and task management in factories, which can lead to issues that reduce project efficiency.

[2397] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2398] In this invention, the server has means for receiving conference audio data in real time and converting the audio content into text using natural language processing technology, means for analyzing the converted text data and detecting when the conversation has stopped for a certain period of time, means for generating the next agenda item and questions to be asked when this is detected and notifying the participant terminals, means for recording the conference audio, converting it into text in real time, sending it to a summarization algorithm to extract important agenda items and action items, automatically generating minutes and action lists and sending them to the participant terminals, and means for analyzing calendar data to grasp the schedule of each member and optimally The system includes a means for automatically extracting task data from chat tools and work breakdown structures and notifying the manager's terminal of important tasks, a means for collecting current project status data daily, evaluating each task, generating response policies for high-risk tasks and notifying the manager's terminal, a means for managing the progress of maintenance work using a smartphone and sending reminders, a means for supporting the progress of meetings using a facilitation function, and a means for collecting current status data within the factory, generating response policies for high-risk work and notifying the manager's terminal. This makes it possible to smoothly progress meetings, automate the creation of meeting minutes, and send timely reminders for maintenance work, significantly reducing the burden on project managers and improving the efficiency and success rate of projects.

[2399] "Conference audio data" refers to all audio data recorded during a conference.

[2400] "Natural language processing technology" refers to technology that enables computers to understand and process the natural language that humans use on a daily basis.

[2401] "Text data" refers to data obtained by converting voice data into character information.

[2402] "Detecting a pause in conversation for a certain period of time" refers to detecting that no speech has occurred within a specified period of time.

[2403] "Generating agendas and questions" refers to automatically creating next topics and questions to help keep the meeting moving.

[2404] The term "participant terminal" refers to an electronic terminal used by a user participating in a conference.

[2405] "Key Agenda and Action Items" refers to the key issues discussed at the meeting and the action items related to those issues.

[2406] "Minutes" refers to a document summarizing the matters discussed at a meeting.

[2407] An "action list" is a list of action plans and task items decided at a meeting.

[2408] "Calendar data" refers to data that records a user's schedule and plans.

[2409] "Progress report reminder" refers to a message that notifies you not to forget to report your progress.

[2410] "Chat tool" refers to an application for text-based communication.

[2411] A "work breakdown structure" refers to a structure in which a project is divided into task units.

[2412] "Task Data" refers to information related to each task in a project.

[2413] "Administrator terminal" refers to the electronic terminal used by the project manager or administrator.

[2414] "Project Status Data" means data regarding the current progress or status of a Project.

[2415] A "high-risk task" is one that is likely to be slow or have problems.

[2416] "Response policy" refers to a specific action plan or response measures to resolve the problem.

[2417] "Maintenance work progress" refers to the current progress of maintenance work on robots and other equipment being carried out within the factory.

[2418] "Facilitation function" refers to the function of supporting the progress of meetings and promoting smooth discussions.

[2419] "Factory current status data" refers to data related to the overall operational operations and equipment status within the factory.

[2420] MODE FOR CARRYING OUT THE INVENTION

[2421] System Overview

[2422] This invention is designed to support project management and communication within factories, with a particular focus on optimizing the role of the project manager. The system consists of a server, terminals (such as smartphones), and users (factory workers and managers). The system provides the following main functions:

[2423] Meeting progress facilitator function

[2424] The server receives audio from the meeting in real time via smartphones. This audio data is converted into text using a speech recognition API (e.g., Google Cloud Speech-to-Text). Next, when a certain period of silence (e.g., 10 seconds) is detected, the server uses a generative AI model (e.g., GPT-3) to generate the next agenda item and questions, and notifies the participants' smartphones. This ensures the meeting proceeds smoothly and prevents delays in work.

[2425] Specific examples

[2426] An example of a prompt generated by the server when silence occurs:

[2427] "Let's move on to the next topic. Let's talk about the progress on robot maintenance."

[2428] Automatic maintenance log summary function

[2429] When a user ends a meeting, their smartphone sends the audio recorded during the meeting to a server. The server converts the audio into text and sends the text to a summarization algorithm (e.g., BERT) to extract important topics and action items. Based on the extracted information, meeting minutes and action lists are automatically generated and sent to the participants' smartphones.

[2430] Specific examples

[2431] An example of a prompt generated by the server after a conference ends:

[2432] "Automatically generate minutes for maintenance meetings."

[2433] Maintenance work reminder function

[2434] The server analyzes each user's calendar data to understand their schedule, finds out when the user is not busy, and sends progress report reminders at the appropriate times. These reminders are displayed on the user's smartphone.

[2435] Specific examples

[2436] Example of a prompt when sending a reminder:

[2437] "Don't forget to perform maintenance on your robot this week."

[2438] In-factory risk management function

[2439] The server collects current status data from within the factory daily and evaluates the progress of each task. It identifies high-risk tasks and tasks that are expected to be delayed, and generates countermeasures for them. The generated countermeasures are then sent to the manager's smartphone.

[2440] Specific examples

[2441] An example of a prompt generated by the server in case of an increased risk:

[2442] "Progress on Task A is behind schedule, so please consider adding more resources or extending the deadline."

[2443] Hardware and software used

[2444] The main hardware and software used to implement this system are as follows:

[2445] Smartphones (Apple iPhone, Android devices, etc.)

[2446] Server (Cloud-based server)

[2447] Speech Recognition API (Google Cloud Speech-to-Text)

[2448] Natural language processing engine (GPT-3)

[2449] Summarization Algorithm (BERT)

[2450] Calendar API (Google Calendar)

[2451] The combination of these elements can significantly improve project management and communication within the factory, making it more efficient and effective.

[2452] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2453] Program processing flow

[2454] Step 1:

[2455] The server receives the audio data of the meeting in real time via smartphones. The input is the audio data spoken during the meeting, and the output is that audio data. The server sends this audio data to a speech recognition API (Google Cloud Speech-to-Text) and converts it into text.

[2456] Step 2:

[2457] The terminal receives text data sent from the server. The input is converted text data, and the output is text data for necessary analysis. The server analyzes this text data and detects when the conversation has stopped for a certain period of time.

[2458] Step 3:

[2459] When the server detects silence for a certain period of time (for example, 10 seconds), it uses a generative AI model (GPT-3) to generate the next agenda item and questions to proceed with. The input is the detected silence information and the current meeting situation, and the output is text data of the generated agenda item and questions. The generated agenda item and questions are notified to the participants' smartphones.

[2460] Step 4:

[2461] When a user ends a meeting, the device sends the recorded meeting audio data to the server. The input is the recording data during the meeting, and the output is the audio data. The server converts the audio back into text and sends the text data to a summarization algorithm (BERT) to extract important topics and action items.

[2462] Step 5:

[2463] The server automatically generates meeting minutes and action lists based on the extracted important agenda items and action items. The input is the extracted agenda items and action items, and the output is text data of the meeting minutes and action list summarizing the entire meeting. The generated meeting minutes and action list are sent to the participants' smartphones.

[2464] Step 6:

[2465] The server analyzes each user's calendar data. The input is the user's calendar data, and the output is the analysis result. The server finds out when the user is not busy and generates progress report reminders.

[2466] Step 7:

[2467] The server sends the generated reminder to the user's smartphone at the appropriate time. The input is the generated reminder, and the output is the notification message sent to the user.

[2468] Step 8:

[2469] The server collects current status data from within the factory daily and evaluates the progress of tasks. The input is various data from within the factory (sensor information, work logs, etc.), and the output is the evaluation result of the progress of each task.

[2470] Step 9:

[2471] Based on the progress evaluation results, the server identifies high-risk tasks and tasks showing signs of delay and generates a response policy...

Claims

1. A means for receiving conference audio data in real time and converting the audio content into text using natural language processing technology; A means for analyzing the converted text data and detecting a pause in the conversation for a certain period of time; When detected, a means to generate the next agenda item or question to be asked and notify the participant's device; A means to record the audio of meetings, convert it into text in real time, send it to a summarization algorithm to extract important topics and action items, automatically generate minutes and action lists, and send them to participants' devices. A way to analyze calendar data to understand each member's schedule and send progress report reminders at the optimal time. A means to automatically extract task data from chat tools and work breakdown structures and notify important tasks to the administrator's terminal, A means for collecting project status data daily, evaluating each task, generating a response policy for high-risk tasks, and notifying the administrator terminal; A system including:

2. 2. The system according to claim 1, further comprising means for automatically extracting task data from a chat tool or a work breakdown structure and notifying an administrator terminal of important tasks.

3. 2. The system according to claim 1, further comprising means for collecting project status data daily, evaluating each task, generating a response policy for a high-risk task, and notifying the manager's terminal of the policy.

Images