System

Abstract

A system is provided.SOLUTION: A system comprising: means for inputting a goal of a conference; means for generating an assumed agenda based on the input goal of the conference; means for recognizing a voice, a text, an image, and a moving image in real time during the conference and supporting conference progress; means for clarifying a decision item and a next action at the end of the conference; and means for automatically creating minutes and automatically sharing the minutes with conference participants.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Issues include unclear meeting objectives and agendas, meetings taking too long due to lack of smooth progress, and unclear decisions and next steps after the meeting. Additionally, creating meeting minutes takes time and their management is cumbersome. These issues result in poor quality meetings and a waste of time. [Means for solving the problem]

[0005] This invention provides a system that automatically generates a hypothetical agenda based on meeting goals entered before the meeting. Furthermore, it supports the progress of the meeting by recognizing voice, text, images, and video in real time during the meeting. At the end of the meeting, it also clarifies decisions and next actions, automatically creates minutes, and automatically shares them with meeting participants. This system can improve the quality of meetings while reducing the time they take.

[0006] A "meeting goal" indicates the purpose of the meeting or the specific results to be achieved.

[0007] A "planned agenda" is a set agenda of topics and order of proceedings to be discussed in a meeting.

[0008] "Real-time voice, text, image, and video recognition" means instantly processing and analyzing various forms of data generated during a meeting.

[0009] "Supporting the progress of a meeting" means assisting the meeting to proceed smoothly.

[0010] "Decisions" refers to matters decided during the meeting.

[0011] "Next actions" refers to specific tasks or procedures that should be carried out after the meeting.

[0012] "Automatically creating minutes" means that the system automatically creates minutes based on statements and decisions made during the meeting.

[0013] "Automatically share with meeting participants" means that the system automatically distributes the created minutes to meeting participants. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0015] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0018] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0019] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0020] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0022] [First embodiment]

[0023] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0024] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0027] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0030] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0034] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0035] A detailed embodiment of the "meeting supporter" according to the present invention will now be described. This system includes a server, a user terminal, and a software program for supporting a meeting through a network.

[0036] Meeting goal setting

[0037] User

[0038] The user logs in to the application from the terminal to be used.

[0039] Before the meeting, the user inputs the goal of the meeting in a text field. For example, the user might input "Goal of the new product planning meeting: Finalize specifications."

[0040] server

[0041] The server receives the meeting goals input by the user and stores them in a database.

[0042] The server collects relevant data based on the goal information.

[0043] Creating a projected agenda

[0044] server

[0045] The server uses natural language processing algorithms to generate a potential agenda based on the meeting goals.

[0046] For example, topics such as "Consider new product specifications," "Share market research results," and "Confirm design proposals" are automatically generated.

[0047] User

[0048] The user can check the generated planned agenda through the terminal and make corrections as necessary, for example, adding "Confirm manufacturing schedule."

[0049] Idea submission

[0050] server

[0051] The server presents ideas related to the meeting goals based on information collected from databases and the Internet.

[0052] The ideas presented include a list of features required for new products and improvements to existing products.

[0053] User

[0054] The user reflects the provided ideas in the content of the meeting.

[0055] In-meeting support

[0056] server

[0057] The server recognizes what users say during the meeting in real time and converts it into text.

[0058] The server monitors the progress of the meeting and ensures that the agenda is being followed.

[0059] Terminal

[0060] The terminal displays the textual utterances to the user in real time.

[0061] For example, the terminal may notify, "This topic has been completed. We will move on to the next topic."

[0062] User

[0063] Users can smoothly conduct the conference based on the display on their terminals.

[0064] Actions at the end of a meeting

[0065] server

[0066] The server automatically generates "decisions" and "next actions" based on what is said during the meeting.

[0067] For example, specific actions such as "Add feature A: Responsible person B, Deadline: next Friday" are listed.

[0068] Terminal

[0069] The terminal presents the generated decisions and next actions to the user, allowing them to review and modify them.

[0070] User

[0071] The user reviews the proposed decisions and next actions and makes any necessary corrections.

[0072] Automatically create and share meeting minutes

[0073] server

[0074] The server automatically creates meeting minutes including decisions and next actions.

[0075] The server automatically shares the created minutes with meeting participants.

[0076] Terminal

[0077] The terminal notifies the user of the shared minutes and prompts him / her to confirm them.

[0078] User

[0079] Users can check the shared minutes on their own devices and prepare for their next actions.

[0080] Specific examples

[0081] The following steps are taken during the new product planning meeting:

[0082] 1. Setting meeting goals

[0083] The user enters "Goal of new product planning meeting: Finalize specifications."

[0084] The server stores this.

[0085] 2. Creating a planned agenda

[0086] The server automatically generates tasks such as "Consider new product specifications," "Share market research results," and "Confirm design proposals," and the user adds "Confirm manufacturing schedule."

[0087] 3. Idea submission

[0088] The server provides related ideas, and users use them to prepare for the meeting.

[0089] 4. Support during meetings

[0090] The server converts the comments into text in real time and the device displays it.

[0091] The user follows the instructions on the terminal to proceed with the conference.

[0092] 5. Actions at the end of the meeting

[0093] The server generates the decisions and next actions, which the terminal presents to the user.

[0094] The user checks and corrects the data.

[0095] 6. Automatically create and share meeting minutes

[0096] The server automatically creates minutes and shares them with participants.

[0097] The terminal notifies the user, and the user checks the minutes.

[0098] In this way, the present invention can improve the quality of the conference and reduce the time it takes to hold the conference.

[0099] The processing flow will be explained below.

[0100] Step 1:

[0101] A user logs into the application using a terminal.

[0102] Step 2:

[0103] The user inputs the goal of the meeting. For example, the user might input "Goal of the new product planning meeting: Finalize specifications."

[0104] Step 3:

[0105] The server receives the meeting goals input by the user.

[0106] Step 4:

[0107] The server stores the received meeting goals in a database.

[0108] Step 5:

[0109] The server automatically generates a hypothetical agenda based on the meeting goals, such as "discussing new product specifications," "sharing market research results," and "confirming design proposals."

[0110] Step 6:

[0111] The user checks the generated assumed agenda through the terminal.

[0112] Step 7:

[0113] The user can modify the expected agenda as needed, for example adding "Confirm manufacturing schedule."

[0114] Step 8:

[0115] The server collects relevant ideas from a stored database or from the Internet.

[0116] Step 9:

[0117] The server presents the collected ideas to the user, such as a list of features required for a new product or improvements to an existing product.

[0118] Step 10:

[0119] The user uses the presented ideas as a reference to prepare the contents of the meeting.

[0120] Step 11:

[0121] The server converts users' comments into text in real time during the meeting using voice recognition technology.

[0122] Step 12:

[0123] The server monitors the progress of the conference and checks whether the agenda is being followed properly.

[0124] Step 13:

[0125] The terminal displays the content of the speech converted into text by speech recognition to the user.

[0126] Step 14:

[0127] The terminal displays suggestions and notes for the user regarding the progress of the meeting in real time, such as "This agenda item has been completed. Let's move on to the next agenda item."

[0128] Step 15:

[0129] The user follows the instructions on the terminal to ensure the conference proceeds smoothly.

[0130] Step 16:

[0131] The server automatically generates "decisions" and "next actions" based on what was said during the meeting. For example, it lists specific actions such as "Add feature A: Person in charge: B, deadline: next Friday."

[0132] Step 17:

[0133] The terminal presents the generated decisions and next actions to the user.

[0134] Step 18:

[0135] The user reviews the proposed decisions and next actions and makes any necessary corrections.

[0136] Step 19:

[0137] The server automatically creates meeting minutes based on confirmed decisions and next actions.

[0138] Step 20:

[0139] The server automatically shares the created minutes with the meeting participants.

[0140] Step 21:

[0141] The terminal notifies the user of the shared minutes.

[0142] Step 22:

[0143] The user receives the minutes from the terminal and prepares for the next action.

[0144] By dividing the steps in this way, the system of the present invention improves the quality of the conference and reduces the time it takes to hold the conference.

[0145] Example 1

[0146] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0147] In many modern meetings, many processes, such as proceeding, taking minutes, and deciding on post-meeting actions, are done manually, resulting in a decrease in meeting efficiency. It is also difficult for all participants to understand the content and progress of the meeting, which often leads to a decline in meeting quality. There is a need for technology that can solve this problem and make meetings more effective.

[0148] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0149] In this invention, the server includes: means for a user to input the goal of the meeting; means for the server to save the input goal of the meeting; means for the server to generate a hypothetical agenda based on the goal of the meeting using a natural language processing algorithm; means for the user to confirm and modify the generated hypothetical agenda; means for the server to collect information related to the goal of the meeting from a database and the Internet and provide ideas; means for the server to recognize and convert user comments in real time during the meeting into text; means for a terminal to display the converted text in real time; means for the server to generate decisions and next actions based on the content of the meeting at the end of the meeting; means for the server to present the generated decisions and next actions to the user; and means for the server to automatically create minutes and share them with meeting participants. This automates the process of understanding the progress and decisions of the meeting, and the creation and sharing of minutes, thereby improving the efficiency and quality of meetings.

[0150] The "meeting goal" represents the specific objective or purpose that the meeting aims to achieve.

[0151] "User" refers to the person who sets up and conducts the meeting and operates each function of the system.

[0152] "Terminal" means a hardware device used by a User to access the System, enter information, and receive notifications from the System.

[0153] "Server" means a central computer facility that manages the entire system and performs data processing.

[0154] "Text conversion" refers to the process of converting audio data into text data.

[0155] "Natural language processing algorithms" are technical methods that allow computers to understand and generate human language, including grammatical analysis and semantic analysis.

[0156] "Database" refers to a digital repository used to efficiently store and retrieve structured data.

[0157] "Internet" refers to the global network that interconnects computer networks around the world.

[0158] "Proceeding a meeting" refers to activities to ensure that each agenda item is properly advanced according to the meeting plan.

[0159] "Decisions" refer to specific actions or decisions agreed upon during a meeting.

[0160] "Next actions" refer to specific tasks or steps to be taken after the meeting.

[0161] "Minutes" refers to an official document that records what was discussed, decisions made, and next actions taken during a meeting.

[0162] The "Complete Meeting Supporter" of this invention is a system that includes a server, a user terminal, and a software program for supporting meetings over a network. This system improves the efficiency and quality of meetings through the following steps. The main hardware used is a user terminal (e.g., a personal computer, tablet, or smartphone) and a server. The software technologies used include natural language processing algorithms, database management systems, and real-time voice recognition technology.

[0163] Meeting goal setting

[0164] User

[0165] The user logs into the application from their own terminal.

[0166] After logging in to the application, the user enters the goal of the meeting in a text field, for example, "Goal of new product planning meeting: Finalize specifications."

[0167] server

[0168] The server receives the meeting goals entered by the user and stores them in a database.

[0169] After saving, the server begins to collect relevant data based on the goals of the meeting.

[0170] Creating a projected agenda

[0171] server

[0172] The server uses natural language processing algorithms to generate a potential agenda based on the meeting goals.

[0173] For example, topics such as "Consider new product specifications," "Share market research results," and "Confirm design proposals" are automatically generated.

[0174] User

[0175] The user can check the generated planned agenda through the terminal and make corrections as necessary, for example, adding "Confirm manufacturing schedule."

[0176] Idea submission

[0177] server

[0178] The server presents ideas related to the meeting goals based on information collected from databases and the Internet.

[0179] The ideas presented include a list of features required for new products and improvements to existing products.

[0180] User

[0181] The user reviews the ideas provided and incorporates them into the content of the meeting.

[0182] In-meeting support

[0183] server

[0184] The server recognizes what users say during the meeting in real time and converts it into text.

[0185] The server monitors the progress of the meeting and ensures that the agenda is being followed.

[0186] Terminal

[0187] The terminal displays the textual utterances to the user in real time.

[0188] For example, the terminal may notify, "This topic has been completed. We will move on to the next topic."

[0189] User

[0190] Users can smoothly conduct the conference based on the display on their terminals.

[0191] Actions at the end of a meeting

[0192] server

[0193] The server automatically generates "decisions" and "next actions" based on what is said during the meeting.

[0194] For example, specific actions such as "Add feature A: Responsible person B, Deadline: next Friday" are listed.

[0195] Terminal

[0196] The terminal presents the generated decisions and next actions to the user, allowing them to review and modify them.

[0197] User

[0198] The user reviews the proposed decisions and next actions and makes any necessary corrections.

[0199] Automatically create and share meeting minutes

[0200] server

[0201] The server automatically creates meeting minutes including decisions and next actions.

[0202] The minutes created are automatically shared by the server with all meeting participants.

[0203] Terminal

[0204] The terminal notifies the user that the minutes have been shared.

[0205] User

[0206] Users can check the shared minutes on their own devices and prepare for their next actions.

[0207] Specific examples

[0208] The following steps are taken during the new product planning meeting:

[0209] 1. Setting meeting goals

[0210] The user enters "Goal of new product planning meeting: Finalize specifications."

[0211] The server stores this.

[0212] 2. Creating a planned agenda

[0213] The server automatically generates tasks such as "Consider new product specifications," "Share market research results," and "Confirm design proposals," and the user adds "Confirm manufacturing schedule."

[0214] 3. Idea submission

[0215] The server provides related ideas, and users use them to prepare for the meeting.

[0216] 4. Support during meetings

[0217] The server converts the comments into text in real time and the device displays it.

[0218] The user follows the instructions on the terminal to proceed with the conference.

[0219] 5. Actions at the end of the meeting

[0220] The server generates the decisions and next actions, which the terminal presents to the user.

[0221] The user checks and corrects the data.

[0222] 6. Automatically create and share meeting minutes

[0223] The server automatically creates minutes and shares them with participants.

[0224] The terminal notifies the user, and the user checks the minutes.

[0225] In this way, the present invention can improve the quality of the conference and reduce the time it takes to hold the conference.

[0226] Prompt Sentence Examples

[0227] "Please create the following agenda for the new product planning meeting, with the goal of finalizing specifications."

[0228] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0229] Step 1: Set meeting goals

[0230] User

[0231] The user logs into the application from their own terminal.

[0232] After logging in, users enter the meeting goal in a text field within the application.

[0233] As an example of input, enter "Goal of new product planning meeting: Finalize specifications."

[0234] The entered goal is sent from the terminal to the server.

[0235] server

[0236] The server receives the goal of the meeting transmitted from the user terminal.

[0237] The received goal information is stored in a database.

[0238] Based on the saved goals, relevant data is collected and stored in a database.

[0239] Input: Meeting goal ("Goal of new product planning meeting: Finalize specifications").

[0240] Output: Meeting goal data stored in a database.

[0241] Step 2: Create a visionary agenda

[0242] server

[0243] The server runs natural language processing algorithms based on the meeting goals stored in a database.

[0244] The algorithm extracts topics related to the goal and generates a hypothetical agenda.

[0245] For example, topics such as "Consider new product specifications," "Share market research results," and "Confirm design proposals" are automatically generated.

[0246] User

[0247] The user checks the generated assumed agenda through the terminal.

[0248] If necessary, the user edits the agenda and adds a new item, for example, "Confirm manufacturing schedule."

[0249] Input: Meeting goal data stored in a database.

[0250] Output: An automatically generated proposed agenda.

[0251] Step 3: Submit your idea

[0252] server

[0253] The server collects information related to the meeting goals from a database or the Internet.

[0254] From the collected information, ideas related to the content of the meeting are generated and presented to the user.

[0255] The ideas presented include a list of features required for new products and improvements to existing products.

[0256] User

[0257] The user checks the provided ideas through the terminal.

[0258] Users can incorporate relevant ideas into the meeting agenda.

[0259] Input: Collected information based on meeting goal data.

[0260] Output: A list of ideas presented to the user.

[0261] Step 4: Support during the meeting

[0262] server

[0263] The server recognizes what users say during the meeting in real time and converts it into text.

[0264] A speech recognition algorithm is used to recognize what is being said.

[0265] The textual comments are sent to a server and recorded.

[0266] The server monitors the progress of the meeting and ensures that the agenda is being followed.

[0267] Terminal

[0268] The terminal displays real-time textual comments sent from the server.

[0269] The device notifies meeting participants of the progress, for example by displaying a notification such as "This topic has been completed. We will move on to the next topic."

[0270] User

[0271] The user proceeds with the conference based on notifications from the terminal.

[0272] Input: User's spoken utterance.

[0273] Output: Server-generated text of speech data and progress notifications.

[0274] Step 5: Actions at the end of the meeting

[0275] server

[0276] The server aggregates the recorded comments made during the meeting and automatically generates "decisions" and "next actions."

[0277] For example, specific actions such as "Add feature A: Responsible person B, Deadline: next Friday" are listed.

[0278] Terminal

[0279] The terminal presents the generated decisions and next actions to the user.

[0280] The user checks these contents and makes corrections as necessary.

[0281] User

[0282] The user reviews the proposed decisions and next actions and applies appropriate corrections.

[0283] Input: Speech data recorded during the meeting.

[0284] Output: Auto-generated decisions and next action list.

[0285] Step 6: Automatically create and share meeting minutes

[0286] server

[0287] The server automatically creates minutes based on the decisions made during the meeting and the next actions.

[0288] The created minutes are shared with all meeting participants by the server.

[0289] Terminal

[0290] The terminal notifies the user that the minutes have been shared.

[0291] Users can review the shared minutes on their own devices and provide feedback or corrections as needed.

[0292] User

[0293] The user checks the minutes and prepares for the next action.

[0294] Input: Decision and next action data.

[0295] Output: Automatically generated meeting minutes.

[0296] (Application example 1)

[0297] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0298] In modern factories, efficient meeting management and work planning are essential to improving work efficiency and reducing errors. However, existing meeting support systems are primarily designed for office work and are often not suitable for use on factory floors. Furthermore, while it is required that comments be recorded in real time during meetings and that next actions be smoothly shared, this is not easy to achieve. Furthermore, there is a lack of a system that allows relevant information and ideas based on goals to be quickly provided, further improving work efficiency on-site.

[0299] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0300] In this invention, the server includes: [means for inputting meeting goals]; [means for generating a hypothetical agenda based on the input meeting goals]; [means for recognizing voice, text, images, and video in real time during the meeting to support the progress of the meeting]; [means for clarifying decisions and next actions at the end of the meeting]; [means for automatically creating minutes and automatically sharing them with meeting participants]; [means for providing relevant ideas based on the goals of factory work and recording comments with timestamps]; and [means for monitoring the progress of factory work and displaying important comments and action items in real time]. This enables meetings to be run efficiently, improving factory work efficiency and reducing errors. Furthermore, by clarifying and quickly sharing next actions, work efficiency on the factory floor can be further improved.

[0301] A "meeting goal" indicates the specific purpose or objective to be achieved in the meeting.

[0302] A "potential agenda" is a list of topics prepared in advance based on the goals of the meeting.

[0303] "Speech recognition" is a technology that converts speech spoken during a meeting into text in real time.

[0304] "Text conversion" is the process of converting voice data obtained by voice recognition into text information.

[0305] "Image recognition" is a technology that analyzes image data used during a meeting and extracts necessary information.

[0306] "Video recognition" is a technology that analyzes video data during a meeting and identifies important scenes, people, and objects.

[0307] "Decisions" are specific actions or policies agreed upon during a meeting.

[0308] "Next actions" are specific tasks or steps that need to be carried out based on decisions.

[0309] Minutes are a record of what was said during a meeting, decisions made, next actions, etc.

[0310] "Factory work" refers to activities such as production, maintenance, and inspection that take place within a factory.

[0311] A "timestamp" is a mark that indicates the date and time when the information was generated.

[0312] "Real-time display" means the immediate display of ongoing information or data.

[0313] Ideation is the process of offering helpful suggestions or thoughts related to the goals of a meeting or task.

[0314] The "factory work support system" of the present invention provides technology to significantly improve the efficiency of meetings and discussions in factories. This system includes a server, user terminals, and a software program for supporting meetings in factories via a network.

[0315] The server generates a hypothetical agenda based on the meeting goals entered by the user, and recognizes and analyzes voice, text, images, and video in real time to support the progress of the meeting. This allows important comments and decisions made during the meeting to be instantly recorded, and the next action to be taken is automatically clarified at the end of the meeting. Furthermore, the system also has a function to automatically create meeting minutes and share them with meeting participants. It also has a function to provide relevant ideas based on the goal of work within the factory and record comments with timestamps.

[0316] Hardware and Software Configuration

[0317] The main components of the system are:

[0318] Server: Equipped with a high-performance processor and large-capacity storage, it processes and stores data related to meeting support.

[0319] User terminal: A device that can be worn or carried by a factory worker, such as a smartphone, smart glasses, or head-mounted display.

[0320] Software: Programs implemented using Python, natural language processing libraries (e.g., spaCy), and generative AI models (e.g., GPT-3).

[0321] Data Processing and Computation

[0322] The server manages a series of processes from setting meeting goals to automatically creating meeting minutes. The main data processing and calculations are explained below:

[0323] 1. Setting the goal of the meeting: The user inputs the goal of the meeting through the terminal. The server receives and stores the information.

[0324] 2. Generation of the expected agenda: The server generates the expected agenda based on the input goal information using a natural language processing algorithm.

[0325] 3. Real-time recognition: The server recognizes and analyzes the voice, text, images, and video during the meeting in real time and displays them on the terminal.

[0326] 4. Recording speech: User speech is converted into text in real time and saved with a timestamp.

[0327] 5. Providing related ideas: The server provides ideas related to the goal based on information collected from databases and the Internet.

[0328] 6. Generation of next actions: At the end of the meeting, the server automatically generates decisions and next actions and displays them on the terminal.

[0329] 7. Automatic minutes creation: The server creates minutes based on all meeting records and automatically shares them with participants.

[0330] Specific examples

[0331] Below is a specific example from a factory meeting regarding the introduction of safety devices for a new product.

[0332] 1. Input example: The user inputs "Meeting regarding the introduction of new product safety devices" into the terminal.

[0333] 2. Generation example: The server automatically generates agendas such as "Confirmation of work procedures," "Discussion of safety measures," and "Proposals for improving efficiency."

[0334] 3. Example of idea provision: The server provides "ideas for introducing new safety devices," and users proceed with the meeting based on these ideas.

[0335] 4. Example of real-time recording: When a user says during a meeting, "Let's consider automation to improve efficiency," the statement is converted into text in real time and recorded with a timestamp.

[0336] 5. Example of next action generation: The server automatically generates and presents an action such as "Testing new safety device: Engineering department, deadline: 2023-12-01."

[0337] 6. Example of automatic minutes creation: After the meeting ends, the server automatically creates minutes including all statements, decisions, and next actions, and shares them with the participants.

[0338] Prompt Sentence Examples

[0339] "Please tell us your ideas for implementing new safety equipment."

[0340] As described above, the present invention enables meetings and discussions within a factory to be conducted efficiently, improving work efficiency and reducing mistakes.

[0341] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0342] Step 1:

[0343] The user inputs the goal of the meeting. Specifically, the user logs into the application from their terminal and inputs the goal of the meeting into a text field. For example, the user might input "Meeting regarding the introduction of new product safety devices." The input goal information is sent to the server. The server receives and saves the input goal information. Input: Meeting goal input by the user, Output: Saved goal information.

[0344] Step 2:

[0345] The server generates a hypothetical agenda based on the input meeting goals. Specifically, the server analyzes goal information using a natural language processing algorithm and automatically generates an agenda such as "confirmation of work procedures," "discussion of safety measures," and "proposals for improving efficiency." The generated agenda is sent to the user's device. Input: saved goal information, Output: generated hypothetical agenda.

[0346] Step 3:

[0347] The user checks the generated expected agenda through the terminal and modifies it as necessary. For example, adding "test plan for new product" as an "additional consideration item." The modified agenda is sent back to the server and saved. Input: Generated expected agenda, Output: Modified agenda.

[0348] Step 4:

[0349] The server provides ideas related to the goals and agenda. Specifically, it uses a generative AI model (e.g., GPT-3) to collect relevant information from databases and the Internet and generate specific proposals such as "ideas for introducing new safety devices." These ideas are sent to the user's device. Input: goal information and agenda, output: provided ideas.

[0350] Step 5:

[0351] During a meeting, the server recognizes voice, text, images, and video in real time to support the progress of the meeting. Specifically, it uses voice recognition technology to convert comments into text, and image and video recognition technology to analyze important scenes. The recognized information is displayed on the device in real time, prompting the user to move on to the next agenda item as necessary. Input: Voice, text, images, and video data during the meeting. Output: Text comments displayed in real time and notifications about the next agenda item.

[0352] Step 6:

[0353] The server records user comments with timestamps and clarifies decisions and next actions at the end of the meeting. Specifically, it analyzes the content of comments and generates specific action items such as "Test new safety devices: Engineering department, deadline: 2023-12-01." The generated action items are sent to the user's device, where they can be checked and modified. Input: Texted comments, Output: Decisions and next actions.

[0354] Step 7:

[0355] The server automatically creates minutes based on all records of the meeting. Specifically, it compiles the recorded remarks, decisions, and next actions, and generates minutes according to a format. The generated minutes are automatically shared with meeting participants, and notifications are sent to user devices. Input: Recorded remarks, decisions, next actions. Output: Automatically created minutes and shared notifications.

[0356] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0357] This section describes a detailed embodiment of a system that combines an emotion engine with the "meeting supporter" of the present invention. This system supports meetings using a software program that includes a server, user terminals, a network, and an emotion engine.

[0358] Meeting goal setting

[0359] User

[0360] The user logs in to the application from the terminal to be used.

[0361] Before the meeting, the user inputs the goal of the meeting in the text field. For example, the user might input "Goal of the new product planning meeting: Finalize specifications."

[0362] server

[0363] The server receives the meeting goals input by the user.

[0364] The server stores the received meeting goals in a database.

[0365] Creating a projected agenda

[0366] server

[0367] The server uses natural language processing algorithms to generate a potential agenda based on the meeting goals.

[0368] For example, it automatically generates agenda items such as "Consider new product specifications," "Share market research results," and "Confirm design proposals."

[0369] User

[0370] The user can check the generated planned agenda through the terminal and make corrections as necessary, for example, adding "Confirm manufacturing schedule."

[0371] Idea submission

[0372] server

[0373] The server presents ideas related to the meeting goals based on information collected from databases and the Internet.

[0374] The ideas presented include a list of features required for new products and improvements to existing products.

[0375] User

[0376] The user reflects the provided ideas in the content of the meeting.

[0377] In-meeting support

[0378] server

[0379] The server converts users' comments into text in real time during the meeting using voice recognition technology.

[0380] The server monitors the progress of the meeting and checks whether the agenda is being followed properly.

[0381] Terminal

[0382] The terminal displays the textual utterances to the user in real time.

[0383] For example, the terminal may notify, "This topic has been completed. We will move on to the next topic."

[0384] Emotion recognition and response using emotion engine

[0385] server

[0386] The server uses an emotion engine to analyze the user's voice, facial expression, and text data during the meeting to recognize the user's emotional state.

[0387] For example, it detects when a user is in a dissatisfied state.

[0388] Terminal

[0389] The device prompts the user in real time to respond appropriately based on their emotional state, for example by notifying them that "The speaker seems dissatisfied. Please provide additional explanation."

[0390] User

[0391] The user checks the instructions presented by the device and responds based on their emotional state.

[0392] Actions at the end of a meeting

[0393] server

[0394] The server automatically generates "decisions" and "next actions" based on what is said during the meeting. For example, it lists specific actions such as "Add feature A: Person in charge B, deadline: next Friday."

[0395] Terminal

[0396] The terminal presents the generated decisions and next actions to the user.

[0397] User

[0398] The user reviews the proposed decisions and next actions and makes any necessary corrections.

[0399] Automatically create and share meeting minutes

[0400] server

[0401] The server automatically creates meeting minutes including decisions and next actions.

[0402] The server automatically shares the created minutes with meeting participants.

[0403] Terminal

[0404] The terminal notifies the user of the shared minutes and prompts him / her to confirm them.

[0405] User

[0406] The user can check the shared minutes on their device and prepare for the next action.

[0407] Specific examples

[0408] A specific example of how a new product planning meeting might proceed is shown below.

[0409] 1. Setting meeting goals

[0410] The user enters "Goal of new product planning meeting: Finalize specifications."

[0411] The server stores this.

[0412] 2. Creating a planned agenda

[0413] The server automatically generates tasks such as "Consider new product specifications," "Share market research results," and "Confirm design proposals," and the user adds "Confirm manufacturing schedule."

[0414] 3. Idea submission

[0415] The server provides related ideas, and the user prepares the meeting content based on them.

[0416] 4. Support during meetings

[0417] The server converts the comments into text in real time and the device displays it.

[0418] The user follows the instructions on the terminal to proceed with the conference.

[0419] 5. Emotion recognition and response using an emotion engine

[0420] The server uses an emotion engine to recognize the user's emotional state, and the device displays appropriate instructions.

[0421] For example, if the user appears dissatisfied, the terminal will notify the user that "additional explanation is required."

[0422] 6. Actions at the end of the meeting

[0423] The server generates the decisions and next actions, which the terminal presents to the user.

[0424] The user checks and corrects the data.

[0425] 7. Automatically create and share meeting minutes

[0426] The server automatically creates minutes and shares them with participants.

[0427] The terminal notifies the user, and the user checks the minutes.

[0428] This system can improve the quality of meetings and reduce meeting times. In addition, by using an emotion engine, it is possible to respond appropriately to the emotions of meeting participants, providing a better meeting environment.

[0429] The processing flow will be explained below.

[0430] Step 1:

[0431] A user logs into the application using a terminal.

[0432] Step 2:

[0433] The user inputs the goal of the meeting. For example, the user might input "Goal of the new product planning meeting: Finalize specifications."

[0434] Step 3:

[0435] The server receives the meeting goals input by the user and stores them in a database.

[0436] Step 4:

[0437] The server uses natural language processing algorithms to generate a potential agenda based on the received meeting goals.

[0438] Step 5:

[0439] The server automatically generates agenda items such as "Consider new product specifications," "Share market research results," and "Confirm design proposals."

[0440] Step 6:

[0441] The user can check the generated anticipated agenda on their device and make corrections or additions as necessary. For example, adding "Confirm manufacturing schedule" is possible.

[0442] Step 7:

[0443] The server collects relevant ideas from databases and the Internet and presents them to the user.

[0444] Step 8:

[0445] The user uses the ideas presented by the server as a reference to prepare the contents of the meeting.

[0446] Step 9:

[0447] The server converts user utterances into text in real time during the conference using a voice recognition engine.

[0448] Step 10:

[0449] The server transmits the textual content of the speech to the terminal and displays it.

[0450] Step 11:

[0451] The device uses voice recognition to convert what is said into text and displays it in real time.

[0452] Step 12:

[0453] The server monitors the progress of the conference and checks whether the agenda is being followed properly.

[0454] Step 13:

[0455] The terminal notifies the user in real time of suggestions and warnings regarding the progress of the meeting, for example, by displaying "This topic has been completed. Let's move on to the next topic."

[0456] Step 14:

[0457] The user follows the instructions on the terminal to proceed with the conference.

[0458] Step 15:

[0459] The server uses an emotion engine to analyze the user's emotional state during the conference.

[0460] Step 16:

[0461] The server generates instructions for adjusting the progress and responses of the meeting based on the user's emotion data recognized by the emotion engine.

[0462] Step 17:

[0463] The device displays the user's emotional state as recognized by the emotion engine and prompts the user to take appropriate action, such as notifying the user that "The speaker seems dissatisfied. Please provide additional explanation."

[0464] Step 18:

[0465] The user follows the instructions on the terminal and adjusts the progress of the conference as appropriate.

[0466] Step 19:

[0467] The server automatically generates "decisions" and "next actions" based on what was said during the meeting. For example, it lists specific actions such as "Add feature A: Person in charge: B, deadline: next Friday."

[0468] Step 20:

[0469] The terminal presents the generated decisions and next actions to the user.

[0470] Step 21:

[0471] The user reviews the proposed decisions and next actions and makes any necessary corrections.

[0472] Step 22:

[0473] The server automatically creates meeting minutes based on confirmed decisions and next actions.

[0474] Step 23:

[0475] The server automatically shares the created minutes with the meeting participants.

[0476] Step 24:

[0477] The terminal notifies the user of the shared minutes.

[0478] Step 25:

[0479] The user receives the minutes from the terminal and prepares for the next action.

[0480] Example 2

[0481] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0482] In conventional meeting support systems, meeting progress and content organization are often done manually, which takes time and effort and makes it easy for important information to be overlooked. Furthermore, many systems lack functionality to appropriately reflect participants' emotions and reactions to support the progress of meetings. This reduces meeting efficiency and makes it difficult to ensure quality.

[0483] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0484] In this invention, the server includes means for inputting the purpose of the meeting, means for generating an agenda based on the input purpose of the meeting, means for recognizing voice, text, and image data in real time during the meeting to support the progress of the meeting, means for monitoring the progress and agenda of the meeting and supporting the appropriate progress, means for analyzing the emotional state of users during the meeting and encouraging them to respond, means for generating decisions and next actions at the end of the meeting, and means for automatically creating minutes and sharing them with the meeting participants. This automates the progress and content organization of the meeting, enables appropriate responses based on the emotional states of the participants, and improves the quality and efficiency of the meeting.

[0485] The "purpose of the meeting" refers to the specific goal or outcome that the user is trying to achieve through the meeting.

[0486] An "agenda" refers to a list of specific topics or items to be addressed in the course of a meeting.

[0487] "Speech recognition" refers to the technology that uses a computer to analyze speech spoken during a meeting and convert it into text data.

[0488] "Texting" refers to the process of converting audio or video data into written information.

[0489] "Image data" refers to visual information, including facial expressions and movements of participants, acquired during a meeting.

[0490] "Support for meeting progress" refers to providing various support functions to ensure that meetings proceed smoothly based on the agenda.

[0491] "Progress monitoring" refers to the process of checking in real time whether progress during a meeting is proceeding as planned.

[0492] "Emotional state" refers to changes in participants' psychological states and emotions during a meeting.

[0493] "Encouraging a response" refers to providing real-time guidance on appropriate actions or responses based on participants' emotional state.

[0494] "Decisions" refer to specific outcomes or courses of action agreed upon during a meeting.

[0495] "Next actions" refers to the specific actions or tasks that should be taken next based on the decisions made in the meeting.

[0496] "Minutes" refers to an official document that summarizes what was discussed, decisions made, and next actions taken at a meeting.

[0497] "Auto-generation" refers to the process by which the system automatically generates data without user intervention.

[0498] "Sharing" refers to providing the generated information and data to all meeting participants.

[0499] The "Meeting Supporter" system of the present invention provides a series of functions for smoothly conducting a meeting. This system is implemented using a software program including a server, user terminals, a network, and an emotion engine.

[0500] Hardware and software used

[0501] 1. Server

[0502] The server uses a relational database (e.g., MySQL) to manage data such as meeting goals, agendas, and statements made during meetings.

[0503] Use natural language processing algorithms (e.g., GPT-3) to automatically generate an agenda from the input meeting goals.

[0504] Use speech recognition technology (e.g., Google Cloud Speech-to-Text) to convert what is said during the meeting into text in real time.

[0505] Using an emotion engine (e.g., Microsoft Azure Emotion API), it recognizes the user's emotional state in real time during a meeting and prompts them to respond.

[0506] 2. Terminal

[0507] The user terminal functions as an interface for displaying real-time information during the meeting and receiving notifications from the emotion engine.

[0508] Using a web or mobile application, users can enter meeting goals, review and revise agendas and ideas.

[0509] Processing flow

[0510] 1. Setting meeting goals

[0511] The user logs in to the application using a terminal and enters the goal of the meeting in a text field. For example, if the user enters "Goal of the new product planning meeting: Finalize specifications," the server receives this and stores it in the database.

[0512] 2. Creating a planned agenda

[0513] The server generates an agenda using a natural language processing algorithm (e.g., GPT-3) based on the saved meeting goals. For example, it automatically generates topics such as "Consider new product specifications," "Share market research results," and "Confirm design proposals." The generated agenda is sent to the device, where the user can review it and make any necessary changes.

[0514] 3. Idea submission

[0515] The server provides ideas related to the meeting goals based on information collected from databases and the Internet. For example, specific ideas such as "a list of features required for a new product" or "improvements to an existing product" can be presented to the terminal, and the user can reflect these ideas in the meeting content.

[0516] 4. Support during meetings

[0517] The server uses speech recognition technology to convert the user's speech into text in real time, which the device displays to the user, for example, notifying them that "This topic has been completed. We will move on to the next topic." The server also monitors the progress and agenda of the meeting to ensure that it is proceeding properly.

[0518] 5. Emotion recognition and response using an emotion engine

[0519] During the conference, the server uses an emotion engine to analyze data such as the user's voice, facial expression, and text to recognize their emotional state. For example, if the user is dissatisfied, the terminal will display an instruction such as "The speaker seems dissatisfied. Please provide additional explanation." The user can then respond accordingly.

[0520] 6. Actions at the end of the meeting

[0521] The server automatically generates "decisions" and "next actions" based on what was said during the meeting. For example, it lists things like "Add feature A: Person in charge: B, deadline: next Friday." These decisions are presented to the user via their terminal, who can then confirm and make any necessary corrections.

[0522] 7. Automatically create and share meeting minutes

[0523] After the meeting ends, the server automatically creates minutes including decisions and next actions, and shares them with the meeting participants via their devices. The devices then notify the users that the minutes have been shared, and the users can confirm this.

[0524] Specific examples

[0525] A specific example of how a new product planning meeting might proceed is shown below.

[0526] 1. Setting meeting goals

[0527] Enter "Goal of new product planning meeting: Finalize specifications." The server saves this.

[0528] 2. Creating a planned agenda

[0529] "Consider new product specifications," "Share market research results," and "Confirm design proposals" are automatically generated, and the user can add "Confirm manufacturing schedule."

[0530] 3. Idea submission

[0531] Related ideas are provided and users prepare meeting content based on them.

[0532] 4. Support during meetings

[0533] The device converts comments into text in real time and displays them, and users follow instructions on the device to conduct the meeting.

[0534] 5. Emotion recognition and response using an emotion engine

[0535] The emotion engine recognizes the user's emotional state and the device displays appropriate instructions. For example, if the user appears dissatisfied, the device will notify them that "additional explanation is required."

[0536] 6. Actions at the end of the meeting

[0537] Generates decisions and next actions and presents them to the user, who can then confirm and modify them.

[0538] 7. Automatically create and share meeting minutes

[0539] Minutes are automatically created and shared with participants. The device notifies the user, who can then check the minutes.

[0540] Examples of prompt statements

[0541] Example of an input prompt for a generative AI model:

[0542] The goal of the new product planning meeting is to "finalize specifications." Based on this goal, generate the next expected agenda.

[0543] If a user expresses dissatisfaction or voices dissatisfaction during a meeting, analyze the situation and suggest appropriate ways to respond.

[0544] This enables the system to generate and present appropriate agendas and emotional responses.

[0545] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0546] Step 1: Set meeting goals

[0547] Input: A user uses a terminal to log into the application and enter the meeting goal in a text field.

[0548] Specific action: The user inputs "Goal of new product planning meeting: Finalize specifications."

[0549] Data processing: The server receives the entered data and stores it in a database.

[0550] Output: Saved meeting goal data.

[0551] Step 2: Create a visionary agenda

[0552] Input: The server retrieves the saved meeting goals from the database.

[0553] Specific operation: The server uses a natural language processing algorithm (e.g., GPT-3) to generate an agenda from the meeting goals.

[0554] Data processing: Analyze meeting goal data and generate relevant agenda items.

[0555] Output: A list of agenda items such as "Consider new product specifications," "Share market research results," and "Confirm design proposals."

[0556] Step 3: Review and revise the agenda

[0557] Input: The server generates an agenda and sends it to the user's device.

[0558] Specific operation: The user checks the agenda through the terminal and makes corrections such as adding "Confirm production schedule."

[0559] Data processing: The user's modifications are sent to the server and the database is updated.

[0560] Output: The final agenda list.

[0561] Step 4: Submit your idea

[0562] Input: The server retrieves the final agenda from the database.

[0563] Specific operation: The server collects relevant information from databases and the Internet.

[0564] Data processing: Analyze the collected information and generate concrete ideas related to the agenda.

[0565] Output: A list of ideas such as "a list of features required for new products" and "improvements to existing products."

[0566] Step 5: Present your idea

[0567] Input: The server sends the generated idea list to the user's terminal.

[0568] Specific actions: The user checks the ideas presented through the terminal and reflects them in the meeting content.

[0569] Data processing: User feedback is stored in a database.

[0570] Output: Ideas reflected in the meeting content.

[0571] Step 6: Support during the meeting

[0572] Input: The terminal collects the user's speech.

[0573] Specific operation: The server uses speech recognition technology (e.g., Google Cloud Speech-to-Text) to convert the spoken voice into text.

[0574] Data processing: Converts voice data into text data in real time.

[0575] Output: Textualized speech.

[0576] Step 7: Displaying Text

[0577] Input: The server converts the speech into text and sends it to the terminal.

[0578] Specific operation: The device displays the text of the remarks to the user in real time. For example, it notifies the user, "This topic has been concluded. We will move on to the next topic."

[0579] Data processing: Converting data into a format for display.

[0580] Output: Real-time display to the user.

[0581] Step 8: Recognize and respond to emotions

[0582] Input: The device collects voice, facial expression, and text data during the meeting.

[0583] Specific operation: The server uses an emotion engine (e.g., Microsoft Azure Emotion API) to analyze the collected data and recognize the user's emotional state.

[0584] Data processing: Analyze emotional data in real time and determine the necessary response.

[0585] Output: Response instructions based on emotional state.

[0586] Step 9: Propose a response

[0587] Input: The server sends instructions based on the user's emotional state to the device.

[0588] Specific operation: The terminal notifies the user, for example, "The speaker seems dissatisfied. Please provide additional explanation."

[0589] Data processing: Converting data into a format for display.

[0590] Output: Real-time notification to the user.

[0591] Step 10: Actions at the end of the meeting

[0592] Input: The server retrieves the content of what was said during the meeting from the database.

[0593] Specific actions: The server automatically generates "decisions" and "next actions." For example, it lists specific actions such as "Add feature A: Person in charge: B, Deadline: Next Friday."

[0594] Data processing: Analyzes the content of statements and automatically generates data.

[0595] Output: List of decisions and next actions.

[0596] Step 11: Present and confirm your actions

[0597] Input: The server sends the generated decisions and next actions to the terminal.

[0598] Specific operation: The terminal displays the decision and next action to the user, and the user confirms and corrects it.

[0599] Data processing: The corrected data is sent back to the server and the database is updated.

[0600] Output: Confirmed and revised decisions and next actions.

[0601] Step 12: Automatically create and share meeting minutes

[0602] Input: The server retrieves the decision and next action from the database.

[0603] Specific operation: The server automatically creates minutes including decisions and next actions, and automatically shares them with meeting participants.

[0604] Data processing: Format the data into minutes format.

[0605] Output: Shared meeting minutes.

[0606] Step 13: Notification of minutes

[0607] Input: The server sends the minutes to each participant.

[0608] Specific operation: The terminal notifies the user that the minutes have been shared.

[0609] Data processing: Formatting of notification content.

[0610] Output: Notification to the user.

[0611] (Application example 2)

[0612] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0613] In modern business, it is important to streamline meetings and negotiations and respond to customers quickly and accurately. However, particularly in brick-and-mortar stores, sales staff often find it difficult to answer customers' questions immediately and accurately grasp and respond to their emotional state. This can lead to missed sales opportunities and a decrease in customer satisfaction. Furthermore, it is difficult to accurately record all information about the sales process and share it instantly.

[0614] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for inputting the meeting goal, a means for generating a hypothetical agenda based on the input meeting goal, a means for recognizing voice, text, images, and video in real time during the meeting to support the progress of the meeting, a means for clarifying decisions and next actions at the end of the meeting, a means for automatically creating minutes and automatically sharing them with meeting participants, a means for converting customer questions into text in real time via a display device worn by the user and providing appropriate answers, and a means for analyzing the user's emotional state and notifying the user of an optimal response based on the emotion in real time. This allows sales staff to respond to customer questions quickly and accurately and provide relevant information in real time, thereby improving customer satisfaction. Furthermore, the emotion engine allows for accurate understanding of the customer's emotional state and optimal responses, thereby providing a better sales experience.

[0615] The "means for inputting the goals of the meeting" is an interface that allows a user to input the goals and expected outcomes of the meeting into the system as text, and the system receives and stores them.

[0616] The "means for generating a potential agenda" is a function that uses a natural language processing algorithm to automatically generate a meeting agenda and progress schedule based on the input meeting goal information.

[0617] "Means to support the progress of the meeting" refers to a function in which the system recognizes voice, text, images, and video in real time during the meeting, monitors the progress of the meeting, and provides appropriate instructions and information.

[0618] "Means for clarifying decisions and next actions" refers to a function in which the system clearly indicates the specific next actions to be taken, the person in charge, deadlines, etc. based on the participants' comments and decisions made at the end of the meeting.

[0619] "Means for automatically creating meeting minutes and automatically sharing them with meeting participants" refers to a function that allows the system to automatically create meeting minutes based on comments and decisions made during the meeting and share them online with all meeting participants.

[0620] "Means of converting customer questions into text in real time through a display device worn by the user and providing appropriate answers" refers to a function that uses a display device such as a head-mounted display (HMD) worn by sales staff to convert customer questions into text using voice recognition technology, and then the system provides appropriate answers in real time based on that text.

[0621] "Means for analyzing the user's emotional state and notifying the optimal response based on that emotion in real time" refers to a function in which the system uses an emotion engine to analyze the voice, facial expression, and other data of the user or customer, and notifies the user in real time of the optimal response based on that emotional state.

[0622] The "sales supporter for brick-and-mortar stores" of the present invention supports sales in brick-and-mortar stores using a system including a server, user terminals, a network, and an emotion engine. The system aims to enable sales staff to respond to customers appropriately and quickly, thereby improving customer satisfaction.

[0623] Initial Setup

[0624] Hardware

[0625] Display device: Head-mounted display (HMD) (e.g., Magic Leap 2)

[0626] Input devices: Microphone, camera

[0627] software

[0628] Emotion Engine: Software for analyzing the emotional state of customers

[0629] Speech recognition system: a system that converts speech into text

[0630] Natural language processing algorithms: Algorithms that analyze text data and generate relevant information

[0631] Cloud server: Cloud services such as AWS

[0632] Real-time database: Firebase, etc.

[0633] System Operation

[0634] Goal setting

[0635] The user puts on the HMD and logs in. The server displays the sales target and saves it in the database.

[0636] Agenda Generation

[0637] The server generates daily tasks and product explanation scripts based on sales targets, which the user can review and adjust as needed.

[0638] Idea submission

[0639] The server provides product features and related information, which the user uses to explain the product to customers, such as the features of new products and campaign information.

[0640] Real-time Support

[0641] The server uses voice recognition technology and natural language processing algorithms to convert customer questions into text and display them on the HMD in real time. The user can then respond to the customer while checking the appropriate answer on the HMD.

[0642] emotion recognition

[0643] The server uses an emotion engine to analyze the customer's facial expressions and tone of voice to identify their emotional state. The HMD then notifies the user of the optimal response based on the customer's emotional state. For example, if the customer is dissatisfied, the HMD will notify the user that "further explanation is required."

[0644] When sales end

[0645] The server generates a list of next actions and customer follow-ups when a sale is completed, and displays it on the HMD. The user can then confirm and adjust the next actions.

[0646] Report creation and sharing

[0647] The server automatically generates a report based on sales results and customer feedback and notifies the user via the HMD. The user can then check and correct the report and take any necessary action.

[0648] Specific examples

[0649] For example, if the user aims to sell new product A, the user inputs "Today's goal: Sell 20 units of new product A" into the HMD. The server saves this and generates a task list related to the sales goal. Also, if a customer asks "What are the features of new product A?" during a sale, the server immediately converts the question into text, and the user provides an explanation based on the answer displayed on the HMD. Furthermore, if the customer shows signs of dissatisfaction, the HMD notifies the user that "Further explanation is required."

[0650] Prompt Sentence Examples

[0651] What are the features of new product A?

[0652]

[0653] My customer looks unhappy. How should I respond?

[0654] In this way, the system of the present invention supports effective and efficient customer service in physical stores, enabling sales staff to provide optimal service.

[0655] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0656] Step 1:

[0657] The user puts on the HMD and logs in. The user enters their sales target in text format into the system. The server receives the entered sales target and stores it in a cloud database. For example, the user might enter "Today's target: Sell 20 units of new product A."

[0658] Input: Sales target text input

[0659] Data processing: Save and analyze received text data

[0660] Output: Save to cloud database

[0661] Step 2:

[0662] The server uses natural language processing algorithms based on sales target information to generate scripts for daily tasks and product descriptions, which are then displayed on the HMD for the user to review and adjust as needed.

[0663] Input: Sales target information

[0664] Data processing: Script generation using natural language processing algorithms

[0665] Output: Script display on HMD

[0666] Step 3:

[0667] The server collects product features and related information and generates this information through the idea provision function. Based on the information provided, the user prepares an explanation for the customer. For example, they can provide information about the features of a new product or campaign information.

[0668] Input: Sales target and related data

[0669] Data processing: information gathering and idea generation

[0670] Output: Information displayed on the HMD

[0671] Step 4:

[0672] If a customer has a question during a sale, the user receives the question through the microphone in the HMD. The server uses voice recognition technology to convert the customer's question into text and display it on the user's HMD in real time.

[0673] Input: Customer Question

[0674] Data processing: speech recognition and text conversion

[0675] Output: Text display on HMD

[0676] Step 5:

[0677] The user provides the customer with an appropriate answer based on the questions displayed on the HMD, including the process of retrieving the most appropriate answer from the information database held by the server.

[0678] Input: Texted question

[0679] Data processing: Information retrieval from databases and optimal answer generation

[0680] Output: Display the answer on the HMD

[0681] Step 6:

[0682] The server uses an emotion engine to analyze the customer's facial expressions and tone of voice via camera and voice input, and identifies the customer's emotional state. For example, it can estimate the customer's emotions based on voice tone and facial recognition technology.

[0683] Input: Customer facial expression data, voice data

[0684] Data processing: Analysis with emotion engine

[0685] Output: Customer emotional state identification result

[0686] Step 7:

[0687] Based on the emotional state, the server notifies the user in real time on the HMD with an appropriate response, such as displaying a message like, "The customer appears dissatisfied. Further explanation is required."

[0688] Input: Customer's emotional state

[0689] Data processing: generating response instructions

[0690] Output: Real-time notification to HMD

[0691] Step 8:

[0692] The server generates a list of next actions and customer follow-ups upon sale completion, which are displayed to the user via the HMD and adjustments made as necessary.

[0693] Input: Sales completion data

[0694] Data processing: Next action generation and follow-up list creation

[0695] Output: List display on HMD

[0696] Step 9:

[0697] The server automatically generates a report based on sales results and customer feedback and notifies the user via the HMD. The user can then review the report and make any necessary corrections.

[0698] Input: Sales results and feedback

[0699] Data Processing: Report Generation

[0700] Output: Report notification to HMD

[0701] By following these steps, the "Sales Supporter for Physical Stores" enables sales staff to efficiently and effectively serve customers, improving customer satisfaction.

[0702] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0703] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0704] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0705] [Second embodiment]

[0706] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0707] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0708] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0709] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0710] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0711] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0712] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0713] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0714] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0715] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0716] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0717] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0718] A detailed embodiment of the "meeting supporter" according to the present invention will now be described. This system includes a server, a user terminal, and a software program for supporting a meeting through a network.

[0719] Meeting goal setting

[0720] User

[0721] The user logs in to the application from the terminal to be used.

[0722] Before the meeting, the user inputs the goal of the meeting in a text field. For example, the user might input "Goal of the new product planning meeting: Finalize specifications."

[0723] server

[0724] The server receives the meeting goals input by the user and stores them in a database.

[0725] The server collects relevant data based on the goal information.

[0726] Creating a projected agenda

[0727] server

[0728] The server uses natural language processing algorithms to generate a potential agenda based on the meeting goals.

[0729] For example, topics such as "Consider new product specifications," "Share market research results," and "Confirm design proposals" are automatically generated.

[0730] User

[0731] The user can check the generated planned agenda through the terminal and make corrections as necessary, for example, adding "Confirm manufacturing schedule."

[0732] Idea submission

[0733] server

[0734] The server presents ideas related to the meeting goals based on information collected from databases and the Internet.

[0735] The ideas presented include a list of features required for new products and improvements to existing products.

[0736] User

[0737] The user reflects the provided ideas in the content of the meeting.

[0738] In-meeting support

[0739] server

[0740] The server recognizes what users say during the meeting in real time and converts it into text.

[0741] The server monitors the progress of the meeting and ensures that the agenda is being followed.

[0742] Terminal

[0743] The terminal displays the textual utterances to the user in real time.

[0744] For example, the terminal may notify, "This topic has been completed. We will move on to the next topic."

[0745] User

[0746] Users can smoothly conduct the conference based on the display on their terminals.

[0747] Actions at the end of a meeting

[0748] server

[0749] The server automatically generates "decisions" and "next actions" based on what is said during the meeting.

[0750] For example, specific actions such as "Add feature A: Responsible person B, Deadline: next Friday" are listed.

[0751] Terminal

[0752] The terminal presents the generated decisions and next actions to the user, allowing them to review and modify them.

[0753] User

[0754] The user reviews the proposed decisions and next actions and makes any necessary corrections.

[0755] Automatically create and share meeting minutes

[0756] server

[0757] The server automatically creates meeting minutes including decisions and next actions.

[0758] The server automatically shares the created minutes with meeting participants.

[0759] Terminal

[0760] The terminal notifies the user of the shared minutes and prompts him / her to confirm them.

[0761] User

[0762] Users can check the shared minutes on their own devices and prepare for their next actions.

[0763] Specific examples

[0764] The following steps are taken during the new product planning meeting:

[0765] 1. Setting meeting goals

[0766] The user enters "Goal of new product planning meeting: Finalize specifications."

[0767] The server stores this.

[0768] 2. Creating a planned agenda

[0769] The server automatically generates tasks such as "Consider new product specifications," "Share market research results," and "Confirm design proposals," and the user adds "Confirm manufacturing schedule."

[0770] 3. Idea submission

[0771] The server provides related ideas, and users use them to prepare for the meeting.

[0772] 4. Support during meetings

[0773] The server converts the comments into text in real time and the device displays it.

[0774] The user follows the instructions on the terminal to proceed with the conference.

[0775] 5. Actions at the end of the meeting

[0776] The server generates the decisions and next actions, which the terminal presents to the user.

[0777] The user checks and corrects the data.

[0778] 6. Automatically create and share meeting minutes

[0779] The server automatically creates minutes and shares them with participants.

[0780] The terminal notifies the user, and the user checks the minutes.

[0781] In this way, the present invention can improve the quality of the conference and reduce the time it takes to hold the conference.

[0782] The processing flow will be explained below.

[0783] Step 1:

[0784] A user logs into the application using a terminal.

[0785] Step 2:

[0786] The user inputs the goal of the meeting. For example, the user might input "Goal of the new product planning meeting: Finalize specifications."

[0787] Step 3:

[0788] The server receives the meeting goals input by the user.

[0789] Step 4:

[0790] The server stores the received meeting goals in a database.

[0791] Step 5:

[0792] The server automatically generates a hypothetical agenda based on the meeting goals, such as "discussing new product specifications," "sharing market research results," and "confirming design proposals."

[0793] Step 6:

[0794] The user checks the generated assumed agenda through the terminal.

[0795] Step 7:

[0796] The user can modify the expected agenda as needed, for example adding "Confirm manufacturing schedule."

[0797] Step 8:

[0798] The server collects relevant ideas from a stored database or from the Internet.

[0799] Step 9:

[0800] The server presents the collected ideas to the user, such as a list of features required for a new product or improvements to an existing product.

[0801] Step 10:

[0802] The user uses the presented ideas as a reference to prepare the contents of the meeting.

[0803] Step 11:

[0804] The server converts users' comments into text in real time during the meeting using voice recognition technology.

[0805] Step 12:

[0806] The server monitors the progress of the conference and checks whether the agenda is being followed properly.

[0807] Step 13:

[0808] The terminal displays the content of the speech converted into text by speech recognition to the user.

[0809] Step 14:

[0810] The terminal displays suggestions and notes for the user regarding the progress of the meeting in real time, such as "This agenda item has been completed. Let's move on to the next agenda item."

[0811] Step 15:

[0812] The user follows the instructions on the terminal to ensure the conference proceeds smoothly.

[0813] Step 16:

[0814] The server automatically generates "decisions" and "next actions" based on what was said during the meeting. For example, it lists specific actions such as "Add feature A: Person in charge: B, deadline: next Friday."

[0815] Step 17:

[0816] The terminal presents the generated decisions and next actions to the user.

[0817] Step 18:

[0818] The user reviews the proposed decisions and next actions and makes any necessary corrections.

[0819] Step 19:

[0820] The server automatically creates meeting minutes based on confirmed decisions and next actions.

[0821] Step 20:

[0822] The server automatically shares the created minutes with the meeting participants.

[0823] Step 21:

[0824] The terminal notifies the user of the shared minutes.

[0825] Step 22:

[0826] The user receives the minutes from the terminal and prepares for the next action.

[0827] By dividing the steps in this way, the system of the present invention improves the quality of the conference and reduces the time it takes to hold the conference.

[0828] Example 1

[0829] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0830] In many modern meetings, many processes, such as proceeding, taking minutes, and deciding on post-meeting actions, are done manually, resulting in a decrease in meeting efficiency. It is also difficult for all participants to understand the content and progress of the meeting, which often leads to a decline in meeting quality. There is a need for technology that can solve this problem and make meetings more effective.

[0831] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0832] In this invention, the server includes: means for a user to input the goal of the meeting; means for the server to save the input goal of the meeting; means for the server to generate a hypothetical agenda based on the goal of the meeting using a natural language processing algorithm; means for the user to confirm and modify the generated hypothetical agenda; means for the server to collect information related to the goal of the meeting from a database and the Internet and provide ideas; means for the server to recognize and convert user comments in real time during the meeting into text; means for a terminal to display the converted text in real time; means for the server to generate decisions and next actions based on the content of the meeting at the end of the meeting; means for the server to present the generated decisions and next actions to the user; and means for the server to automatically create minutes and share them with meeting participants. This automates the process of understanding the progress and decisions of the meeting, and the creation and sharing of minutes, thereby improving the efficiency and quality of meetings.

[0833] The "meeting goal" represents the specific objective or purpose that the meeting aims to achieve.

[0834] "User" refers to the person who sets up and conducts the meeting and operates each function of the system.

[0835] "Terminal" means a hardware device used by a User to access the System, enter information, and receive notifications from the System.

[0836] "Server" means a central computer facility that manages the entire system and performs data processing.

[0837] "Text conversion" refers to the process of converting audio data into text data.

[0838] "Natural language processing algorithms" are technical methods that allow computers to understand and generate human language, including grammatical analysis and semantic analysis.

[0839] "Database" refers to a digital repository used to efficiently store and retrieve structured data.

[0840] "Internet" refers to the global network that interconnects computer networks around the world.

[0841] "Proceeding a meeting" refers to activities to ensure that each agenda item is properly advanced according to the meeting plan.

[0842] "Decisions" refer to specific actions or decisions agreed upon during a meeting.

[0843] "Next actions" refer to specific tasks or steps to be taken after the meeting.

[0844] "Minutes" refers to an official document that records what was discussed, decisions made, and next actions taken during a meeting.

[0845] The "Complete Meeting Supporter" of this invention is a system that includes a server, a user terminal, and a software program for supporting meetings over a network. This system improves the efficiency and quality of meetings through the following steps. The main hardware used is a user terminal (e.g., a personal computer, tablet, or smartphone) and a server. The software technologies used include natural language processing algorithms, database management systems, and real-time voice recognition technology.

[0846] Meeting goal setting

[0847] User

[0848] The user logs into the application from their own terminal.

[0849] After logging in to the application, the user enters the goal of the meeting in a text field, for example, "Goal of new product planning meeting: Finalize specifications."

[0850] server

[0851] The server receives the meeting goals entered by the user and stores them in a database.

[0852] After saving, the server begins to collect relevant data based on the goals of the meeting.

[0853] Creating a projected agenda

[0854] server

[0855] The server uses natural language processing algorithms to generate a potential agenda based on the meeting goals.

[0856] For example, topics such as "Consider new product specifications," "Share market research results," and "Confirm design proposals" are automatically generated.

[0857] User

[0858] The user can check the generated planned agenda through the terminal and make corrections as necessary, for example, adding "Confirm manufacturing schedule."

[0859] Idea submission

[0860] server

[0861] The server presents ideas related to the meeting goals based on information collected from databases and the Internet.

[0862] The ideas presented include a list of features required for new products and improvements to existing products.

[0863] User

[0864] The user reviews the ideas provided and incorporates them into the content of the meeting.

[0865] In-meeting support

[0866] server

[0867] The server recognizes what users say during the meeting in real time and converts it into text.

[0868] The server monitors the progress of the meeting and ensures that the agenda is being followed.

[0869] Terminal

[0870] The terminal displays the textual utterances to the user in real time.

[0871] For example, the terminal may notify, "This topic has been completed. We will move on to the next topic."

[0872] User

[0873] Users can smoothly conduct the conference based on the display on their terminals.

[0874] Actions at the end of a meeting

[0875] server

[0876] The server automatically generates "decisions" and "next actions" based on what is said during the meeting.

[0877] For example, specific actions such as "Add feature A: Responsible person B, Deadline: next Friday" are listed.

[0878] Terminal

[0879] The terminal presents the generated decisions and next actions to the user, allowing them to review and modify them.

[0880] User

[0881] The user reviews the proposed decisions and next actions and makes any necessary corrections.

[0882] Automatically create and share meeting minutes

[0883] server

[0884] The server automatically creates meeting minutes including decisions and next actions.

[0885] The minutes created are automatically shared by the server with all meeting participants.

[0886] Terminal

[0887] The terminal notifies the user that the minutes have been shared.

[0888] User

[0889] Users can check the shared minutes on their own devices and prepare for their next actions.

[0890] Specific examples

[0891] The following steps are taken during the new product planning meeting:

[0892] 1. Setting meeting goals

[0893] The user enters "Goal of new product planning meeting: Finalize specifications."

[0894] The server stores this.

[0895] 2. Creating a planned agenda

[0896] The server automatically generates tasks such as "Consider new product specifications," "Share market research results," and "Confirm design proposals," and the user adds "Confirm manufacturing schedule."

[0897] 3. Idea submission

[0898] The server provides related ideas, and users use them to prepare for the meeting.

[0899] 4. Support during meetings

[0900] The server converts the comments into text in real time and the device displays it.

[0901] The user follows the instructions on the terminal to proceed with the conference.

[0902] 5. Actions at the end of the meeting

[0903] The server generates the decisions and next actions, which the terminal presents to the user.

[0904] The user checks and corrects the data.

[0905] 6. Automatically create and share meeting minutes

[0906] The server automatically creates minutes and shares them with participants.

[0907] The terminal notifies the user, and the user checks the minutes.

[0908] In this way, the present invention can improve the quality of the conference and reduce the time it takes to hold the conference.

[0909] Prompt Sentence Examples

[0910] "Please create the following agenda for the new product planning meeting, with the goal of finalizing specifications."

[0911] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0912] Step 1: Set meeting goals

[0913] User

[0914] The user logs into the application from their own terminal.

[0915] After logging in, users enter the meeting goal in a text field within the application.

[0916] As an example of input, enter "Goal of new product planning meeting: Finalize specifications."

[0917] The entered goal is sent from the terminal to the server.

[0918] server

[0919] The server receives the goal of the meeting transmitted from the user terminal.

[0920] The received goal information is stored in a database.

[0921] Based on the saved goals, relevant data is collected and stored in a database.

[0922] Input: Meeting goal ("Goal of new product planning meeting: Finalize specifications").

[0923] Output: Meeting goal data stored in a database.

[0924] Step 2: Create a visionary agenda

[0925] server

[0926] The server runs natural language processing algorithms based on the meeting goals stored in a database.

[0927] The algorithm extracts topics related to the goal and generates a hypothetical agenda.

[0928] For example, topics such as "Consider new product specifications," "Share market research results," and "Confirm design proposals" are automatically generated.

[0929] User

[0930] The user checks the generated assumed agenda through the terminal.

[0931] If necessary, the user edits the agenda and adds a new item, for example, "Confirm manufacturing schedule."

[0932] Input: Meeting goal data stored in a database.

[0933] Output: An automatically generated proposed agenda.

[0934] Step 3: Submit your idea

[0935] server

[0936] The server collects information related to the meeting goals from a database or the Internet.

[0937] From the collected information, ideas related to the content of the meeting are generated and presented to the user.

[0938] The ideas presented include a list of features required for new products and improvements to existing products.

[0939] User

[0940] The user checks the provided ideas through the terminal.

[0941] Users can incorporate relevant ideas into the meeting agenda.

[0942] Input: Collected information based on meeting goal data.

[0943] Output: A list of ideas presented to the user.

[0944] Step 4: Support during the meeting

[0945] server

[0946] The server recognizes what users say during the meeting in real time and converts it into text.

[0947] A speech recognition algorithm is used to recognize what is being said.

[0948] The textual comments are sent to a server and recorded.

[0949] The server monitors the progress of the meeting and ensures that the agenda is being followed.

[0950] Terminal

[0951] The terminal displays real-time textual comments sent from the server.

[0952] The device notifies meeting participants of the progress, for example by displaying a notification such as "This topic has been completed. We will move on to the next topic."

[0953] User

[0954] The user proceeds with the conference based on notifications from the terminal.

[0955] Input: User's spoken utterance.

[0956] Output: Server-generated text of speech data and progress notifications.

[0957] Step 5: Actions at the end of the meeting

[0958] server

[0959] The server aggregates the recorded comments made during the meeting and automatically generates "decisions" and "next actions."

[0960] For example, specific actions such as "Add feature A: Responsible person B, Deadline: next Friday" are listed.

[0961] Terminal

[0962] The terminal presents the generated decisions and next actions to the user.

[0963] The user checks these contents and makes corrections as necessary.

[0964] User

[0965] The user reviews the proposed decisions and next actions and applies appropriate corrections.

[0966] Input: Speech data recorded during the meeting.

[0967] Output: Auto-generated decisions and next action list.

[0968] Step 6: Automatically create and share meeting minutes

[0969] server

[0970] The server automatically creates minutes based on the decisions made during the meeting and the next actions.

[0971] The created minutes are shared with all meeting participants by the server.

[0972] Terminal

[0973] The terminal notifies the user that the minutes have been shared.

[0974] Users can review the shared minutes on their own devices and provide feedback or corrections as needed.

[0975] User

[0976] The user checks the minutes and prepares for the next action.

[0977] Input: Decision and next action data.

[0978] Output: Automatically generated meeting minutes.

[0979] (Application example 1)

[0980] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0981] In modern factories, efficient meeting management and work planning are essential to improving work efficiency and reducing errors. However, existing meeting support systems are primarily designed for office work and are often not suitable for use on factory floors. Furthermore, while it is required that comments be recorded in real time during meetings and that next actions be smoothly shared, this is not easy to achieve. Furthermore, there is a lack of a system that allows relevant information and ideas based on goals to be quickly provided, further improving work efficiency on-site.

[0982] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0983] In this invention, the server includes: [means for inputting meeting goals]; [means for generating a hypothetical agenda based on the input meeting goals]; [means for recognizing voice, text, images, and video in real time during the meeting to support the progress of the meeting]; [means for clarifying decisions and next actions at the end of the meeting]; [means for automatically creating minutes and automatically sharing them with meeting participants]; [means for providing relevant ideas based on the goals of factory work and recording comments with timestamps]; and [means for monitoring the progress of factory work and displaying important comments and action items in real time]. This enables meetings to be run efficiently, improving factory work efficiency and reducing errors. Furthermore, by clarifying and quickly sharing next actions, work efficiency on the factory floor can be further improved.

[0984] A "meeting goal" indicates the specific purpose or objective to be achieved in the meeting.

[0985] A "potential agenda" is a list of topics prepared in advance based on the goals of the meeting.

[0986] "Speech recognition" is a technology that converts speech spoken during a meeting into text in real time.

[0987] "Text conversion" is the process of converting voice data obtained by voice recognition into text information.

[0988] "Image recognition" is a technology that analyzes image data used during a meeting and extracts necessary information.

[0989] "Video recognition" is a technology that analyzes video data during a meeting and identifies important scenes, people, and objects.

[0990] "Decisions" are specific actions or policies agreed upon during a meeting.

[0991] "Next actions" are specific tasks or steps that need to be carried out based on decisions.

[0992] Minutes are a record of what was said during a meeting, decisions made, next actions, etc.

[0993] "Factory work" refers to activities such as production, maintenance, and inspection that take place within a factory.

[0994] A "timestamp" is a mark that indicates the date and time when the information was generated.

[0995] "Real-time display" means the immediate display of ongoing information or data.

[0996] Ideation is the process of offering helpful suggestions or thoughts related to the goals of a meeting or task.

[0997] The "factory work support system" of the present invention provides technology to significantly improve the efficiency of meetings and discussions in factories. This system includes a server, user terminals, and a software program for supporting meetings in factories via a network.

[0998] The server generates a hypothetical agenda based on the meeting goals entered by the user, and recognizes and analyzes voice, text, images, and video in real time to support the progress of the meeting. This allows important comments and decisions made during the meeting to be instantly recorded, and the next action to be taken is automatically clarified at the end of the meeting. Furthermore, the system also has a function to automatically create meeting minutes and share them with meeting participants. It also has a function to provide relevant ideas based on the goal of work within the factory and record comments with timestamps.

[0999] Hardware and Software Configuration

[1000] The main components of the system are:

[1001] Server: Equipped with a high-performance processor and large-capacity storage, it processes and stores data related to meeting support.

[1002] User terminal: A device that can be worn or carried by a factory worker, such as a smartphone, smart glasses, or head-mounted display.

[1003] Software: Programs implemented using Python, natural language processing libraries (e.g., spaCy), and generative AI models (e.g., GPT-3).

[1004] Data Processing and Computation

[1005] The server manages a series of processes from setting meeting goals to automatically creating meeting minutes. The main data processing and calculations are explained below:

[1006] 1. Setting the goal of the meeting: The user inputs the goal of the meeting through the terminal. The server receives and stores the information.

[1007] 2. Generation of the expected agenda: The server generates the expected agenda based on the input goal information using a natural language processing algorithm.

[1008] 3. Real-time recognition: The server recognizes and analyzes the voice, text, images, and video during the meeting in real time and displays them on the terminal.

[1009] 4. Recording speech: User speech is converted into text in real time and saved with a timestamp.

[1010] 5. Providing related ideas: The server provides ideas related to the goal based on information collected from databases and the Internet.

[1011] 6. Generation of next actions: At the end of the meeting, the server automatically generates decisions and next actions and displays them on the terminal.

[1012] 7. Automatic minutes creation: The server creates minutes based on all meeting records and automatically shares them with participants.

[1013] Specific examples

[1014] Below is a specific example from a factory meeting regarding the introduction of safety devices for a new product.

[1015] 1. Input example: The user inputs "Meeting regarding the introduction of new product safety devices" into the terminal.

[1016] 2. Generation example: The server automatically generates agendas such as "Confirmation of work procedures," "Discussion of safety measures," and "Proposals for improving efficiency."

[1017] 3. Example of idea provision: The server provides "ideas for introducing new safety devices," and users proceed with the meeting based on these ideas.

[1018] 4. Example of real-time recording: When a user says during a meeting, "Let's consider automation to improve efficiency," the statement is converted into text in real time and recorded with a timestamp.

[1019] 5. Example of next action generation: The server automatically generates and presents an action such as "Testing new safety device: Engineering department, deadline: 2023-12-01."

[1020] 6. Example of automatic minutes creation: After the meeting ends, the server automatically creates minutes including all statements, decisions, and next actions, and shares them with the participants.

[1021] Prompt Sentence Examples

[1022] "Please tell us your ideas for implementing new safety equipment."

[1023] As described above, the present invention enables meetings and discussions within a factory to be conducted efficiently, improving work efficiency and reducing mistakes.

[1024] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1025] Step 1:

[1026] The user inputs the goal of the meeting. Specifically, the user logs into the application from their terminal and inputs the goal of the meeting into a text field. For example, the user might input "Meeting regarding the introduction of new product safety devices." The input goal information is sent to the server. The server receives and saves the input goal information. Input: Meeting goal input by the user, Output: Saved goal information.

[1027] Step 2:

[1028] The server generates a hypothetical agenda based on the input meeting goals. Specifically, the server analyzes goal information using a natural language processing algorithm and automatically generates an agenda such as "confirmation of work procedures," "discussion of safety measures," and "proposals for improving efficiency." The generated agenda is sent to the user's device. Input: saved goal information, Output: generated hypothetical agenda.

[1029] Step 3:

[1030] The user checks the generated expected agenda through the terminal and modifies it as necessary. For example, adding "test plan for new product" as an "additional consideration item." The modified agenda is sent back to the server and saved. Input: Generated expected agenda, Output: Modified agenda.

[1031] Step 4:

[1032] The server provides ideas related to the goals and agenda. Specifically, it uses a generative AI model (e.g., GPT-3) to collect relevant information from databases and the Internet and generate specific proposals such as "ideas for introducing new safety devices." These ideas are sent to the user's device. Input: goal information and agenda, output: provided ideas.

[1033] Step 5:

[1034] During a meeting, the server recognizes voice, text, images, and video in real time to support the progress of the meeting. Specifically, it uses voice recognition technology to convert comments into text, and image and video recognition technology to analyze important scenes. The recognized information is displayed on the device in real time, prompting the user to move on to the next agenda item as necessary. Input: Voice, text, images, and video data during the meeting. Output: Text comments displayed in real time and notifications about the next agenda item.

[1035] Step 6:

[1036] The server records user comments with timestamps and clarifies decisions and next actions at the end of the meeting. Specifically, it analyzes the content of comments and generates specific action items such as "Test new safety devices: Engineering department, deadline: 2023-12-01." The generated action items are sent to the user's device, where they can be checked and modified. Input: Texted comments, Output: Decisions and next actions.

[1037] Step 7:

[1038] The server automatically creates minutes based on all records of the meeting. Specifically, it compiles the recorded remarks, decisions, and next actions, and generates minutes according to a format. The generated minutes are automatically shared with meeting participants, and notifications are sent to user devices. Input: Recorded remarks, decisions, next actions. Output: Automatically created minutes and shared notifications.

[1039] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1040] This section describes a detailed embodiment of a system that combines an emotion engine with the "meeting supporter" of the present invention. This system supports meetings using a software program that includes a server, user terminals, a network, and an emotion engine.

[1041] Meeting goal setting

[1042] User

[1043] The user logs in to the application from the terminal to be used.

[1044] Before the meeting, the user inputs the goal of the meeting in the text field. For example, the user might input "Goal of the new product planning meeting: Finalize specifications."

[1045] server

[1046] The server receives the meeting goals input by the user.

[1047] The server stores the received meeting goals in a database.

[1048] Creating a projected agenda

[1049] server

[1050] The server uses natural language processing algorithms to generate a potential agenda based on the meeting goals.

[1051] For example, it automatically generates agenda items such as "Consider new product specifications," "Share market research results," and "Confirm design proposals."

[1052] User

[1053] The user can check the generated planned agenda through the terminal and make corrections as necessary, for example, adding "Confirm manufacturing schedule."

[1054] Idea submission

[1055] server

[1056] The server presents ideas related to the meeting goals based on information collected from databases and the Internet.

[1057] The ideas presented include a list of features required for new products and improvements to existing products.

[1058] User

[1059] The user reflects the provided ideas in the content of the meeting.

[1060] In-meeting support

[1061] server

[1062] The server converts users' comments into text in real time during the meeting using voice recognition technology.

[1063] The server monitors the progress of the meeting and checks whether the agenda is being followed properly.

[1064] Terminal

[1065] The terminal displays the textual utterances to the user in real time.

[1066] For example, the terminal may notify, "This topic has been completed. We will move on to the next topic."

[1067] Emotion recognition and response using emotion engine

[1068] server

[1069] The server uses an emotion engine to analyze the user's voice, facial expression, and text data during the meeting to recognize the user's emotional state.

[1070] For example, it detects when a user is in a dissatisfied state.

[1071] Terminal

[1072] The device prompts the user in real time to respond appropriately based on their emotional state, for example by notifying them that "The speaker seems dissatisfied. Please provide additional explanation."

[1073] User

[1074] The user checks the instructions presented by the device and responds based on their emotional state.

[1075] Actions at the end of a meeting

[1076] server

[1077] The server automatically generates "decisions" and "next actions" based on what is said during the meeting. For example, it lists specific actions such as "Add feature A: Person in charge B, deadline: next Friday."

[1078] Terminal

[1079] The terminal presents the generated decisions and next actions to the user.

[1080] User

[1081] The user reviews the proposed decisions and next actions and makes any necessary corrections.

[1082] Automatically create and share meeting minutes

[1083] server

[1084] The server automatically creates meeting minutes including decisions and next actions.

[1085] The server automatically shares the created minutes with meeting participants.

[1086] Terminal

[1087] The terminal notifies the user of the shared minutes and prompts him / her to confirm them.

[1088] User

[1089] The user can check the shared minutes on their device and prepare for the next action.

[1090] Specific examples

[1091] A specific example of how a new product planning meeting might proceed is shown below.

[1092] 1. Setting meeting goals

[1093] The user enters "Goal of new product planning meeting: Finalize specifications."

[1094] The server stores this.

[1095] 2. Creating a planned agenda

[1096] The server automatically generates tasks such as "Consider new product specifications," "Share market research results," and "Confirm design proposals," and the user adds "Confirm manufacturing schedule."

[1097] 3. Idea submission

[1098] The server provides related ideas, and the user prepares the meeting content based on them.

[1099] 4. Support during meetings

[1100] The server converts the comments into text in real time and the device displays it.

[1101] The user follows the instructions on the terminal to proceed with the conference.

[1102] 5. Emotion recognition and response using an emotion engine

[1103] The server uses an emotion engine to recognize the user's emotional state, and the device displays appropriate instructions.

[1104] For example, if the user appears dissatisfied, the terminal will notify the user that "additional explanation is required."

[1105] 6. Actions at the end of the meeting

[1106] The server generates the decisions and next actions, which the terminal presents to the user.

[1107] The user checks and corrects the data.

[1108] 7. Automatically create and share meeting minutes

[1109] The server automatically creates minutes and shares them with participants.

[1110] The terminal notifies the user, and the user checks the minutes.

[1111] This system can improve the quality of meetings and reduce meeting times. In addition, by using an emotion engine, it is possible to respond appropriately to the emotions of meeting participants, providing a better meeting environment.

[1112] The processing flow will be explained below.

[1113] Step 1:

[1114] A user logs into the application using a terminal.

[1115] Step 2:

[1116] The user inputs the goal of the meeting. For example, the user might input "Goal of the new product planning meeting: Finalize specifications."

[1117] Step 3:

[1118] The server receives the meeting goals input by the user and stores them in a database.

[1119] Step 4:

[1120] The server uses natural language processing algorithms to generate a potential agenda based on the received meeting goals.

[1121] Step 5:

[1122] The server automatically generates agenda items such as "Consider new product specifications," "Share market research results," and "Confirm design proposals."

[1123] Step 6:

[1124] The user can check the generated anticipated agenda on their device and make corrections or additions as necessary. For example, adding "Confirm manufacturing schedule" is possible.

[1125] Step 7:

[1126] The server collects relevant ideas from databases and the Internet and presents them to the user.

[1127] Step 8:

[1128] The user uses the ideas presented by the server as a reference to prepare the contents of the meeting.

[1129] Step 9:

[1130] The server converts user utterances into text in real time during the conference using a voice recognition engine.

[1131] Step 10:

[1132] The server transmits the textual content of the speech to the terminal and displays it.

[1133] Step 11:

[1134] The device uses voice recognition to convert what is said into text and displays it in real time.

[1135] Step 12:

[1136] The server monitors the progress of the conference and checks whether the agenda is being followed properly.

[1137] Step 13:

[1138] The terminal notifies the user in real time of suggestions and warnings regarding the progress of the meeting, for example, by displaying "This topic has been completed. Let's move on to the next topic."

[1139] Step 14:

[1140] The user follows the instructions on the terminal to proceed with the conference.

[1141] Step 15:

[1142] The server uses an emotion engine to analyze the user's emotional state during the conference.

[1143] Step 16:

[1144] The server generates instructions for adjusting the progress and responses of the meeting based on the user's emotion data recognized by the emotion engine.

[1145] Step 17:

[1146] The device displays the user's emotional state as recognized by the emotion engine and prompts the user to take appropriate action, such as notifying the user that "The speaker seems dissatisfied. Please provide additional explanation."

[1147] Step 18:

[1148] The user follows the instructions on the terminal and adjusts the progress of the conference as appropriate.

[1149] Step 19:

[1150] The server automatically generates "decisions" and "next actions" based on what was said during the meeting. For example, it lists specific actions such as "Add feature A: Person in charge: B, deadline: next Friday."

[1151] Step 20:

[1152] The terminal presents the generated decisions and next actions to the user.

[1153] Step 21:

[1154] The user reviews the proposed decisions and next actions and makes any necessary corrections.

[1155] Step 22:

[1156] The server automatically creates meeting minutes based on confirmed decisions and next actions.

[1157] Step 23:

[1158] The server automatically shares the created minutes with the meeting participants.

[1159] Step 24:

[1160] The terminal notifies the user of the shared minutes.

[1161] Step 25:

[1162] The user receives the minutes from the terminal and prepares for the next action.

[1163] Example 2

[1164] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1165] In conventional meeting support systems, meeting progress and content organization are often done manually, which takes time and effort and makes it easy for important information to be overlooked. Furthermore, many systems lack functionality to appropriately reflect participants' emotions and reactions to support the progress of meetings. This reduces meeting efficiency and makes it difficult to ensure quality.

[1166] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1167] In this invention, the server includes means for inputting the purpose of the meeting, means for generating an agenda based on the input purpose of the meeting, means for recognizing voice, text, and image data in real time during the meeting to support the progress of the meeting, means for monitoring the progress and agenda of the meeting and supporting the appropriate progress, means for analyzing the emotional state of users during the meeting and encouraging them to respond, means for generating decisions and next actions at the end of the meeting, and means for automatically creating minutes and sharing them with the meeting participants. This automates the progress and content organization of the meeting, enables appropriate responses based on the emotional states of the participants, and improves the quality and efficiency of the meeting.

[1168] The "purpose of the meeting" refers to the specific goal or outcome that the user is trying to achieve through the meeting.

[1169] An "agenda" refers to a list of specific topics or items to be addressed in the course of a meeting.

[1170] "Speech recognition" refers to the technology that uses a computer to analyze speech spoken during a meeting and convert it into text data.

[1171] "Texting" refers to the process of converting audio or video data into written information.

[1172] "Image data" refers to visual information, including facial expressions and movements of participants, acquired during a meeting.

[1173] "Support for meeting progress" refers to providing various support functions to ensure that meetings proceed smoothly based on the agenda.

[1174] "Progress monitoring" refers to the process of checking in real time whether progress during a meeting is proceeding as planned.

[1175] "Emotional state" refers to changes in participants' psychological states and emotions during a meeting.

[1176] "Encouraging a response" refers to providing real-time guidance on appropriate actions or responses based on participants' emotional state.

[1177] "Decisions" refer to specific outcomes or courses of action agreed upon during a meeting.

[1178] "Next actions" refers to the specific actions or tasks that should be taken next based on the decisions made in the meeting.

[1179] "Minutes" refers to an official document that summarizes what was discussed, decisions made, and next actions taken at a meeting.

[1180] "Auto-generation" refers to the process by which the system automatically generates data without user intervention.

[1181] "Sharing" refers to providing the generated information and data to all meeting participants.

[1182] The "Meeting Supporter" system of the present invention provides a series of functions for smoothly conducting a meeting. This system is implemented using a software program including a server, user terminals, a network, and an emotion engine.

[1183] Hardware and software used

[1184] 1. Server

[1185] The server uses a relational database (e.g., MySQL) to manage data such as meeting goals, agendas, and statements made during meetings.

[1186] Use natural language processing algorithms (e.g., GPT-3) to automatically generate an agenda from the input meeting goals.

[1187] Use speech recognition technology (e.g., Google Cloud Speech-to-Text) to convert what is said during the meeting into text in real time.

[1188] Using an emotion engine (e.g., Microsoft Azure Emotion API), it recognizes the user's emotional state in real time during a meeting and prompts them to respond.

[1189] 2. Terminal

[1190] The user terminal functions as an interface for displaying real-time information during the meeting and receiving notifications from the emotion engine.

[1191] Using a web or mobile application, users can enter meeting goals, review and revise agendas and ideas.

[1192] Processing flow

[1193] 1. Setting meeting goals

[1194] The user logs in to the application using a terminal and enters the goal of the meeting in a text field. For example, if the user enters "Goal of the new product planning meeting: Finalize specifications," the server receives this and stores it in the database.

[1195] 2. Creating a planned agenda

[1196] The server generates an agenda using a natural language processing algorithm (e.g., GPT-3) based on the saved meeting goals. For example, it automatically generates topics such as "Consider new product specifications," "Share market research results," and "Confirm design proposals." The generated agenda is sent to the device, where the user can review it and make any necessary changes.

[1197] 3. Idea submission

[1198] The server provides ideas related to the meeting goals based on information collected from databases and the Internet. For example, specific ideas such as "a list of features required for a new product" or "improvements to an existing product" can be presented to the terminal, and the user can reflect these ideas in the meeting content.

[1199] 4. Support during meetings

[1200] The server uses speech recognition technology to convert the user's speech into text in real time, which the device displays to the user, for example, notifying them that "This topic has been completed. We will move on to the next topic." The server also monitors the progress and agenda of the meeting to ensure that it is proceeding properly.

[1201] 5. Emotion recognition and response using an emotion engine

[1202] During the conference, the server uses an emotion engine to analyze data such as the user's voice, facial expression, and text to recognize their emotional state. For example, if the user is dissatisfied, the terminal will display an instruction such as "The speaker seems dissatisfied. Please provide additional explanation." The user can then respond accordingly.

[1203] 6. Actions at the end of the meeting

[1204] The server automatically generates "decisions" and "next actions" based on what was said during the meeting. For example, it lists things like "Add feature A: Person in charge: B, deadline: next Friday." These decisions are presented to the user via their terminal, who can then confirm and make any necessary corrections.

[1205] 7. Automatically create and share meeting minutes

[1206] After the meeting ends, the server automatically creates minutes including decisions and next actions, and shares them with the meeting participants via their devices. The devices then notify the users that the minutes have been shared, and the users can confirm this.

[1207] Specific examples

[1208] A specific example of how a new product planning meeting might proceed is shown below.

[1209] 1. Setting meeting goals

[1210] Enter "Goal of new product planning meeting: Finalize specifications." The server saves this.

[1211] 2. Creating a planned agenda

[1212] "Consider new product specifications," "Share market research results," and "Confirm design proposals" are automatically generated, and the user can add "Confirm manufacturing schedule."

[1213] 3. Idea submission

[1214] Related ideas are provided and users prepare meeting content based on them.

[1215] 4. Support during meetings

[1216] The device converts comments into text in real time and displays them, and users follow instructions on the device to conduct the meeting.

[1217] 5. Emotion recognition and response using an emotion engine

[1218] The emotion engine recognizes the user's emotional state and the device displays appropriate instructions. For example, if the user appears dissatisfied, the device will notify them that "additional explanation is required."

[1219] 6. Actions at the end of the meeting

[1220] Generates decisions and next actions and presents them to the user, who can then confirm and modify them.

[1221] 7. Automatically create and share meeting minutes

[1222] Minutes are automatically created and shared with participants. The device notifies the user, who can then check the minutes.

[1223] Examples of prompt statements

[1224] Example of an input prompt for a generative AI model:

[1225] The goal of the new product planning meeting is to "finalize specifications." Based on this goal, generate the next expected agenda.

[1226] If a user expresses dissatisfaction or voices dissatisfaction during a meeting, analyze the situation and suggest appropriate ways to respond.

[1227] This enables the system to generate and present appropriate agendas and emotional responses.

[1228] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1229] Step 1: Set meeting goals

[1230] Input: A user uses a terminal to log into the application and enter the meeting goal in a text field.

[1231] Specific action: The user inputs "Goal of new product planning meeting: Finalize specifications."

[1232] Data processing: The server receives the entered data and stores it in a database.

[1233] Output: Saved meeting goal data.

[1234] Step 2: Create a visionary agenda

[1235] Input: The server retrieves the saved meeting goals from the database.

[1236] Specific operation: The server uses a natural language processing algorithm (e.g., GPT-3) to generate an agenda from the meeting goals.

[1237] Data processing: Analyze meeting goal data and generate relevant agenda items.

[1238] Output: A list of agenda items such as "Consider new product specifications," "Share market research results," and "Confirm design proposals."

[1239] Step 3: Review and revise the agenda

[1240] Input: The server generates an agenda and sends it to the user's device.

[1241] Specific operation: The user checks the agenda through the terminal and makes corrections such as adding "Confirm production schedule."

[1242] Data processing: The user's modifications are sent to the server and the database is updated.

[1243] Output: The final agenda list.

[1244] Step 4: Submit your idea

[1245] Input: The server retrieves the final agenda from the database.

[1246] Specific operation: The server collects relevant information from databases and the Internet.

[1247] Data processing: Analyze the collected information and generate concrete ideas related to the agenda.

[1248] Output: A list of ideas such as "a list of features required for new products" and "improvements to existing products."

[1249] Step 5: Present your idea

[1250] Input: The server sends the generated idea list to the user's terminal.

[1251] Specific actions: The user checks the ideas presented through the terminal and reflects them in the meeting content.

[1252] Data processing: User feedback is stored in a database.

[1253] Output: Ideas reflected in the meeting content.

[1254] Step 6: Support during the meeting

[1255] Input: The terminal collects the user's speech.

[1256] Specific operation: The server uses speech recognition technology (e.g., Google Cloud Speech-to-Text) to convert the spoken voice into text.

[1257] Data processing: Converts voice data into text data in real time.

[1258] Output: Textualized speech.

[1259] Step 7: Displaying Text

[1260] Input: The server converts the speech into text and sends it to the terminal.

[1261] Specific operation: The device displays the text of the remarks to the user in real time. For example, it notifies the user, "This topic has been concluded. We will move on to the next topic."

[1262] Data processing: Converting data into a format for display.

[1263] Output: Real-time display to the user.

[1264] Step 8: Recognize and respond to emotions

[1265] Input: The device collects voice, facial expression, and text data during the meeting.

[1266] Specific operation: The server uses an emotion engine (e.g., Microsoft Azure Emotion API) to analyze the collected data and recognize the user's emotional state.

[1267] Data processing: Analyze emotional data in real time and determine the necessary response.

[1268] Output: Response instructions based on emotional state.

[1269] Step 9: Propose a response

[1270] Input: The server sends instructions based on the user's emotional state to the device.

[1271] Specific operation: The terminal notifies the user, for example, "The speaker seems dissatisfied. Please provide additional explanation."

[1272] Data processing: Converting data into a format for display.

[1273] Output: Real-time notification to the user.

[1274] Step 10: Actions at the end of the meeting

[1275] Input: The server retrieves the content of what was said during the meeting from the database.

[1276] Specific actions: The server automatically generates "decisions" and "next actions." For example, it lists specific actions such as "Add feature A: Person in charge: B, Deadline: Next Friday."

[1277] Data processing: Analyzes the content of statements and automatically generates data.

[1278] Output: List of decisions and next actions.

[1279] Step 11: Present and confirm your actions

[1280] Input: The server sends the generated decisions and next actions to the terminal.

[1281] Specific operation: The terminal displays the decision and next action to the user, and the user confirms and corrects it.

[1282] Data processing: The corrected data is sent back to the server and the database is updated.

[1283] Output: Confirmed and revised decisions and next actions.

[1284] Step 12: Automatically create and share meeting minutes

[1285] Input: The server retrieves the decision and next action from the database.

[1286] Specific operation: The server automatically creates minutes including decisions and next actions, and automatically shares them with meeting participants.

[1287] Data processing: Format the data into minutes format.

[1288] Output: Shared meeting minutes.

[1289] Step 13: Notification of minutes

[1290] Input: The server sends the minutes to each participant.

[1291] Specific operation: The terminal notifies the user that the minutes have been shared.

[1292] Data processing: Formatting of notification content.

[1293] Output: Notification to the user.

[1294] (Application example 2)

[1295] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1296] In modern business, it is important to streamline meetings and negotiations and respond to customers quickly and accurately. However, particularly in brick-and-mortar stores, sales staff often find it difficult to answer customers' questions immediately and accurately grasp and respond to their emotional state. This can lead to missed sales opportunities and a decrease in customer satisfaction. Furthermore, it is difficult to accurately record all information about the sales process and share it instantly.

[1297] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for inputting the meeting goal, a means for generating a hypothetical agenda based on the input meeting goal, a means for recognizing voice, text, images, and video in real time during the meeting to support the progress of the meeting, a means for clarifying decisions and next actions at the end of the meeting, a means for automatically creating minutes and automatically sharing them with meeting participants, a means for converting customer questions into text in real time via a display device worn by the user and providing appropriate answers, and a means for analyzing the user's emotional state and notifying the user of an optimal response based on the emotion in real time. This allows sales staff to respond to customer questions quickly and accurately and provide relevant information in real time, thereby improving customer satisfaction. Furthermore, the emotion engine allows for accurate understanding of the customer's emotional state and optimal responses, thereby providing a better sales experience.

[1298] The "means for inputting the goals of the meeting" is an interface that allows a user to input the goals and expected outcomes of the meeting into the system as text, and the system receives and stores them.

[1299] The "means for generating a potential agenda" is a function that uses a natural language processing algorithm to automatically generate a meeting agenda and progress schedule based on the input meeting goal information.

[1300] "Means to support the progress of the meeting" refers to a function in which the system recognizes voice, text, images, and video in real time during the meeting, monitors the progress of the meeting, and provides appropriate instructions and information.

[1301] "Means for clarifying decisions and next actions" refers to a function in which the system clearly indicates the specific next actions to be taken, the person in charge, deadlines, etc. based on the participants' comments and decisions made at the end of the meeting.

[1302] "Means for automatically creating meeting minutes and automatically sharing them with meeting participants" refers to a function that allows the system to automatically create meeting minutes based on comments and decisions made during the meeting and share them online with all meeting participants.

[1303] "Means of converting customer questions into text in real time through a display device worn by the user and providing appropriate answers" refers to a function that uses a display device such as a head-mounted display (HMD) worn by sales staff to convert customer questions into text using voice recognition technology, and then the system provides appropriate answers in real time based on that text.

[1304] "Means for analyzing the user's emotional state and notifying the optimal response based on that emotion in real time" refers to a function in which the system uses an emotion engine to analyze the voice, facial expression, and other data of the user or customer, and notifies the user in real time of the optimal response based on that emotional state.

[1305] The "sales supporter for brick-and-mortar stores" of the present invention supports sales in brick-and-mortar stores using a system including a server, user terminals, a network, and an emotion engine. The system aims to enable sales staff to respond to customers appropriately and quickly, thereby improving customer satisfaction.

[1306] Initial Setup

[1307] Hardware

[1308] Display device: Head-mounted display (HMD) (e.g., Magic Leap 2)

[1309] Input devices: Microphone, camera

[1310] software

[1311] Emotion Engine: Software for analyzing the emotional state of customers

[1312] Speech recognition system: a system that converts speech into text

[1313] Natural language processing algorithms: Algorithms that analyze text data and generate relevant information

[1314] Cloud server: Cloud services such as AWS

[1315] Real-time database: Firebase, etc.

[1316] System Operation

[1317] Goal setting

[1318] The user puts on the HMD and logs in. The server displays the sales target and saves it in the database.

[1319] Agenda Generation

[1320] The server generates daily tasks and product explanation scripts based on sales targets, which the user can review and adjust as needed.

[1321] Idea submission

[1322] The server provides product features and related information, which the user uses to explain the product to customers, such as the features of new products and campaign information.

[1323] Real-time Support

[1324] The server uses voice recognition technology and natural language processing algorithms to convert customer questions into text and display them on the HMD in real time. The user can then respond to the customer while checking the appropriate answer on the HMD.

[1325] emotion recognition

[1326] The server uses an emotion engine to analyze the customer's facial expressions and tone of voice to identify their emotional state. The HMD then notifies the user of the optimal response based on the customer's emotional state. For example, if the customer is dissatisfied, the HMD will notify the user that "further explanation is required."

[1327] When sales end

[1328] The server generates a list of next actions and customer follow-ups when a sale is completed, and displays it on the HMD. The user can then confirm and adjust the next actions.

[1329] Report creation and sharing

[1330] The server automatically generates a report based on sales results and customer feedback and notifies the user via the HMD. The user can then check and correct the report and take any necessary action.

[1331] Specific examples

[1332] For example, if the user aims to sell new product A, the user inputs "Today's goal: Sell 20 units of new product A" into the HMD. The server saves this and generates a task list related to the sales goal. Also, if a customer asks "What are the features of new product A?" during a sale, the server immediately converts the question into text, and the user provides an explanation based on the answer displayed on the HMD. Furthermore, if the customer shows signs of dissatisfaction, the HMD notifies the user that "Further explanation is required."

[1333] Prompt Sentence Examples

[1334] What are the features of new product A?

[1335]

[1336] My customer looks unhappy. How should I respond?

[1337] In this way, the system of the present invention supports effective and efficient customer service in physical stores, enabling sales staff to provide optimal service.

[1338] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1339] Step 1:

[1340] The user puts on the HMD and logs in. The user enters their sales target in text format into the system. The server receives the entered sales target and stores it in a cloud database. For example, the user might enter "Today's target: Sell 20 units of new product A."

[1341] Input: Sales target text input

[1342] Data processing: Save and analyze received text data

[1343] Output: Save to cloud database

[1344] Step 2:

[1345] The server uses natural language processing algorithms based on sales target information to generate scripts for daily tasks and product descriptions, which are then displayed on the HMD for the user to review and adjust as needed.

[1346] Input: Sales target information

[1347] Data processing: Script generation using natural language processing algorithms

[1348] Output: Script display on HMD

[1349] Step 3:

[1350] The server collects product features and related information and generates this information through the idea provision function. Based on the information provided, the user prepares an explanation for the customer. For example, they can provide information about the features of a new product or campaign information.

[1351] Input: Sales target and related data

[1352] Data processing: information gathering and idea generation

[1353] Output: Information displayed on the HMD

[1354] Step 4:

[1355] If a customer has a question during a sale, the user receives the question through the microphone in the HMD. The server uses voice recognition technology to convert the customer's question into text and display it on the user's HMD in real time.

[1356] Input: Customer Question

[1357] Data processing: speech recognition and text conversion

[1358] Output: Text display on HMD

[1359] Step 5:

[1360] The user provides the customer with an appropriate answer based on the questions displayed on the HMD, including the process of retrieving the most appropriate answer from the information database held by the server.

[1361] Input: Texted question

[1362] Data processing: Information retrieval from databases and optimal answer generation

[1363] Output: Display the answer on the HMD

[1364] Step 6:

[1365] The server uses an emotion engine to analyze the customer's facial expressions and tone of voice via camera and voice input, and identifies the customer's emotional state. For example, it can estimate the customer's emotions based on voice tone and facial recognition technology.

[1366] Input: Customer facial expression data, voice data

[1367] Data processing: Analysis with emotion engine

[1368] Output: Customer emotional state identification result

[1369] Step 7:

[1370] Based on the emotional state, the server notifies the user in real time on the HMD with an appropriate response, such as displaying a message like, "The customer appears dissatisfied. Further explanation is required."

[1371] Input: Customer's emotional state

[1372] Data processing: generating response instructions

[1373] Output: Real-time notification to HMD

[1374] Step 8:

[1375] The server generates a list of next actions and customer follow-ups upon sale completion, which are displayed to the user via the HMD and adjustments made as necessary.

[1376] Input: Sales completion data

[1377] Data processing: Next action generation and follow-up list creation

[1378] Output: List display on HMD

[1379] Step 9:

[1380] The server automatically generates a report based on sales results and customer feedback and notifies the user via the HMD. The user can then review the report and make any necessary corrections.

[1381] Input: Sales results and feedback

[1382] Data Processing: Report Generation

[1383] Output: Report notification to HMD

[1384] By following these steps, the "Sales Supporter for Physical Stores" enables sales staff to efficiently and effectively serve customers, improving customer satisfaction.

[1385] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1386] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1387] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1388] [Third embodiment]

[1389] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1390] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1391] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1392] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1393] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1394] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1395] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1396] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1397] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1398] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1399] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1400] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1401] A detailed embodiment of the "meeting supporter" according to the present invention will now be described. This system includes a server, a user terminal, and a software program for supporting a meeting via a network.

[1402] Meeting goal setting

[1403] User

[1404] The user logs in to the application from the terminal to be used.

[1405] Before the meeting, the user inputs the goal of the meeting in the text field. For example, the user might input "Goal of the new product planning meeting: Finalize specifications."

[1406] server

[1407] The server receives the meeting goals input by the user and stores them in a database.

[1408] The server collects relevant data based on the goal information.

[1409] Creating a projected agenda

[1410] server

[1411] The server uses natural language processing algorithms to generate a potential agenda based on the meeting goals.

[1412] For example, topics such as "Consider new product specifications," "Share market research results," and "Confirm design proposals" are automatically generated.

[1413] User

[1414] The user can check the generated planned agenda through the terminal and make corrections as necessary, for example, adding "Confirm manufacturing schedule."

[1415] Idea submission

[1416] server

[1417] The server presents ideas related to the meeting goals based on information collected from databases and the Internet.

[1418] The ideas presented include a list of features required for new products and improvements to existing products.

[1419] User

[1420] The user reflects the provided ideas in the content of the meeting.

[1421] In-meeting support

[1422] server

[1423] The server recognizes what users say during the meeting in real time and converts it into text.

[1424] The server monitors the progress of the meeting and ensures that the agenda is being followed.

[1425] Terminal

[1426] The terminal displays the textual utterances to the user in real time.

[1427] For example, the terminal may notify, "This topic has been completed. We will move on to the next topic."

[1428] User

[1429] Users can smoothly conduct the conference based on the display on their terminals.

[1430] Actions at the end of a meeting

[1431] server

[1432] The server automatically generates "decisions" and "next actions" based on what is said during the meeting.

[1433] For example, specific actions such as "Add feature A: Responsible person B, Deadline: next Friday" are listed.

[1434] Terminal

[1435] The terminal presents the generated decisions and next actions to the user, allowing them to review and modify them.

[1436] User

[1437] The user reviews the proposed decisions and next actions and makes any necessary corrections.

[1438] Automatically create and share meeting minutes

[1439] server

[1440] The server automatically creates meeting minutes including decisions and next actions.

[1441] The server automatically shares the created minutes with meeting participants.

[1442] Terminal

[1443] The terminal notifies the user of the shared minutes and prompts him / her to confirm them.

[1444] User

[1445] Users can check the shared minutes on their own devices and prepare for their next actions.

[1446] Specific examples

[1447] The following steps are taken during the new product planning meeting:

[1448] 1. Setting meeting goals

[1449] The user enters "Goal of new product planning meeting: Finalize specifications."

[1450] The server stores this.

[1451] 2. Creating a projected agenda

[1452] The server automatically generates tasks such as "Consider new product specifications," "Share market research results," and "Confirm design proposals," and the user adds "Confirm manufacturing schedule."

[1453] 3. Idea submission

[1454] The server provides related ideas, and users use them to prepare for the meeting.

[1455] 4. Support during meetings

[1456] The server converts the comments into text in real time and the device displays it.

[1457] The user follows the instructions on the terminal to proceed with the conference.

[1458] 5. Actions at the end of the meeting

[1459] The server generates the decisions and next actions, which the terminal presents to the user.

[1460] The user checks and corrects the data.

[1461] 6. Automatically create and share meeting minutes

[1462] The server automatically creates minutes and shares them with participants.

[1463] The terminal notifies the user, and the user checks the minutes.

[1464] In this way, the present invention can improve the quality of the conference and reduce the time it takes to hold the conference.

[1465] The processing flow will be explained below.

[1466] Step 1:

[1467] A user logs into the application using a terminal.

[1468] Step 2:

[1469] The user inputs the goal of the meeting. For example, the user might input "Goal of the new product planning meeting: Finalize specifications."

[1470] Step 3:

[1471] The server receives the meeting goals input by the user.

[1472] Step 4:

[1473] The server stores the received meeting goals in a database.

[1474] Step 5:

[1475] The server automatically generates a hypothetical agenda based on the meeting goals, such as "discussing new product specifications," "sharing market research results," and "confirming design proposals."

[1476] Step 6:

[1477] The user checks the generated assumed agenda through the terminal.

[1478] Step 7:

[1479] The user can modify the expected agenda as needed, for example adding "Confirm manufacturing schedule."

[1480] Step 8:

[1481] The server collects relevant ideas from a stored database or from the Internet.

[1482] Step 9:

[1483] The server presents the collected ideas to the user, such as a list of features required for a new product or improvements to an existing product.

[1484] Step 10:

[1485] The user uses the presented ideas as a reference to prepare the contents of the meeting.

[1486] Step 11:

[1487] The server converts users' comments into text in real time during the meeting using voice recognition technology.

[1488] Step 12:

[1489] The server monitors the progress of the conference and checks whether the agenda is being followed properly.

[1490] Step 13:

[1491] The terminal displays the content of the speech converted into text by speech recognition to the user.

[1492] Step 14:

[1493] The terminal displays suggestions and notes for the user regarding the progress of the meeting in real time, such as "This agenda item has been completed. Let's move on to the next agenda item."

[1494] Step 15:

[1495] The user follows the instructions on the terminal to ensure the conference proceeds smoothly.

[1496] Step 16:

[1497] The server automatically generates "decisions" and "next actions" based on what is said during the meeting. For example, it lists specific actions such as "Add feature A: Person in charge: B, deadline: next Friday."

[1498] Step 17:

[1499] The terminal presents the generated decisions and next actions to the user.

[1500] Step 18:

[1501] The user reviews the proposed decisions and next actions and makes any necessary corrections.

[1502] Step 19:

[1503] The server automatically creates meeting minutes based on confirmed decisions and next actions.

[1504] Step 20:

[1505] The server automatically shares the created minutes with the meeting participants.

[1506] Step 21:

[1507] The terminal notifies the user of the shared minutes.

[1508] Step 22:

[1509] The user receives the minutes from the terminal and prepares for the next action.

[1510] By dividing the steps in this way, the system of the present invention improves the quality of the conference and reduces the time it takes to hold the conference.

[1511] Example 1

[1512] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1513] In many modern meetings, many processes, such as proceeding, taking minutes, and deciding on post-meeting actions, are done manually, resulting in a decrease in meeting efficiency. It is also difficult for all participants to understand the content and progress of the meeting, which often leads to a decline in meeting quality. There is a need for technology that can solve this problem and make meetings more effective.

[1514] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1515] In this invention, the server includes: means for a user to input the goal of the meeting; means for the server to save the input goal of the meeting; means for the server to generate a hypothetical agenda based on the goal of the meeting using a natural language processing algorithm; means for the user to confirm and modify the generated hypothetical agenda; means for the server to collect information related to the goal of the meeting from a database and the Internet and provide ideas; means for the server to recognize and convert user comments in real time during the meeting into text; means for a terminal to display the converted text in real time; means for the server to generate decisions and next actions based on the content of the meeting at the end of the meeting; means for the server to present the generated decisions and next actions to the user; and means for the server to automatically create minutes and share them with meeting participants. This automates the process of understanding the progress and decisions of the meeting, and the creation and sharing of minutes, thereby improving the efficiency and quality of meetings.

[1516] The "meeting goal" represents the specific objective or purpose that the meeting aims to achieve.

[1517] "User" refers to the person who sets up and conducts the meeting and operates each function of the system.

[1518] "Terminal" means a hardware device used by a User to access the System, enter information, and receive notifications from the System.

[1519] "Server" means a central computer facility that manages the entire system and performs data processing.

[1520] "Text conversion" refers to the process of converting audio data into text data.

[1521] "Natural language processing algorithms" are technical methods that allow computers to understand and generate human language, including grammatical analysis and semantic analysis.

[1522] "Database" refers to a digital repository used to efficiently store and retrieve structured data.

[1523] "Internet" refers to the global network that interconnects computer networks around the world.

[1524] "Proceeding a meeting" refers to activities to ensure that each agenda item is properly advanced according to the meeting plan.

[1525] "Decisions" refer to specific actions or decisions agreed upon during a meeting.

[1526] "Next actions" refer to specific tasks or steps to be taken after the meeting.

[1527] "Minutes" refers to an official document that records what was discussed, decisions made, and next actions taken during a meeting.

[1528] The "Complete Meeting Supporter" of this invention is a system that includes a server, a user terminal, and a software program for supporting meetings over a network. This system improves the efficiency and quality of meetings through the following steps. The main hardware used is a user terminal (e.g., a personal computer, tablet, or smartphone) and a server. The software technologies used include natural language processing algorithms, database management systems, and real-time voice recognition technology.

[1529] Meeting goal setting

[1530] User

[1531] The user logs into the application from their own terminal.

[1532] After logging in to the application, the user enters the goal of the meeting in a text field, for example, "Goal of new product planning meeting: Finalize specifications."

[1533] server

[1534] The server receives the meeting goals entered by the user and stores them in a database.

[1535] After saving, the server begins to collect relevant data based on the goals of the meeting.

[1536] Creating a projected agenda

[1537] server

[1538] The server uses natural language processing algorithms to generate a potential agenda based on the meeting goals.

[1539] For example, topics such as "Consider new product specifications," "Share market research results," and "Confirm design proposals" are automatically generated.

[1540] User

[1541] The user can check the generated planned agenda through the terminal and make corrections as necessary, for example, adding "Confirm manufacturing schedule."

[1542] Idea submission

[1543] server

[1544] The server presents ideas related to the meeting goals based on information collected from databases and the Internet.

[1545] The ideas presented include a list of features required for new products and improvements to existing products.

[1546] User

[1547] The user reviews the ideas provided and incorporates them into the content of the meeting.

[1548] In-meeting support

[1549] server

[1550] The server recognizes what users say during the meeting in real time and converts it into text.

[1551] The server monitors the progress of the meeting and ensures that the agenda is being followed.

[1552] Terminal

[1553] The terminal displays the textual utterances to the user in real time.

[1554] For example, the terminal may notify, "This topic has been completed. We will move on to the next topic."

[1555] User

[1556] Users can smoothly conduct the conference based on the display on their terminals.

[1557] Actions at the end of a meeting

[1558] server

[1559] The server automatically generates "decisions" and "next actions" based on what is said during the meeting.

[1560] For example, specific actions such as "Add feature A: Responsible person B, Deadline: next Friday" are listed.

[1561] Terminal

[1562] The terminal presents the generated decisions and next actions to the user, allowing them to review and modify them.

[1563] User

[1564] The user reviews the proposed decisions and next actions and makes any necessary corrections.

[1565] Automatically create and share meeting minutes

[1566] server

[1567] The server automatically creates meeting minutes including decisions and next actions.

[1568] The minutes created are automatically shared by the server with all meeting participants.

[1569] Terminal

[1570] The terminal notifies the user that the minutes have been shared.

[1571] User

[1572] Users can check the shared minutes on their own devices and prepare for their next actions.

[1573] Specific examples

[1574] The following steps are taken during the new product planning meeting:

[1575] 1. Setting meeting goals

[1576] The user enters "Goal of new product planning meeting: Finalize specifications."

[1577] The server stores this.

[1578] 2. Creating a projected agenda

[1579] The server automatically generates tasks such as "Consider new product specifications," "Share market research results," and "Confirm design proposals," and the user adds "Confirm manufacturing schedule."

[1580] 3. Idea submission

[1581] The server provides related ideas, and users use them to prepare for the meeting.

[1582] 4. Support during meetings

[1583] The server converts the comments into text in real time and the device displays it.

[1584] The user follows the instructions on the terminal to proceed with the conference.

[1585] 5. Actions at the end of the meeting

[1586] The server generates the decisions and next actions, which the terminal presents to the user.

[1587] The user checks and corrects the data.

[1588] 6. Automatically create and share meeting minutes

[1589] The server automatically creates minutes and shares them with participants.

[1590] The terminal notifies the user, and the user checks the minutes.

[1591] In this way, the present invention can improve the quality of the conference and reduce the time it takes to hold the conference.

[1592] Prompt Sentence Examples

[1593] "Please create the following agenda for the new product planning meeting, with the goal of finalizing specifications."

[1594] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1595] Step 1: Set meeting goals

[1596] User

[1597] The user logs into the application from their own terminal.

[1598] After logging in, users enter the meeting goal in a text field within the application.

[1599] As an example of input, enter "Goal of new product planning meeting: Finalize specifications."

[1600] The entered goal is sent from the terminal to the server.

[1601] server

[1602] The server receives the goal of the meeting transmitted from the user terminal.

[1603] The received goal information is stored in a database.

[1604] Based on the saved goals, relevant data is collected and stored in a database.

[1605] Input: Meeting goal ("Goal of new product planning meeting: Finalize specifications").

[1606] Output: Meeting goal data stored in a database.

[1607] Step 2: Create a visionary agenda

[1608] server

[1609] The server runs natural language processing algorithms based on the meeting goals stored in a database.

[1610] The algorithm extracts topics related to the goal and generates a hypothetical agenda.

[1611] For example, topics such as "Consider new product specifications," "Share market research results," and "Confirm design proposals" are automatically generated.

[1612] User

[1613] The user checks the generated assumed agenda through the terminal.

[1614] If necessary, the user edits the agenda and adds a new item, for example, "Confirm manufacturing schedule."

[1615] Input: Meeting goal data stored in a database.

[1616] Output: An automatically generated proposed agenda.

[1617] Step 3: Submit your idea

[1618] server

[1619] The server collects information related to the meeting goals from a database or the Internet.

[1620] From the collected information, ideas related to the content of the meeting are generated and presented to the user.

[1621] The ideas presented include a list of features required for new products and improvements to existing products.

[1622] User

[1623] The user checks the provided ideas through the terminal.

[1624] Users can incorporate relevant ideas into the meeting agenda.

[1625] Input: Collected information based on meeting goal data.

[1626] Output: A list of ideas presented to the user.

[1627] Step 4: Support during the meeting

[1628] server

[1629] The server recognizes what users say during the meeting in real time and converts it into text.

[1630] A speech recognition algorithm is used to recognize what is being said.

[1631] The textual comments are sent to a server and recorded.

[1632] The server monitors the progress of the meeting and ensures that the agenda is being followed.

[1633] Terminal

[1634] The terminal displays real-time textual comments sent from the server.

[1635] The device notifies meeting participants of the progress, for example by displaying a notification such as "This topic has been completed. We will move on to the next topic."

[1636] User

[1637] The user proceeds with the conference based on notifications from the terminal.

[1638] Input: User's spoken utterance.

[1639] Output: Server-generated text of speech data and progress notifications.

[1640] Step 5: Actions at the end of the meeting

[1641] server

[1642] The server aggregates the recorded comments made during the meeting and automatically generates "decisions" and "next actions."

[1643] For example, specific actions such as "Add feature A: Responsible person B, Deadline: next Friday" are listed.

[1644] Terminal

[1645] The terminal presents the generated decisions and next actions to the user.

[1646] The user checks these contents and makes corrections as necessary.

[1647] User

[1648] The user reviews the proposed decisions and next actions and applies appropriate corrections.

[1649] Input: Speech data recorded during the meeting.

[1650] Output: Auto-generated decisions and next action list.

[1651] Step 6: Automatically create and share meeting minutes

[1652] server

[1653] The server automatically creates minutes based on the decisions made during the meeting and the next actions.

[1654] The created minutes are shared with all meeting participants by the server.

[1655] Terminal

[1656] The terminal notifies the user that the minutes have been shared.

[1657] Users can review the shared minutes on their own devices and provide feedback or corrections as needed.

[1658] User

[1659] The user checks the minutes and prepares for the next action.

[1660] Input: Decision and next action data.

[1661] Output: Automatically generated meeting minutes.

[1662] (Application example 1)

[1663] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1664] In modern factories, efficient meeting management and work planning are essential to improving work efficiency and reducing errors. However, existing meeting support systems are primarily designed for office work and are often not suitable for use on factory floors. Furthermore, while it is required that comments be recorded in real time during meetings and that next actions be smoothly shared, this is not easy to achieve. Furthermore, there is a lack of a system that allows relevant information and ideas based on goals to be quickly provided, further improving work efficiency on-site.

[1665] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1666] In this invention, the server includes: [means for inputting meeting goals]; [means for generating a hypothetical agenda based on the input meeting goals]; [means for recognizing voice, text, images, and video in real time during the meeting to support the progress of the meeting]; [means for clarifying decisions and next actions at the end of the meeting]; [means for automatically creating minutes and automatically sharing them with meeting participants]; [means for providing relevant ideas based on the goals of factory work and recording comments with timestamps]; and [means for monitoring the progress of factory work and displaying important comments and action items in real time]. This enables meetings to be run efficiently, improving factory work efficiency and reducing errors. Furthermore, by clarifying and quickly sharing next actions, work efficiency on the factory floor can be further improved.

[1667] A "meeting goal" indicates the specific purpose or objective to be achieved in the meeting.

[1668] A "potential agenda" is a list of topics prepared in advance based on the goals of the meeting.

[1669] "Speech recognition" is a technology that converts speech spoken during a meeting into text in real time.

[1670] "Text conversion" is the process of converting voice data obtained by voice recognition into text information.

[1671] "Image recognition" is a technology that analyzes image data used during a meeting and extracts necessary information.

[1672] "Video recognition" is a technology that analyzes video data during a meeting and identifies important scenes, people, and objects.

[1673] "Decisions" are specific actions or policies agreed upon during a meeting.

[1674] "Next actions" are specific tasks or steps that need to be carried out based on decisions.

[1675] Minutes are a record of what was said during a meeting, decisions made, next actions, etc.

[1676] "Factory work" refers to activities such as production, maintenance, and inspection that take place within a factory.

[1677] A "timestamp" is a mark that indicates the date and time when the information was generated.

[1678] "Real-time display" means the immediate display of ongoing information or data.

[1679] Ideation is the process of offering helpful suggestions or thoughts related to the goals of a meeting or task.

[1680] The "factory work support system" of the present invention provides technology to significantly improve the efficiency of meetings and discussions in factories. This system includes a server, user terminals, and a software program for supporting meetings in factories via a network.

[1681] The server generates a hypothetical agenda based on the meeting goals entered by the user, and recognizes and analyzes voice, text, images, and video in real time to support the progress of the meeting. This allows important comments and decisions made during the meeting to be instantly recorded, and the next action to be taken is automatically clarified at the end of the meeting. Furthermore, the system also has a function to automatically create meeting minutes and share them with meeting participants. It also has a function to provide relevant ideas based on the goal of work within the factory and record comments with timestamps.

[1682] Hardware and Software Configuration

[1683] The main components of the system are:

[1684] Server: Equipped with a high-performance processor and large-capacity storage, it processes and stores data related to meeting support.

[1685] User terminal: A device that can be worn or carried by a factory worker, such as a smartphone, smart glasses, or head-mounted display.

[1686] Software: Programs implemented using Python, natural language processing libraries (e.g., spaCy), and generative AI models (e.g., GPT-3).

[1687] Data Processing and Computation

[1688] The server manages a series of processes from setting meeting goals to automatically creating meeting minutes. The main data processing and calculations are explained below:

[1689] 1. Setting the goal of the meeting: The user inputs the goal of the meeting through the terminal. The server receives and stores the information.

[1690] 2. Generation of the expected agenda: The server generates the expected agenda based on the input goal information using a natural language processing algorithm.

[1691] 3. Real-time recognition: The server recognizes and analyzes the voice, text, images, and video during the meeting in real time and displays them on the terminal.

[1692] 4. Recording speech: User speech is converted into text in real time and saved with a timestamp.

[1693] 5. Providing related ideas: The server provides ideas related to the goal based on information collected from databases and the Internet.

[1694] 6. Generation of next actions: At the end of the meeting, the server automatically generates decisions and next actions and displays them on the terminal.

[1695] 7. Automatic minutes creation: The server creates minutes based on all meeting records and automatically shares them with participants.

[1696] Specific examples

[1697] Below is a specific example from a factory meeting regarding the introduction of safety devices for a new product.

[1698] 1. Input example: The user inputs "Meeting regarding the introduction of new product safety devices" into the terminal.

[1699] 2. Generation example: The server automatically generates agendas such as "Confirmation of work procedures," "Discussion of safety measures," and "Proposals for improving efficiency."

[1700] 3. Example of idea provision: The server provides "ideas for introducing new safety devices," and users proceed with the meeting based on these ideas.

[1701] 4. Example of real-time recording: When a user says during a meeting, "Let's consider automation to improve efficiency," the statement is converted into text in real time and recorded with a timestamp.

[1702] 5. Example of next action generation: The server automatically generates and presents an action such as "Testing new safety device: Engineering department, deadline: 2023-12-01."

[1703] 6. Example of automatic minutes creation: After the meeting ends, the server automatically creates minutes including all statements, decisions, and next actions, and shares them with the participants.

[1704] Prompt Sentence Examples

[1705] "Please tell us your ideas for implementing new safety equipment."

[1706] As described above, the present invention enables meetings and discussions within a factory to be conducted efficiently, improving work efficiency and reducing mistakes.

[1707] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1708] Step 1:

[1709] The user inputs the goal of the meeting. Specifically, the user logs into the application from their terminal and inputs the goal of the meeting into a text field. For example, the user might input "Meeting regarding the introduction of new product safety devices." The input goal information is sent to the server. The server receives and saves the input goal information. Input: Meeting goal input by the user, Output: Saved goal information.

[1710] Step 2:

[1711] The server generates a hypothetical agenda based on the input meeting goals. Specifically, the server analyzes goal information using a natural language processing algorithm and automatically generates an agenda such as "confirmation of work procedures," "discussion of safety measures," and "proposals for improving efficiency." The generated agenda is sent to the user's device. Input: saved goal information, Output: generated hypothetical agenda.

[1712] Step 3:

[1713] The user checks the generated expected agenda through the terminal and modifies it as necessary. For example, adding "test plan for new product" as an "additional consideration item." The modified agenda is sent back to the server and saved. Input: Generated expected agenda, Output: Modified agenda.

[1714] Step 4:

[1715] The server provides ideas related to the goals and agenda. Specifically, it uses a generative AI model (e.g., GPT-3) to collect relevant information from databases and the Internet and generate specific proposals such as "ideas for introducing new safety devices." These ideas are sent to the user's device. Input: goal information and agenda, output: provided ideas.

[1716] Step 5:

[1717] During a meeting, the server recognizes voice, text, images, and video in real time to support the progress of the meeting. Specifically, it uses voice recognition technology to convert comments into text, and image and video recognition technology to analyze important scenes. The recognized information is displayed on the device in real time, prompting the user to move on to the next agenda item as necessary. Input: Voice, text, images, and video data during the meeting. Output: Text comments displayed in real time and notifications about the next agenda item.

[1718] Step 6:

[1719] The server records user comments with timestamps and clarifies decisions and next actions at the end of the meeting. Specifically, it analyzes the content of comments and generates specific action items such as "Test new safety devices: Engineering department, deadline: 2023-12-01." The generated action items are sent to the user's device, where they can be checked and modified. Input: Texted comments, Output: Decisions and next actions.

[1720] Step 7:

[1721] The server automatically creates minutes based on all records of the meeting. Specifically, it compiles the recorded remarks, decisions, and next actions, and generates minutes according to a format. The generated minutes are automatically shared with meeting participants, and notifications are sent to user devices. Input: Recorded remarks, decisions, next actions. Output: Automatically created minutes and shared notifications.

[1722] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1723] This section describes a detailed embodiment of a system that combines an emotion engine with the "meeting supporter" of the present invention. This system supports meetings using a software program that includes a server, user terminals, a network, and an emotion engine.

[1724] Meeting goal setting

[1725] User

[1726] The user logs in to the application from the terminal to be used.

[1727] Before the meeting, the user inputs the goal of the meeting in the text field. For example, the user might input "Goal of the new product planning meeting: Finalize specifications."

[1728] server

[1729] The server receives the meeting goals input by the user.

[1730] The server stores the received meeting goals in a database.

[1731] Creating a projected agenda

[1732] server

[1733] The server uses natural language processing algorithms to generate a potential agenda based on the meeting goals.

[1734] For example, it automatically generates agenda items such as "Consider new product specifications," "Share market research results," and "Confirm design proposals."

[1735] User

[1736] The user can check the generated planned agenda through the terminal and make corrections as necessary, for example, adding "Confirm manufacturing schedule."

[1737] Idea submission

[1738] server

[1739] The server presents ideas related to the meeting goals based on information collected from databases and the Internet.

[1740] The ideas presented include a list of features required for new products and improvements to existing products.

[1741] User

[1742] The user reflects the provided ideas in the content of the meeting.

[1743] In-meeting support

[1744] server

[1745] The server converts users' comments into text in real time during the meeting using voice recognition technology.

[1746] The server monitors the progress of the meeting and checks whether the agenda is being followed properly.

[1747] Terminal

[1748] The terminal displays the textual utterances to the user in real time.

[1749] For example, the terminal may notify, "This topic has been completed. We will move on to the next topic."

[1750] Emotion recognition and response using emotion engine

[1751] server

[1752] The server uses an emotion engine to analyze the user's voice, facial expression, and text data during the meeting to recognize the user's emotional state.

[1753] For example, it detects when a user is in a dissatisfied state.

[1754] Terminal

[1755] The device prompts the user in real time to respond appropriately based on their emotional state, for example by notifying them that "The speaker seems dissatisfied. Please provide additional explanation."

[1756] User

[1757] The user checks the instructions presented by the device and responds based on their emotional state.

[1758] Actions at the end of a meeting

[1759] server

[1760] The server automatically generates "decisions" and "next actions" based on what is said during the meeting. For example, it lists specific actions such as "Add feature A: Person in charge B, deadline: next Friday."

[1761] Terminal

[1762] The terminal presents the generated decisions and next actions to the user.

[1763] User

[1764] The user reviews the proposed decisions and next actions and makes any necessary corrections.

[1765] Automatically create and share meeting minutes

[1766] server

[1767] The server automatically creates meeting minutes including decisions and next actions.

[1768] The server automatically shares the created minutes with meeting participants.

[1769] Terminal

[1770] The terminal notifies the user of the shared minutes and prompts him / her to confirm them.

[1771] User

[1772] The user can check the shared minutes on their device and prepare for the next action.

[1773] Specific examples

[1774] A specific example of how a new product planning meeting might proceed is shown below.

[1775] 1. Setting meeting goals

[1776] The user enters "Goal of new product planning meeting: Finalize specifications."

[1777] The server stores this.

[1778] 2. Creating a projected agenda

[1779] The server automatically generates tasks such as "Consider new product specifications," "Share market research results," and "Confirm design proposals," and the user adds "Confirm manufacturing schedule."

[1780] 3. Idea submission

[1781] The server provides related ideas, and the user prepares the meeting content based on them.

[1782] 4. Support during meetings

[1783] The server converts the comments into text in real time and the device displays it.

[1784] The user follows the instructions on the terminal to proceed with the conference.

[1785] 5. Emotion recognition and response using an emotion engine

[1786] The server uses an emotion engine to recognize the user's emotional state, and the device displays appropriate instructions.

[1787] For example, if the user appears dissatisfied, the terminal will notify the user that "additional explanation is required."

[1788] 6. Actions at the end of the meeting

[1789] The server generates the decisions and next actions, which the terminal presents to the user.

[1790] The user checks and corrects the data.

[1791] 7. Automatically create and share meeting minutes

[1792] The server automatically creates minutes and shares them with participants.

[1793] The terminal notifies the user, and the user checks the minutes.

[1794] This system can improve the quality of meetings and reduce meeting times. In addition, by using an emotion engine, it is possible to respond appropriately to the emotions of meeting participants, providing a better meeting environment.

[1795] The processing flow will be explained below.

[1796] Step 1:

[1797] A user logs into the application using a terminal.

[1798] Step 2:

[1799] The user inputs the goal of the meeting. For example, the user might input "Goal of the new product planning meeting: Finalize specifications."

[1800] Step 3:

[1801] The server receives the meeting goals input by the user and stores them in a database.

[1802] Step 4:

[1803] The server uses natural language processing algorithms to generate a potential agenda based on the received meeting goals.

[1804] Step 5:

[1805] The server automatically generates agenda items such as "Consider new product specifications," "Share market research results," and "Confirm design proposals."

[1806] Step 6:

[1807] The user can check the generated anticipated agenda on their device and make corrections or additions as necessary. For example, adding "Confirm manufacturing schedule" is possible.

[1808] Step 7:

[1809] The server collects relevant ideas from databases and the Internet and presents them to the user.

[1810] Step 8:

[1811] The user uses the ideas presented by the server as a reference to prepare the contents of the meeting.

[1812] Step 9:

[1813] The server converts user utterances into text in real time during the conference using a voice recognition engine.

[1814] Step 10:

[1815] The server transmits the textual content of the speech to the terminal and displays it.

[1816] Step 11:

[1817] The device uses voice recognition to convert what is said into text and displays it in real time.

[1818] Step 12:

[1819] The server monitors the progress of the conference and checks whether the agenda is being followed properly.

[1820] Step 13:

[1821] The terminal notifies the user in real time of suggestions and warnings regarding the progress of the meeting, for example, by displaying "This topic has been completed. Let's move on to the next topic."

[1822] Step 14:

[1823] The user follows the instructions on the terminal to proceed with the conference.

[1824] Step 15:

[1825] The server uses an emotion engine to analyze the user's emotional state during the conference.

[1826] Step 16:

[1827] The server generates instructions for adjusting the progress and responses of the meeting based on the user's emotion data recognized by the emotion engine.

[1828] Step 17:

[1829] The device displays the user's emotional state as recognized by the emotion engine and prompts the user to take appropriate action, such as notifying the user that "The speaker seems dissatisfied. Please provide additional explanation."

[1830] Step 18:

[1831] The user follows the instructions on the terminal and adjusts the progress of the conference as appropriate.

[1832] Step 19:

[1833] The server automatically generates "decisions" and "next actions" based on what was said during the meeting. For example, it lists specific actions such as "Add feature A: Person in charge: B, deadline: next Friday."

[1834] Step 20:

[1835] The terminal presents the generated decisions and next actions to the user.

[1836] Step 21:

[1837] The user reviews the proposed decisions and next actions and makes any necessary corrections.

[1838] Step 22:

[1839] The server automatically creates meeting minutes based on confirmed decisions and next actions.

[1840] Step 23:

[1841] The server automatically shares the created minutes with the meeting participants.

[1842] Step 24:

[1843] The terminal notifies the user of the shared minutes.

[1844] Step 25:

[1845] The user receives the minutes from the terminal and prepares for the next action.

[1846] Example 2

[1847] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1848] In conventional meeting support systems, meeting progress and content organization are often done manually, which takes time and effort and makes it easy for important information to be overlooked. Furthermore, many systems lack functionality to appropriately reflect participants' emotions and reactions to support the progress of meetings. This reduces meeting efficiency and makes it difficult to ensure quality.

[1849] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1850] In this invention, the server includes means for inputting the purpose of the meeting, means for generating an agenda based on the input purpose of the meeting, means for recognizing voice, text, and image data in real time during the meeting to support the progress of the meeting, means for monitoring the progress and agenda of the meeting and supporting the appropriate progress, means for analyzing the emotional state of users during the meeting and encouraging them to respond, means for generating decisions and next actions at the end of the meeting, and means for automatically creating minutes and sharing them with the meeting participants. This automates the progress and content organization of the meeting, enables appropriate responses based on the emotional states of the participants, and improves the quality and efficiency of the meeting.

[1851] The "purpose of the meeting" refers to the specific goal or outcome that the user is trying to achieve through the meeting.

[1852] An "agenda" refers to a list of specific topics or items to be addressed in the course of a meeting.

[1853] "Speech recognition" refers to the technology that uses a computer to analyze speech spoken during a meeting and convert it into text data.

[1854] "Texting" refers to the process of converting audio or video data into written information.

[1855] "Image data" refers to visual information, including facial expressions and movements of participants, acquired during a meeting.

[1856] "Support for meeting progress" refers to providing various support functions to ensure that meetings proceed smoothly based on the agenda.

[1857] "Progress monitoring" refers to the process of checking in real time whether progress during a meeting is proceeding as planned.

[1858] "Emotional state" refers to changes in participants' psychological states and emotions during a meeting.

[1859] "Encouraging a response" refers to providing real-time guidance on appropriate actions or responses based on participants' emotional state.

[1860] "Decisions" refer to specific outcomes or courses of action agreed upon during a meeting.

[1861] "Next actions" refers to the specific actions or tasks that should be taken next based on the decisions made in the meeting.

[1862] "Minutes" refers to an official document that summarizes what was discussed, decisions made, and next actions taken at a meeting.

[1863] "Auto-generation" refers to the process by which the system automatically generates data without user intervention.

[1864] "Sharing" refers to providing the generated information and data to all meeting participants.

[1865] The "Meeting Supporter" system of the present invention provides a series of functions for smoothly conducting meetings. This system is implemented using a software program including a server, user terminals, a network, and an emotion engine.

[1866] Hardware and software used

[1867] 1. Server

[1868] The server uses a relational database (e.g., MySQL) to manage data such as meeting goals, agendas, and statements made during meetings.

[1869] Use natural language processing algorithms (e.g., GPT-3) to automatically generate an agenda from the input meeting goals.

[1870] Use speech recognition technology (e.g., Google Cloud Speech-to-Text) to convert what is said during the meeting into text in real time.

[1871] Using an emotion engine (e.g., Microsoft Azure Emotion API), it recognizes the user's emotional state in real time during a meeting and prompts them to respond.

[1872] 2. Terminal

[1873] The user terminal functions as an interface for displaying real-time information during the meeting and receiving notifications from the emotion engine.

[1874] Using a web or mobile application, users can enter meeting goals, review and revise agendas and ideas.

[1875] Processing flow

[1876] 1. Setting meeting goals

[1877] The user logs in to the application using a terminal and enters the goal of the meeting in a text field. For example, if the user enters "Goal of the new product planning meeting: Finalize specifications," the server receives this and stores it in the database.

[1878] 2. Creating a projected agenda

[1879] The server generates an agenda using a natural language processing algorithm (e.g., GPT-3) based on the saved meeting goals. For example, it automatically generates topics such as "Consider new product specifications," "Share market research results," and "Confirm design proposals." The generated agenda is sent to the device, where the user can review it and make any necessary changes.

[1880] 3. Idea submission

[1881] The server provides ideas related to the meeting goals based on information collected from databases and the Internet. For example, specific ideas such as "a list of features required for a new product" or "improvements to an existing product" can be presented to the terminal, and the user can reflect these ideas in the meeting content.

[1882] 4. Support during meetings

[1883] The server uses speech recognition technology to convert the user's speech into text in real time, which the device displays to the user, for example, notifying them that "This topic has been completed. We will move on to the next topic." The server also monitors the progress and agenda of the meeting to ensure that it is proceeding properly.

[1884] 5. Emotion recognition and response using an emotion engine

[1885] During the conference, the server uses an emotion engine to analyze data such as the user's voice, facial expression, and text to recognize their emotional state. For example, if the user is dissatisfied, the terminal will display an instruction such as "The speaker seems dissatisfied. Please provide additional explanation." The user can then respond accordingly.

[1886] 6. Actions at the end of the meeting

[1887] The server automatically generates "decisions" and "next actions" based on what was said during the meeting. For example, it lists things like "Add feature A: Person in charge: B, deadline: next Friday." These decisions are presented to the user via their terminal, who can then confirm and make any necessary corrections.

[1888] 7. Automatically create and share meeting minutes

[1889] After the meeting ends, the server automatically creates minutes including decisions and next actions, and shares them with the meeting participants via their devices. The devices then notify the users that the minutes have been shared, and the users can confirm this.

[1890] Specific examples

[1891] A specific example of how a new product planning meeting might proceed is shown below.

[1892] 1. Setting meeting goals

[1893] Enter "Goal of new product planning meeting: Finalize specifications." The server saves this.

[1894] 2. Creating a projected agenda

[1895] "Consider new product specifications," "Share market research results," and "Confirm design proposals" are automatically generated, and the user can add "Confirm manufacturing schedule."

[1896] 3. Idea submission

[1897] Related ideas are provided and users prepare meeting content based on them.

[1898] 4. Support during meetings

[1899] The device converts comments into text in real time and displays them, and users follow instructions on the device to conduct the meeting.

[1900] 5. Emotion recognition and response using an emotion engine

[1901] The emotion engine recognizes the user's emotional state and the device displays appropriate instructions. For example, if the user appears dissatisfied, the device will notify them that "additional explanation is required."

[1902] 6. Actions at the end of the meeting

[1903] Generates decisions and next actions and presents them to the user, who can then confirm and modify them.

[1904] 7. Automatically create and share meeting minutes

[1905] Minutes are automatically created and shared with participants. The device notifies the user, who can then check the minutes.

[1906] Examples of prompt statements

[1907] Example of an input prompt for a generative AI model:

[1908] The goal of the new product planning meeting is to "finalize specifications." Based on this goal, generate the next expected agenda.

[1909] If a user expresses dissatisfaction or voices dissatisfaction during a meeting, analyze the situation and suggest appropriate ways to respond.

[1910] This enables the system to generate and present appropriate agendas and emotional responses.

[1911] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1912] Step 1: Set meeting goals

[1913] Input: A user uses a terminal to log into the application and enter the meeting goal in a text field.

[1914] Specific action: The user inputs "Goal of new product planning meeting: Finalize specifications."

[1915] Data processing: The server receives the entered data and stores it in a database.

[1916] Output: Saved meeting goal data.

[1917] Step 2: Create a visionary agenda

[1918] Input: The server retrieves the saved meeting goals from the database.

[1919] Specific operation: The server uses a natural language processing algorithm (e.g., GPT-3) to generate an agenda from the meeting goals.

[1920] Data processing: Analyze meeting goal data and generate relevant agenda items.

[1921] Output: A list of agenda items such as "Consider new product specifications," "Share market research results," and "Confirm design proposals."

[1922] Step 3: Review and revise the agenda

[1923] Input: The server generates an agenda and sends it to the user's device.

[1924] Specific operation: The user checks the agenda through the terminal and makes corrections such as adding "Confirm production schedule."

[1925] Data processing: The user's modifications are sent to the server and the database is updated.

[1926] Output: The final agenda list.

[1927] Step 4: Submit your idea

[1928] Input: The server retrieves the final agenda from the database.

[1929] Specific operation: The server collects relevant information from databases and the Internet.

[1930] Data processing: Analyze the collected information and generate concrete ideas related to the agenda.

[1931] Output: A list of ideas such as "a list of features required for new products" and "improvements to existing products."

[1932] Step 5: Present your idea

[1933] Input: The server sends the generated idea list to the user's terminal.

[1934] Specific actions: The user checks the ideas presented through the terminal and reflects them in the meeting content.

[1935] Data processing: User feedback is stored in a database.

[1936] Output: Ideas reflected in the meeting content.

[1937] Step 6: Support during the meeting

[1938] Input: The terminal collects the user's speech.

[1939] Specific operation: The server uses speech recognition technology (e.g., Google Cloud Speech-to-Text) to convert the spoken voice into text.

[1940] Data processing: Converts voice data into text data in real time.

[1941] Output: Textualized speech.

[1942] Step 7: Displaying Text

[1943] Input: The server converts the speech into text and sends it to the terminal.

[1944] Specific operation: The device displays the text of the remarks to the user in real time. For example, it notifies the user, "This topic has been concluded. We will move on to the next topic."

[1945] Data processing: Converting data into a format for display.

[1946] Output: Real-time display to the user.

[1947] Step 8: Recognize and respond to emotions

[1948] Input: The device collects voice, facial expression, and text data during the meeting.

[1949] Specific operation: The server uses an emotion engine (e.g., Microsoft Azure Emotion API) to analyze the collected data and recognize the user's emotional state.

[1950] Data processing: Analyze emotional data in real time and determine the necessary response.

[1951] Output: Response instructions based on emotional state.

[1952] Step 9: Propose a response

[1953] Input: The server sends instructions based on the user's emotional state to the device.

[1954] Specific operation: The terminal notifies the user, for example, "The speaker seems dissatisfied. Please provide additional explanation."

[1955] Data processing: Converting data into a format for display.

[1956] Output: Real-time notification to the user.

[1957] Step 10: Actions at the end of the meeting

[1958] Input: The server retrieves the content of what was said during the meeting from the database.

[1959] Specific actions: The server automatically generates "decisions" and "next actions." For example, it lists specific actions such as "Add feature A: Person in charge: B, Deadline: Next Friday."

[1960] Data processing: Analyzes the content of statements and automatically generates data.

[1961] Output: List of decisions and next actions.

[1962] Step 11: Present and confirm the action

[1963] Input: The server sends the generated decisions and next actions to the terminal.

[1964] Specific operation: The terminal displays the decision and next action to the user, and the user confirms and corrects it.

[1965] Data processing: The corrected data is sent back to the server and the database is updated.

[1966] Output: Confirmed and revised decisions and next actions.

[1967] Step 12: Automatically create and share meeting minutes

[1968] Input: The server retrieves the decision and next action from the database.

[1969] Specific operation: The server automatically creates minutes including decisions and next actions, and automatically shares them with meeting participants.

[1970] Data processing: Format the data into minutes format.

[1971] Output: Shared meeting minutes.

[1972] Step 13: Notification of minutes

[1973] Input: The server sends the minutes to each participant.

[1974] Specific operation: The terminal notifies the user that the minutes have been shared.

[1975] Data processing: Formatting of notification content.

[1976] Output: Notification to the user.

[1977] (Application example 2)

[1978] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1979] In modern business, it is important to streamline meetings and negotiations and respond to customers quickly and accurately. However, particularly in brick-and-mortar stores, sales staff often find it difficult to answer customers' questions immediately and accurately grasp and respond to their emotional state. This can lead to missed sales opportunities and a decrease in customer satisfaction. Furthermore, it is difficult to accurately record all information about the sales process and share it instantly.

[1980] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for inputting the meeting goal, a means for generating a hypothetical agenda based on the input meeting goal, a means for recognizing voice, text, images, and video in real time during the meeting to support the progress of the meeting, a means for clarifying decisions and next actions at the end of the meeting, a means for automatically creating minutes and automatically sharing them with meeting participants, a means for converting customer questions into text in real time via a display device worn by the user and providing appropriate answers, and a means for analyzing the user's emotional state and notifying the user of an optimal response based on the emotion in real time. This allows sales staff to respond to customer questions quickly and accurately and provide relevant information in real time, thereby improving customer satisfaction. Furthermore, the emotion engine allows for accurate understanding of the customer's emotional state and optimal responses, thereby providing a better sales experience.

[1981] The "means for inputting the goals of the meeting" is an interface that allows a user to input the goals and expected outcomes of the meeting into the system as text, and the system receives and stores them.

[1982] The "means for generating a potential agenda" is a function that uses a natural language processing algorithm to automatically generate a meeting agenda and progress schedule based on the input meeting goal information.

[1983] "Means to support the progress of the meeting" refers to a function in which the system recognizes voice, text, images, and video in real time during the meeting, monitors the progress of the meeting, and provides appropriate instructions and information.

[1984] "Means for clarifying decisions and next actions" refers to a function in which the system clearly indicates the specific next actions to be taken, the person in charge, deadlines, etc. based on the participants' comments and decisions made at the end of the meeting.

[1985] "Means for automatically creating meeting minutes and automatically sharing them with meeting participants" refers to a function that allows the system to automatically create meeting minutes based on comments and decisions made during the meeting and share them online with all meeting participants.

[1986] "Means of converting customer questions into text in real time through a display device worn by the user and providing appropriate answers" refers to a function that uses a display device such as a head-mounted display (HMD) worn by sales staff to convert customer questions into text using voice recognition technology, and then the system provides appropriate answers in real time based on that text.

[1987] "Means for analyzing the user's emotional state and notifying the optimal response based on that emotion in real time" refers to a function in which the system uses an emotion engine to analyze the voice, facial expression, and other data of the user or customer, and notifies the user in real time of the optimal response based on that emotional state.

[1988] The "sales supporter for brick-and-mortar stores" of the present invention supports sales in brick-and-mortar stores using a system including a server, user terminals, a network, and an emotion engine. The system aims to enable sales staff to respond to customers appropriately and quickly, thereby improving customer satisfaction.

[1989] Initial Setup

[1990] Hardware

[1991] Display device: Head-mounted display (HMD) (e.g., Magic Leap 2)

[1992] Input devices: Microphone, camera

[1993] software

[1994] Emotion Engine: Software for analyzing the emotional state of customers

[1995] Speech recognition system: a system that converts speech into text

[1996] Natural language processing algorithms: Algorithms that analyze text data and generate relevant information

[1997] Cloud server: Cloud services such as AWS

[1998] Real-time database: Firebase, etc.

[1999] System Operation

[2000] Goal setting

[2001] The user puts on the HMD and logs in. The server displays the sales target and saves it in the database.

[2002] Agenda Generation

[2003] The server generates daily tasks and product explanation scripts based on sales targets, which the user can review and adjust as needed.

[2004] Idea submission

[2005] The server provides product features and related information, which the user uses to explain the product to customers, such as the features of new products and campaign information.

[2006] Real-time Support

[2007] The server uses voice recognition technology and natural language processing algorithms to convert customer questions into text and display them on the HMD in real time. The user can then respond to the customer while checking the appropriate answer on the HMD.

[2008] emotion recognition

[2009] The server uses an emotion engine to analyze the customer's facial expressions and tone of voice to identify their emotional state. The HMD then notifies the user of the optimal response based on the customer's emotional state. For example, if the customer is dissatisfied, the HMD will notify the user that "further explanation is required."

[2010] When sales end

[2011] The server generates a list of next actions and customer follow-ups when a sale is completed, and displays it on the HMD. The user can then confirm and adjust the next actions.

[2012] Report creation and sharing

[2013] The server automatically generates a report based on sales results and customer feedback and notifies the user via the HMD. The user can then check and correct the report and take any necessary action.

[2014] Specific examples

[2015] For example, if the user aims to sell new product A, the user inputs "Today's goal: Sell 20 units of new product A" into the HMD. The server saves this and generates a task list related to the sales goal. Also, if a customer asks "What are the features of new product A?" during a sale, the server immediately converts the question into text, and the user provides an explanation based on the answer displayed on the HMD. Furthermore, if the customer shows signs of dissatisfaction, the HMD notifies the user that "Further explanation is required."

[2016] Prompt Sentence Examples

[2017] What are the features of new product A?

[2018]

[2019] My customer looks unhappy. How should I respond?

[2020] In this way, the system of the present invention supports effective and efficient customer service in physical stores, enabling sales staff to provide optimal service.

[2021] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2022] Step 1:

[2023] The user puts on the HMD and logs in. The user enters their sales target in text format into the system. The server receives the entered sales target and stores it in a cloud database. For example, the user might enter "Today's target: Sell 20 units of new product A."

[2024] Input: Sales target text input

[2025] Data processing: Save and analyze received text data

[2026] Output: Save to cloud database

[2027] Step 2:

[2028] The server uses natural language processing algorithms based on sales target information to generate scripts for daily tasks and product descriptions, which are then displayed on the HMD for the user to review and adjust as needed.

[2029] Input: Sales target information

[2030] Data processing: Script generation using natural language processing algorithms

[2031] Output: Script display on HMD

[2032] Step 3:

[2033] The server collects product features and related information and generates this information through the idea provision function. Based on the information provided, the user prepares an explanation for the customer. For example, they can provide information about the features of a new product or campaign information.

[2034] Input: Sales target and related data

[2035] Data processing: information gathering and idea generation

[2036] Output: Information displayed on the HMD

[2037] Step 4:

[2038] If a customer has a question during a sale, the user receives the question through the microphone in the HMD. The server uses voice recognition technology to convert the customer's question into text and display it on the user's HMD in real time.

[2039] Input: Customer Question

[2040] Data processing: speech recognition and text conversion

[2041] Output: Text display on HMD

[2042] Step 5:

[2043] The user provides the customer with an appropriate answer based on the questions displayed on the HMD, including the process of retrieving the most appropriate answer from the information database held by the server.

[2044] Input: Texted question

[2045] Data processing: Information retrieval from databases and optimal answer generation

[2046] Output: Display the answer on the HMD

[2047] Step 6:

[2048] The server uses an emotion engine to analyze the customer's facial expressions and tone of voice via camera and voice input, and identifies the customer's emotional state. For example, it can estimate the customer's emotions based on voice tone and facial recognition technology.

[2049] Input: Customer facial expression data, voice data

[2050] Data processing: Analysis with emotion engine

[2051] Output: Customer emotional state identification result

[2052] Step 7:

[2053] Based on the emotional state, the server notifies the user in real time on the HMD with an appropriate response, such as displaying a message like, "The customer appears dissatisfied. Further explanation is required."

[2054] Input: Customer's emotional state

[2055] Data processing: generating response instructions

[2056] Output: Real-time notification to HMD

[2057] Step 8:

[2058] The server generates a list of next actions and customer follow-ups upon sale completion, which are displayed to the user via the HMD and adjustments made as necessary.

[2059] Input: Sales completion data

[2060] Data processing: Next action generation and follow-up list creation

[2061] Output: List display on HMD

[2062] Step 9:

[2063] The server automatically generates a report based on sales results and customer feedback and notifies the user via the HMD. The user can then review the report and make any necessary corrections.

[2064] Input: Sales results and feedback

[2065] Data Processing: Report Generation

[2066] Output: Report notification to HMD

[2067] By following these steps, the "Sales Supporter for Physical Stores" enables sales staff to efficiently and effectively serve customers, improving customer satisfaction.

[2068] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[2069] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2070] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[2071] [Fourth embodiment]

[2072] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[2073] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[2074] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[2075] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[2076] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[2077] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[2078] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[2079] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[2080] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[2081] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[2082] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[2083] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[2084] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2085] A detailed embodiment of the "meeting supporter" according to the present invention will now be described. This system includes a server, a user terminal, and a software program for supporting a meeting via a network.

[2086] Meeting goal setting

[2087] User

[2088] The user logs in to the application from the terminal to be used.

[2089] Before the meeting, the user inputs the goal of the meeting in the text field. For example, the user might input "Goal of the new product planning meeting: Finalize specifications."

[2090] server

[2091] The server receives the meeting goals input by the user and stores them in a database.

[2092] The server collects relevant data based on the goal information.

[2093] Creating a projected agenda

[2094] server

[2095] The server uses natural language processing algorithms to generate a potential agenda based on the meeting goals.

[2096] For example, topics such as "Consider new product specifications," "Share market research results," and "Confirm design proposals" are automatically generated.

[2097] User

[2098] The user can check the generated planned agenda through the terminal and make corrections as necessary, for example, adding "Confirm manufacturing schedule."

[2099] Idea submission

[2100] server

[2101] The server presents ideas related to the meeting goals based on information collected from databases and the Internet.

[2102] The ideas presented include a list of features required for new products and improvements to existing products.

[2103] User

[2104] The user reflects the provided ideas in the content of the meeting.

[2105] In-meeting support

[2106] server

[2107] The server recognizes what users say during the meeting in real time and converts it into text.

[2108] The server monitors the progress of the meeting and ensures that the agenda is being followed.

[2109] Terminal

[2110] The terminal displays the textual utterances to the user in real time.

[2111] For example, the terminal may notify, "This topic has been completed. We will move on to the next topic."

[2112] User

[2113] Users can smoothly conduct the conference based on the display on their terminals.

[2114] Actions at the end of a meeting

[2115] server

[2116] The server automatically generates "decisions" and "next actions" based on what is said during the meeting.

[2117] For example, specific actions such as "Add feature A: Responsible person B, Deadline: next Friday" are listed.

[2118] Terminal

[2119] The terminal presents the generated decisions and next actions to the user, allowing them to review and modify them.

[2120] User

[2121] The user reviews the proposed decisions and next actions and makes any necessary corrections.

[2122] Automatically create and share meeting minutes

[2123] server

[2124] The server automatically creates meeting minutes including decisions and next actions.

[2125] The server automatically shares the created minutes with meeting participants.

[2126] Terminal

[2127] The terminal notifies the user of the shared minutes and prompts him / her to confirm them.

[2128] User

[2129] Users can check the shared minutes on their own devices and prepare for their next actions.

[2130] Specific examples

[2131] The following steps are taken during the new product planning meeting:

[2132] 1. Setting meeting goals

[2133] The user enters "Goal of new product planning meeting: Finalize specifications."

[2134] The server stores this.

[2135] 2. Creating a projected agenda

[2136] The server automatically generates tasks such as "Consider new product specifications," "Share market research results," and "Confirm design proposals," and the user adds "Confirm manufacturing schedule."

[2137] 3. Idea submission

[2138] The server provides related ideas, and users use them to prepare for the meeting.

[2139] 4. Support during meetings

[2140] The server converts the comments into text in real time and the device displays it.

[2141] The user follows the instructions on the terminal to proceed with the conference.

[2142] 5. Actions at the end of the meeting

[2143] The server generates the decisions and next actions, which the terminal presents to the user.

[2144] The user checks and corrects the data.

[2145] 6. Automatically create and share meeting minutes

[2146] The server automatically creates minutes and shares them with participants.

[2147] The terminal notifies the user, and the user checks the minutes.

[2148] In this way, the present invention can improve the quality of the conference and reduce the time it takes to hold the conference.

[2149] The processing flow will be explained below.

[2150] Step 1:

[2151] A user logs into the application using a terminal.

[2152] Step 2:

[2153] The user inputs the goal of the meeting. For example, the user might input "Goal of the new product planning meeting: Finalize specifications."

[2154] Step 3:

[2155] The server receives the meeting goals input by the user.

[2156] Step 4:

[2157] The server stores the received meeting goals in a database.

[2158] Step 5:

[2159] The server automatically generates a hypothetical agenda based on the meeting goals, such as "discussing new product specifications," "sharing market research results," and "confirming design proposals."

[2160] Step 6:

[2161] The user checks the generated assumed agenda through the terminal.

[2162] Step 7:

[2163] The user can modify the expected agenda as needed, for example adding "Confirm manufacturing schedule."

[2164] Step 8:

[2165] The server collects relevant ideas from a stored database or from the Internet.

[2166] Step 9:

[2167] The server presents the collected ideas to the user, such as a list of features required for a new product or improvements to an existing product.

[2168] Step 10:

[2169] The user uses the presented ideas as a reference to prepare the contents of the meeting.

[2170] Step 11:

[2171] The server converts users' comments into text in real time during the meeting using voice recognition technology.

[2172] Step 12:

[2173] The server monitors the progress of the conference and checks whether the agenda is being followed properly.

[2174] Step 13:

[2175] The terminal displays the content of the speech converted into text by speech recognition to the user.

[2176] Step 14:

[2177] The terminal displays suggestions and notes for the user regarding the progress of the meeting in real time, such as "This agenda item has been completed. Let's move on to the next agenda item."

[2178] Step 15:

[2179] The user follows the instructions on the terminal to ensure the conference proceeds smoothly.

[2180] Step 16:

[2181] The server automatically generates "decisions" and "next actions" based on what is said during the meeting. For example, it lists specific actions such as "Add feature A: Person in charge: B, deadline: next Friday."

[2182] Step 17:

[2183] The terminal presents the generated decisions and next actions to the user.

[2184] Step 18:

[2185] The user reviews the proposed decisions and next actions and makes any necessary corrections.

[2186] Step 19:

[2187] The server automatically creates meeting minutes based on confirmed decisions and next actions.

[2188] Step 20:

[2189] The server automatically shares the created minutes with the meeting participants.

[2190] Step 21:

[2191] The terminal notifies the user of the shared minutes.

[2192] Step 22:

[2193] The user receives the minutes from the terminal and prepares for the next action.

[2194] By dividing the steps in this way, the system of the present invention improves the quality of the conference and reduces the time it takes to hold the conference.

[2195] Example 1

[2196] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2197] In many modern meetings, many processes, such as proceeding, taking minutes, and deciding on post-meeting actions, are done manually, resulting in a decrease in meeting efficiency. It is also difficult for all participants to understand the content and progress of the meeting, which often leads to a decline in meeting quality. There is a need for technology that can solve this problem and make meetings more effective.

[2198] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[2199] In this invention, the server includes: means for a user to input the goal of the meeting; means for the server to save the input goal of the meeting; means for the server to generate a hypothetical agenda based on the goal of the meeting using a natural language processing algorithm; means for the user to confirm and modify the generated hypothetical agenda; means for the server to collect information related to the goal of the meeting from a database and the Internet and provide ideas; means for the server to recognize and convert user comments in real time during the meeting into text; means for a terminal to display the converted text in real time; means for the server to generate decisions and next actions based on the content of the meeting at the end of the meeting; means for the server to present the generated decisions and next actions to the user; and means for the server to automatically create minutes and share them with meeting participants. This automates the process of understanding the progress and decisions of the meeting, and the creation and sharing of minutes, thereby improving the efficiency and quality of meetings.

[2200] The "meeting goal" represents the specific objective or purpose that the meeting aims to achieve.

[2201] "User" refers to the person who sets up and conducts the meeting and operates each function of the system.

[2202] "Terminal" means a hardware device used by a User to access the System, enter information, and receive notifications from the System.

[2203] "Server" means a central computer facility that manages the entire system and performs data processing.

[2204] "Text conversion" refers to the process of converting audio data into text data.

[2205] "Natural language processing algorithms" are technical methods that allow computers to understand and generate human language, including grammatical analysis and semantic analysis.

[2206] "Database" refers to a digital repository used to efficiently store and retrieve structured data.

[2207] "Internet" refers to the global network that interconnects computer networks around the world.

[2208] "Proceeding a meeting" refers to activities to ensure that each agenda item is properly advanced according to the meeting plan.

[2209] "Decisions" refer to specific actions or decisions agreed upon during a meeting.

[2210] "Next actions" refer to specific tasks or steps to be taken after the meeting.

[2211] "Minutes" refers to an official document that records what was discussed, decisions made, and next actions taken during a meeting.

[2212] The "Complete Meeting Supporter" of this invention is a system that includes a server, a user terminal, and a software program for supporting meetings over a network. This system improves the efficiency and quality of meetings through the following steps. The main hardware used is a user terminal (e.g., a personal computer, tablet, or smartphone) and a server. The software technologies used include natural language processing algorithms, database management systems, and real-time voice recognition technology.

[2213] Meeting goal setting

[2214] User

[2215] The user logs into the application from their own terminal.

[2216] After logging in to the application, the user enters the goal of the meeting in a text field, for example, "Goal of new product planning meeting: Finalize specifications."

[2217] server

[2218] The server receives the meeting goals entered by the user and stores them in a database.

[2219] After saving, the server begins to collect relevant data based on the goals of the meeting.

[2220] Creating a projected agenda

[2221] server

[2222] The server uses natural language processing algorithms to generate a potential agenda based on the meeting goals.

[2223] For example, topics such as "Consider new product specifications," "Share market research results," and "Confirm design proposals" are automatically generated.

[2224] User

[2225] The user can check the generated planned agenda through the terminal and make corrections as necessary, for example, adding "Confirm manufacturing schedule."

[2226] Idea submission

[2227] server

[2228] The server presents ideas related to the meeting goals based on information collected from databases and the Internet.

[2229] The ideas presented include a list of features required for new products and improvements to existing products.

[2230] User

[2231] The user reviews the ideas provided and incorporates them into the content of the meeting.

[2232] In-meeting support

[2233] server

[2234] The server recognizes what users say during the meeting in real time and converts it into text.

[2235] The server monitors the progress of the meeting and ensures that the agenda is being followed.

[2236] Terminal

[2237] The terminal displays the textual utterances to the user in real time.

[2238] For example, the terminal may notify, "This topic has been completed. We will move on to the next topic."

[2239] User

[2240] Users can smoothly conduct the conference based on the display on their terminals.

[2241] Actions at the end of a meeting

[2242] server

[2243] The server automatically generates "decisions" and "next actions" based on what is said during the meeting.

[2244] For example, specific actions such as "Add feature A: Responsible person B, Deadline: next Friday" are listed.

[2245] Terminal

[2246] The terminal presents the generated decisions and next actions to the user, allowing them to review and modify them.

[2247] User

[2248] The user reviews the proposed decisions and next actions and makes any necessary corrections.

[2249] Automatically create and share meeting minutes

[2250] server

[2251] The server automatically creates meeting minutes including decisions and next actions.

[2252] The minutes created are automatically shared by the server with all meeting participants.

[2253] Terminal

[2254] The terminal notifies the user that the minutes have been shared.

[2255] User

[2256] Users can check the shared minutes on their own devices and prepare for their next actions.

[2257] Specific examples

[2258] The following steps are taken during the new product planning meeting:

[2259] 1. Setting meeting goals

[2260] The user enters "Goal of new product planning meeting: Finalize specifications."

[2261] The server stores this.

[2262] 2. Creating a projected agenda

[2263] The server automatically generates tasks such as "Consider new product specifications," "Share market research results," and "Confirm design proposals," and the user adds "Confirm manufacturing schedule."

[2264] 3. Idea submission

[2265] The server provides related ideas, and users use them to prepare for the meeting.

[2266] 4. Support during meetings

[2267] The server converts the comments into text in real time and the device displays it.

[2268] The user follows the instructions on the terminal to proceed with the conference.

[2269] 5. Actions at the end of the meeting

[2270] The server generates the decisions and next actions, which the terminal presents to the user.

[2271] The user checks and corrects the data.

[2272] 6. Automatically create and share meeting minutes

[2273] The server automatically creates minutes and shares them with participants.

[2274] The terminal notifies the user, and the user checks the minutes.

[2275] In this way, the present invention can improve the quality of the conference and reduce the time it takes to hold the conference.

[2276] Prompt Sentence Examples

[2277] "Please create the following agenda for the new product planning meeting, with the goal of finalizing specifications."

[2278] The flow of the identification process in the first embodiment will be described with reference to FIG.

[2279] Step 1: Set meeting goals

[2280] User

[2281] The user logs into the application from their own terminal.

[2282] After logging in, users enter the meeting goal in a text field within the application.

[2283] As an example of input, enter "Goal of new product planning meeting: Finalize specifications."

[2284] The entered goal is sent from the terminal to the server.

[2285] server

[2286] The server receives the goal of the meeting transmitted from the user terminal.

[2287] The received goal information is stored in a database.

[2288] Based on the saved goals, relevant data is collected and stored in a database.

[2289] Input: Meeting goal ("Goal of new product planning meeting: Finalize specifications").

[2290] Output: Meeting goal data stored in a database.

[2291] Step 2: Create a visionary agenda

[2292] server

[2293] The server runs natural language processing algorithms based on the meeting goals stored in a database.

[2294] The algorithm extracts topics related to the goal and generates a hypothetical agenda.

[2295] For example, topics such as "Consider new product specifications," "Share market research results," and "Confirm design proposals" are automatically generated.

[2296] User

[2297] The user checks the generated assumed agenda through the terminal.

[2298] If necessary, the user edits the agenda and adds a new item, for example, "Confirm manufacturing schedule."

[2299] Input: Meeting goal data stored in a database.

[2300] Output: An automatically generated proposed agenda.

[2301] Step 3: Submit your idea

[2302] server

[2303] The server collects information related to the meeting goals from a database or the Internet.

[2304] From the collected information, ideas related to the content of the meeting are generated and presented to the user.

[2305] The ideas presented include a list of features required for new products and improvements to existing products.

[2306] User

[2307] The user checks the provided ideas through the terminal.

[2308] Users can incorporate relevant ideas into the meeting agenda.

[2309] Input: Collected information based on meeting goal data.

[2310] Output: A list of ideas presented to the user.

[2311] Step 4: Support during the meeting

[2312] server

[2313] The server recognizes what users say during the meeting in real time and converts it into text.

[2314] A speech recognition algorithm is used to recognize what is being said.

[2315] The textual comments are sent to a server and recorded.

[2316] The server monitors the progress of the meeting and ensures that the agenda is being followed.

[2317] Terminal

[2318] The terminal displays real-time textual comments sent from the server.

[2319] The device notifies meeting participants of the progress, for example by displaying a notification such as "This topic has been completed. We will move on to the next topic."

[2320] User

[2321] The user proceeds with the conference based on notifications from the terminal.

[2322] Input: User's spoken utterance.

[2323] Output: Server-generated text of speech data and progress notifications.

[2324] Step 5: Actions at the end of the meeting

[2325] server

[2326] The server aggregates the recorded comments made during the meeting and automatically generates "decisions" and "next actions."

[2327] For example, specific actions such as "Add feature A: Responsible person B, Deadline: next Friday" are listed.

[2328] Terminal

[2329] The terminal presents the generated decisions and next actions to the user.

[2330] The user checks these contents and makes corrections as necessary.

[2331] User

[2332] The user reviews the proposed decisions and next actions and applies appropriate corrections.

[2333] Input: Speech data recorded during the meeting.

[2334] Output: Auto-generated decisions and next action list.

[2335] Step 6: Automatically create and share meeting minutes

[2336] server

[2337] The server automatically creates minutes based on the decisions made during the meeting and the next actions.

[2338] The created minutes are shared with all meeting participants by the server.

[2339] Terminal

[2340] The terminal notifies the user that the minutes have been shared.

[2341] Users can review the shared minutes on their own devices and provide feedback or corrections as needed.

[2342] User

[2343] The user checks the minutes and prepares for the next action.

[2344] Input: Decision and next action data.

[2345] Output: Automatically generated meeting minutes.

[2346] (Application example 1)

[2347] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2348] In modern factories, efficient meeting management and work planning are essential to improving work efficiency and reducing errors. However, existing meeting support systems are primarily designed for office work and are often not suitable for use on factory floors. Furthermore, while it is required that comments be recorded in real time during meetings and that next actions be smoothly shared, this is not easy to achieve. Furthermore, there is a lack of a system that allows relevant information and ideas based on goals to be quickly provided, further improving work efficiency on-site.

[2349] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2350] In this invention, the server includes: [means for inputting meeting goals]; [means for generating a hypothetical agenda based on the input meeting goals]; [means for recognizing voice, text, images, and video in real time during the meeting to support the progress of the meeting]; [means for clarifying decisions and next actions at the end of the meeting]; [means for automatically creating minutes and automatically sharing them with meeting participants]; [means for providing relevant ideas based on the goals of factory work and recording comments with timestamps]; and [means for monitoring the progress of factory work and displaying important comments and action items in real time]. This enables meetings to be run efficiently, improving factory work efficiency and reducing errors. Furthermore, by clarifying and quickly sharing next actions, work efficiency on the factory floor can be further improved.

[2351] A "meeting goal" indicates the specific purpose or objective to be achieved in the meeting.

[2352] A "potential agenda" is a list of topics prepared in advance based on the goals of the meeting.

[2353] "Speech recognition" is a technology that converts speech spoken during a meeting into text in real time.

[2354] "Text conversion" is the process of converting voice data obtained by voice recognition into text information.

[2355] "Image recognition" is a technology that analyzes image data used during a meeting and extracts necessary information.

[2356] "Video recognition" is a technology that analyzes video data during a meeting and identifies important scenes, people, and objects.

[2357] "Decisions" are specific actions or policies agreed upon during a meeting.

[2358] "Next actions" are specific tasks or steps that need to be carried out based on decisions.

[2359] Minutes are a record of what was said during a meeting, decisions made, next actions, etc.

[2360] "Factory work" refers to activities such as production, maintenance, and inspection that take place within a factory.

[2361] A "timestamp" is a mark that indicates the date and time when the information was generated.

[2362] "Real-time display" means the immediate display of ongoing information or data.

[2363] Ideation is the process of offering helpful suggestions or thoughts related to the goals of a meeting or task.

[2364] The "factory work support system" of the present invention provides technology to significantly improve the efficiency of meetings and discussions in factories. This system includes a server, user terminals, and a software program for supporting meetings in factories via a network.

[2365] The server generates a hypothetical agenda based on the meeting goals entered by the user, and recognizes and analyzes voice, text, images, and video in real time to support the progress of the meeting. This allows important comments and decisions made during the meeting to be instantly recorded, and the next action to be taken is automatically clarified at the end of the meeting. Furthermore, the system also has a function to automatically create meeting minutes and share them with meeting participants. It also has a function to provide relevant ideas based on the goal of work within the factory and record comments with timestamps.

[2366] Hardware and Software Configuration

[2367] The main components of the system are:

[2368] Server: Equipped with a high-performance processor and large-capacity storage, it processes and stores data related to meeting support.

[2369] User terminal: A device that can be worn or carried by a factory worker, such as a smartphone, smart glasses, or head-mounted display.

[2370] Software: Programs implemented using Python, natural language processing libraries (e.g., spaCy), and generative AI models (e.g., GPT-3).

[2371] Data Processing and Computation

[2372] The server manages a series of processes from setting meeting goals to automatically creating meeting minutes. The main data processing and calculations are explained below:

[2373] 1. Setting the goal of the meeting: The user inputs the goal of the meeting through the terminal. The server receives and stores the information.

[2374] 2. Generation of the expected agenda: The server generates the expected agenda based on the input goal information using a natural language processing algorithm.

[2375] 3. Real-time recognition: The server recognizes and analyzes the voice, text, images, and video during the meeting in real time and displays them on the terminal.

[2376] 4. Recording speech: User speech is converted into text in real time and saved with a timestamp.

[2377] 5. Providing related ideas: The server provides ideas related to the goal based on information collected from databases and the Internet.

[2378] 6. Generation of next actions: At the end of the meeting, the server automatically generates decisions and next actions and displays them on the terminal.

[2379] 7. Automatic minutes creation: The server creates minutes based on all meeting records and automatically shares them with participants.

[2380] Specific examples

[2381] Below is a specific example from a factory meeting regarding the introduction of safety devices for a new product.

[2382] 1. Input example: The user inputs "Meeting regarding the introduction of new product safety devices" into the terminal.

[2383] 2. Generation example: The server automatically generates agendas such as "Confirmation of work procedures," "Discussion of safety measures," and "Proposals for improving efficiency."

[2384] 3. Example of idea provision: The server provides "ideas for introducing new safety devices," and users proceed with the meeting based on these ideas.

[2385] 4. Example of real-time recording: When a user says during a meeting, "Let's consider automation to improve efficiency," the statement is converted into text in real time and recorded with a timestamp.

[2386] 5. Example of next action generation: The server automatically generates and presents an action such as "Testing new safety device: Engineering department, deadline: 2023-12-01."

[2387] 6. Example of automatic minutes creation: After the meeting ends, the server automatically creates minutes including all statements, decisions, and next actions, and shares them with the participants.

[2388] Prompt Sentence Examples

[2389] "Please tell us your ideas for implementing new safety equipment."

[2390] As described above, the present invention enables meetings and discussions within a factory to be conducted efficiently, improving work efficiency and reducing mistakes.

[2391] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2392] Step 1:

[2393] The user inputs the goal of the meeting. Specifically, the user logs into the application from their terminal and inputs the goal of the meeting into a text field. For example, the user might input "Meeting regarding the introduction of new product safety devices." The input goal information is sent to the server. The server receives and saves the input goal information. Input: Meeting goal input by the user, Output: Saved goal information.

[2394] Step 2:

[2395] The server generates a hypothetical agenda based on the input meeting goals. Specifically, the server analyzes goal information using a natural language processing algorithm and automatically generates an agenda such as "confirmation of work procedures," "discussion of safety measures," and "proposals for improving efficiency." The generated agenda is sent to the user's device. Input: saved goal information, Output: generated hypothetical agenda.

[2396] Step 3:

[2397] The user checks the generated expected agenda through the terminal and modifies it as necessary. For example, adding "test plan for new product" as an "additional consideration item." The modified agenda is sent back to the server and saved. Input: Generated expected agenda, Output: Modified agenda.

[2398] Step 4:

[2399] The server provides ideas related to the goals and agenda. Specifically, it uses a generative AI model (e.g., GPT-3) to collect relevant information from databases and the Internet and generate specific proposals such as "ideas for introducing new safety devices." These ideas are sent to the user's device. Input: goal information and agenda, output: provided ideas.

[2400] Step 5:

[2401] During a meeting, the server recognizes voice, text, images, and video in real time to support the progress of the meeting. Specifically, it uses voice recognition technology to convert comments into text, and image and video recognition technology to analyze important scenes. The recognized information is displayed on the device in real time, prompting the user to move on to the next agenda item as necessary. Input: Voice, text, images, and video data during the meeting. Output: Text comments displayed in real time and notifications about the next agenda item.

[2402] Step 6:

[2403] The server records user comments with timestamps and clarifies decisions and next actions at the end of the meeting. Specifically, it analyzes the content of comments and generates specific action items such as "Test new safety devices: Engineering department, deadline: 2023-12-01." The generated action items are sent to the user's device, where they can be checked and modified. Input: Texted comments, Output: Decisions and next actions.

[2404] Step 7:

[2405] The server automatically creates minutes based on all records of the meeting. Specifically, it compiles the recorded remarks, decisions, and next actions, and generates minutes according to a format. The generated minutes are automatically shared with meeting participants, and notifications are sent to user devices. Input: Recorded remarks, decisions, next actions. Output: Automatically created minutes and shared notifications.

[2406] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2407] This section describes a detailed embodiment of a system that combines an emotion engine with the "meeting supporter" of the present invention. This system supports meetings using a software program that includes a server, user terminals, a network, and an emotion engine.

[2408] Meeting goal setting

[2409] User

[2410] The user logs in to the application from the terminal to be used.

[2411] Before the meeting, the user inputs the goal of the meeting in the text field. For example, the user might input "Goal of the new product planning meeting: Finalize specifications."

[2412] server

[2413] The server receives the meeting goals input by the user.

[2414] The server stores the received meeting goals in a database.

[2415] Creating a projected agenda

[2416] server

[2417] The server uses natural language processing algorithms to generate a potential agenda based on the meeting goals.

[2418] For example, it automatically generates agenda items such as "Consider new product specifications," "Share market research results," and "Confirm design proposals."

[2419] User

[2420] The user can check the generated planned agenda through the terminal and make corrections as necessary, for example, adding "Confirm manufacturing schedule."

[2421] Idea submission

[2422] server

[2423] The server presents ideas related to the meeting goals based on information collected from databases and the Internet.

[2424] The ideas presented include a list of features required for new products and improvements to existing products.

[2425] User

[2426] The user reflects the provided ideas in the content of the meeting.

[2427] In-meeting support

[2428] server

[2429] The server converts users' comments into text in real time during the meeting using voice recognition technology.

[2430] The server monitors the progress of the meeting and checks whether the agenda is being followed properly.

[2431] Terminal

[2432] The terminal displays the textual utterances to the user in real time.

[2433] For example, the terminal may notify, "This topic has been completed. We will move on to the next topic."

[2434] Emotion recognition and response using emotion engine

[2435] server

[2436] The server uses an emotion engine to analyze the user's voice, facial expression, and text data during the meeting to recognize the user's emotional state.

[2437] For example, it detects when a user is in a dissatisfied state.

[2438] Terminal

[2439] The device prompts the user in real time to respond appropriately based on their emotional state, for example by notifying them that "The speaker seems dissatisfied. Please provide additional explanation."

[2440] User

[2441] The user checks the instructions presented by the device and responds based on their emotional state.

[2442] Actions at the end of a meeting

[2443] server

[2444] The server automatically generates "decisions" and "next actions" based on what is said during the meeting. For example, it lists specific actions such as "Add feature A: Person in charge B, deadline: next Friday."

[2445] Terminal

[2446] The terminal presents the generated decisions and next actions to the user.

[2447] User

[2448] The user reviews the proposed decisions and next actions and makes any necessary corrections.

[2449] Automatically create and share meeting minutes

[2450] server

[2451] The server automatically creates meeting minutes including decisions and next actions.

[2452] The server automatically shares the created minutes with meeting participants.

[2453] Terminal

[2454] The terminal notifies the user of the shared minutes and prompts him / her to confirm them.

[2455] User

[2456] The user can check the shared minutes on their device and prepare for the next action.

[2457] Specific examples

[2458] A specific example of how a new product planning meeting might proceed is shown below.

[2459] 1. Setting meeting goals

[2460] The user enters "Goal of new product planning meeting: Finalize specifications."

[2461] The server stores this.

[2462] 2. Creating a projected agenda

[2463] The server automatically generates tasks such as "Consider new product specifications," "Share market research results," and "Confirm design proposals," and the user adds "Confirm manufacturing schedule."

[2464] 3. Idea submission

[2465] The server provides related ideas, and the user prepares the meeting content based on them.

[2466] 4. Support during meetings

[2467] The server converts the comments into text in real time and the device displays it.

[2468] The user follows the instructions on the terminal to proceed with the conference.

[2469] 5. Emotion recognition and response using an emotion engine

[2470] The server uses an emotion engine to recognize the user's emotional state, and the device displays appropriate instructions.

[2471] For example, if the user appears dissatisfied, the terminal will notify the user that "additional explanation is required."

[2472] 6. Actions at the end of the meeting

[2473] The server generates the decisions and next actions, which the terminal presents to the user.

[2474] The user checks and corrects the data.

[2475] 7. Automatically create and share meeting minutes

[2476] The server automatically creates minutes and shares them with participants.

[2477] The terminal notifies the user, and the user checks the minutes.

[2478] This system can improve the quality of meetings and reduce meeting times. In addition, by using an emotion engine, it is possible to respond appropriately to the emotions of meeting participants, providing a better meeting environment.

[2479] The processing flow will be explained below.

[2480] Step 1:

[2481] A user logs into the application using a terminal.

[2482] Step 2:

[2483] The user inputs the goal of the meeting. For example, the user might input "Goal of the new product planning meeting: Finalize specifications."

[2484] Step 3:

[2485] The server receives the meeting goals input by the user and stores them in a database.

[2486] Step 4:

[2487] The server uses natural language processing algorithms to generate a potential agenda based on the received meeting goals.

[2488] Step 5:

[2489] The server automatically generates agenda items such as "Consider new product specifications," "Share market research results," and "Confirm design proposals."

[2490] Step 6:

[2491] The user can check the generated anticipated agenda on their device and make corrections or additions as necessary. For example, adding "Confirm manufacturing schedule" is possible.

[2492] Step 7:

[2493] The server collects relevant ideas from databases and the Internet and presents them to the user.

[2494] Step 8:

[2495] The user uses the ideas presented by the server as a reference to prepare the contents of the meeting.

[2496] Step 9:

[2497] The server converts user utterances into text in real time during the conference using a voice recognition engine.

[2498] Step 10:

[2499] The server transmits the textual content of the speech to the terminal and displays it.

[2500] Step 11:

[2501] The device uses voice recognition to convert what is said into text and displays it in real time.

[2502] Step 12:

[2503] The server monitors the progress of the conference and checks whether the agenda is being followed properly.

[2504] Step 13:

[2505] The terminal notifies the user in real time of suggestions and warnings regarding the progress of the meeting, for example, by displaying "This topic has been completed. Let's move on to the next topic."

[2506] Step 14:

[2507] The user follows the instructions on the terminal to proceed with the conference.

[2508] Step 15:

[2509] The server uses an emotion engine to analyze the user's emotional state during the conference.

[2510] Step 16:

[2511] The server generates instructions for adjusting the progress and responses of the meeting based on the user's emotion data recognized by the emotion engine.

[2512] Step 17:

[2513] The device displays the user's emotional state as recognized by the emotion engine and prompts the user to take appropriate action, such as notifying the user that "The speaker seems dissatisfied. Please provide additional explanation."

[2514] Step 18:

[2515] The user follows the instructions on the terminal and adjusts the progress of the conference as appropriate.

[2516] Step 19:

[2517] The server automatically generates "decisions" and "next actions" based on what was said during the meeting. For example, it lists specific actions such as "Add feature A: Person in charge: B, deadline: next Friday."

[2518] Step 20:

[2519] The terminal presents the generated decisions and next actions to the user.

[2520] Step 21:

[2521] The user reviews the proposed decisions and next actions and makes any necessary corrections.

[2522] Step 22:

[2523] The server automatically creates meeting minutes based on confirmed decisions and next actions.

[2524] Step 23:

[2525] The server automatically shares the created minutes with the meeting participants.

[2526] Step 24:

[2527] The terminal notifies the user of the shared minutes.

[2528] Step 25:

[2529] The user receives the minutes from the terminal and prepares for the next action.

[2530] Example 2

[2531] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2532] In conventional meeting support systems, meeting progress and content organization are often done manually, which takes time and effort and makes it easy for important information to be overlooked. Furthermore, many systems lack functionality to appropriately reflect participants' emotions and reactions to support the progress of meetings. This reduces meeting efficiency and makes it difficult to ensure quality.

[2533] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2534] In this invention, the server includes means for inputting the purpose of the meeting, means for generating an agenda based on the input purpose of the meeting, means for recognizing voice, text, and image data in real time during the meeting to support the progress of the meeting, means for monitoring the progress and agenda of the meeting and supporting the appropriate progress, means for analyzing the emotional state of users during the meeting and encouraging them to respond, means for generating decisions and next actions at the end of the meeting, and means for automatically creating minutes and sharing them with the meeting participants. This automates the progress and content organization of the meeting, enables appropriate responses based on the emotional states of the participants, and improves the quality and efficiency of the meeting.

[2535] The "purpose of the meeting" refers to the specific goal or outcome that the user is trying to achieve through the meeting.

[2536] An "agenda" refers to a list of specific topics or items to be addressed in the course of a meeting.

[2537] "Speech recognition" refers to the technology that uses a computer to analyze speech spoken during a meeting and convert it into text data.

[2538] "Texting" refers to the process of converting audio or video data into written information.

[2539] "Image data" refers to visual information, including facial expressions and movements of participants, acquired during a meeting.

[2540] "Support for meeting progress" refers to providing various support functions to ensure that meetings proceed smoothly based on the agenda.

[2541] "Progress monitoring" refers to the process of checking in real time whether progress during a meeting is proceeding as planned.

[2542] "Emotional state" refers to changes in participants' psychological states and emotions during a meeting.

[2543] "Encouraging a response" refers to providing real-time guidance on appropriate actions or responses based on participants' emotional state.

[2544] "Decisions" refer to specific outcomes or courses of action agreed upon during a meeting.

[2545] "Next actions" refers to the specific actions or tasks that should be taken next based on the decisions made in the meeting.

[2546] "Minutes" refers to an official document that summarizes what was discussed, decisions made, and next actions taken at a meeting.

[2547] "Auto-generation" refers to the process by which the system automatically generates data without user intervention.

[2548] "Sharing" refers to providing the generated information and data to all meeting participants.

[2549] The "Meeting Supporter" system of the present invention provides a series of functions for smoothly conducting meetings. This system is implemented using a software program including a server, user terminals, a network, and an emotion engine.

[2550] Hardware and software used

[2551] 1. Server

[2552] The server uses a relational database (e.g., MySQL) to manage data such as meeting goals, agendas, and statements made during meetings.

[2553] Use natural language processing algorithms (e.g., GPT-3) to automatically generate an agenda from the input meeting goals.

[2554] Use speech recognition technology (e.g., Google Cloud Speech-to-Text) to convert what is said during the meeting into text in real time.

[2555] Using an emotion engine (e.g., Microsoft Azure Emotion API), it recognizes the user's emotional state in real time during a meeting and prompts them to respond.

[2556] 2. Terminal

[2557] The user terminal functions as an interface for displaying real-time information during the meeting and receiving notifications from the emotion engine.

[2558] Using a web or mobile application, users can enter meeting goals, review and revise agendas and ideas.

[2559] Processing flow

[2560] 1. Setting meeting goals

[2561] The user logs in to the application using a terminal and enters the goal of the meeting in a text field. For example, if the user enters "Goal of the new product planning meeting: Finalize specifications," the server receives this and stores it in the database.

[2562] 2. Creating a projected agenda

[2563] The server generates an agenda using a natural language processing algorithm (e.g., GPT-3) based on the saved meeting goals. For example, it automatically generates topics such as "Consider new product specifications," "Share market research results," and "Confirm design proposals." The generated agenda is sent to the device, where the user can review it and make any necessary changes.

[2564] 3. Idea submission

[2565] The server provides ideas related to the meeting goals based on information collected from databases and the Internet. For example, specific ideas such as "a list of features required for a new product" or "improvements to an existing product" can be presented to the terminal, and the user can reflect these ideas in the meeting content.

[2566] 4. Support during meetings

[2567] The server uses speech recognition technology to convert the user's speech into text in real time, which the device displays to the user, for example, notifying them that "This topic has been completed. We will move on to the next topic." The server also monitors the progress and agenda of the meeting to ensure that it is proceeding properly.

[2568] 5. Emotion recognition and response using an emotion engine

[2569] During the conference, the server uses an emotion engine to analyze data such as the user's voice, facial expression, and text to recognize their emotional state. For example, if the user is dissatisfied, the terminal will display an instruction such as "The speaker seems dissatisfied. Please provide additional explanation." The user can then respond accordingly.

[2570] 6. Actions at the end of the meeting

[2571] The server automatically generates "decisions" and "next actions" based on what was said during the meeting. For example, it lists things like "Add feature A: Person in charge: B, deadline: next Friday." These decisions are presented to the user via their terminal, who can then confirm and make any necessary corrections.

[2572] 7. Automatically create and share meeting minutes

[2573] After the meeting ends, the server automatically creates minutes including decisions and next actions, and shares them with the meeting participants via their devices. The devices then notify the users that the minutes have been shared, and the users can confirm this.

[2574] Specific examples

[2575] A specific example of how a new product planning meeting might proceed is shown below.

[2576] 1. Setting meeting goals

[2577] Enter "Goal of new product planning meeting: Finalize specifications." The server saves this.

[2578] 2. Creating a projected agenda

[2579] "Consider new product specifications," "Share market research results," and "Confirm design proposals" are automatically generated, and the user can add "Confirm manufacturing schedule."

[2580] 3. Idea submission

[2581] Related ideas are provided and users prepare meeting content based on them.

[2582] 4. Support during meetings

[2583] The device converts comments into text in real time and displays them, and users follow instructions on the device to conduct the meeting.

[2584] 5. Emotion recognition and response using an emotion engine

[2585] The emotion engine recognizes the user's emotional state and the device displays appropriate instructions. For example, if the user appears dissatisfied, the device will notify them that "additional explanation is required."

[2586] 6. Actions at the end of the meeting

[2587] Generates decisions and next actions and presents them to the user, who can then confirm and modify them.

[2588] 7. Automatically create and share meeting minutes

[2589] Minutes are automatically created and shared with participants. The device notifies the user, who can then check the minutes.

[2590] Examples of prompt statements

[2591] Example of an input prompt for a generative AI model:

[2592] The goal of the new product planning meeting is to "finalize specifications." Based on this goal, generate the next expected agenda.

[2593] If a user expresses dissatisfaction or voices dissatisfaction during a meeting, analyze the situation and suggest appropriate ways to respond.

[2594] This enables the system to generate and present appropriate agendas and emotional responses.

[2595] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2596] Step 1: Set meeting goals

[2597] Input: A user uses a terminal to log into the application and enter the meeting goal in a text field.

[2598] Specific action: The user inputs "Goal of new product planning meeting: Finalize specifications."

[2599] Data processing: The server receives the entered data and stores it in a database.

[2600] Output: Saved meeting goal data.

[2601] Step 2: Create a visionary agenda

[2602] Input: The server retrieves the saved meeting goals from the database.

[2603] Specific operation: The server uses a natural language processing algorithm (e.g., GPT-3) to generate an agenda from the meeting goals.

[2604] Data processing: Analyze meeting goal data and generate relevant agenda items.

[2605] Output: A list of agenda items such as "Consider new product specifications," "Share market research results," and "Confirm design proposals."

[2606] Step 3: Review and revise the agenda

[2607] Input: The server generates an agenda and sends it to the user's device.

[2608] Specific operation: The user checks the agenda through the terminal and makes corrections such as adding "Confirm production schedule."

[2609] Data processing: The user's modifications are sent to the server and the database is updated.

[2610] Output: The final agenda list.

[2611] Step 4: Submit your idea

[2612] Input: The server retrieves the final agen...

Claims

1. a means of inputting the meeting goals; means for generating a hypothetical agenda based on input meeting goals; A means to recognize voice, text, images, and videos in real time during a meeting to support the progress of the meeting; A means to clarify decisions and next actions at the end of the meeting; A system that includes a means for automatically creating meeting minutes and automatically sharing them with meeting participants.

2. The system of claim 1 , wherein the system provides ideas related to the input meeting goal and the generated potential agenda.

3. 2. The system according to claim 1, wherein the contents of user comments are converted into text during the conference and important comments and decisions are displayed in real time.

Images