System

Abstract

A system is provided.SOLUTION: A system comprising: means for a user to sign up and log in; means for the user to schedule a business-based web meeting; means for a support person to participate in the web meeting and provide optimal prompts based on the business context of the user; and means for storing a record of the meeting in a server.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Modern companies are trying to use generative AI to improve business efficiency, but there are only a limited number of personnel with the skills to understand business flows and the structure of generative AI and input optimal prompts. As a result, many companies are unable to fully utilize the potential of generative AI, which does not lead to improved business efficiency. In addition, consulting services for utilizing generative AI are expensive, which is a significant cost burden, especially for small and medium-sized enterprises. To solve these issues, a system that provides optimal prompt suggestions based on business context is needed. [Means for solving the problem]

[0005] The present invention provides a system that includes a means for users to sign up and log in, a means for users to book web meetings based on their tasks, a means for support staff to participate in the web meetings and provide optimal prompts based on the user's work background, and a means for saving meeting records on a server. The system also includes a means for users to share their screens to explain the work background to support staff in real time, and a means for saving meeting logs, chat history, shared files, and generated prompt ideas in a database. This allows users to easily improve work efficiency by utilizing AI generation, eliminating the need to use expensive consulting services.

[0006] "User" refers to an end user who intends to use this system to improve business efficiency.

[0007] "Sign up" refers to the process by which a user creates an account before starting to use the system.

[0008] "Login" refers to the process by which a User accesses a System using existing account information.

[0009] "Web meeting" refers to an online conference held over the Internet.

[0010] "Reservation" refers to the act of a user scheduling a web meeting for a specific date and time.

[0011] "Support staff" refers to specialized staff who use the system to assist users in improving the business efficiency they desire.

[0012] A "prompt" refers to an instruction or question entered into a generative AI to obtain an appropriate output.

[0013] "Screen sharing" refers to the ability for users to display their device screen to other participants in a web meeting in real time.

[0014] "Log" refers to a record of conversations and operations that took place during a web meeting.

[0015] "Chat History" means a record of messages sent and received between participants during a Web meeting.

[0016] "Shared files" refers to documents and materials shared among participants during a web meeting.

[0017] A "database" refers to a structured collection of data that allows information to be efficiently stored within a system and retrieved as needed.

[0018] "Server" refers to a computer that performs central processing in the system and exchanges data with the terminals of users and support staff. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0020] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0021] First, the terms used in the following description will be explained.

[0022] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0023] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0024] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0025] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0026] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0027] [First embodiment]

[0028] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0029] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0030] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0031] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0032] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0033] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0034] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0035] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0036] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0037] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0038] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0039] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0040] This invention is a system that provides support to users to improve work efficiency by utilizing generative AI. The system involves a series of processes in which users sign up and log in, book a web meeting, and receive optimal prompt suggestions through screen sharing with a support staff member.

[0041] Program processing overview

[0042] 1. Sign up and login

[0043] Device: The user accesses the web app and creates an account (signs up) or logs in with an existing account.

[0044] Server: Stores the information entered by the user in a database, performs authentication, and sends a confirmation email to the user once sign-up is complete.

[0045] 2. Schedule a web meeting

[0046] Terminal: Users can see available slots on their dashboard and select the desired date and time for the meeting.

[0047] Server: Saves reservation information in a database and notifies support staff.

[0048] 3. Participating in a web meeting

[0049] Terminal: When the meeting start time arrives, the user and support staff click on the provided link to join the web meeting.

[0050] Server: Sends reminders and generates and provides meeting links to participants.

[0051] 4. Explanation of the user's business background

[0052] Terminal: The user can use the screen sharing function to display the current status of the work to the support staff, provide verbal explanations, and share relevant materials and documents.

[0053] Support staff's terminal: The support staff understands the current situation based on the user's explanation and takes notes.

[0054] 5. Providing optimal prompts

[0055] Support staff device: Based on the user's work background, the system creates prompt ideas suitable for the generation AI and proposes them to the user.

[0056] User's device: The user receives the suggestions and asks questions or requests corrections as needed.

[0057] 6. Meeting Recording

[0058] Server: Stores meeting logs, chat history, shared files, and generated prompts in a database.

[0059] Device: Users can review recordings after the meeting and download them if necessary.

[0060] Specific examples

[0061] 1. User signs up and logs in

[0062] On the device: The user opens the web app, clicks the "Sign Up" button, enters their username, email address, and password, and clicks "Register."

[0063] Server: Stores the entered information in a database and sends a confirmation email to the user. When the user clicks on the link in the email, the login page is displayed.

[0064] 2. A user books a web meeting

[0065] On the device: The user goes to the "Book a Meeting" section of their dashboard, selects the desired date and time from the calendar, clicks the "Book" button and receives a confirmation message.

[0066] Server: Saves reservation information in a database and sends reservation notifications to support staff.

[0067] 3. The user and the support staff join a web meeting

[0068] On your device: The user opens the web app at the designated time and clicks the "Join Meeting" button. The support staff also joins.

[0069] Server: Generates secure web meeting links and sends reminder notifications.

[0070] 4. User explains the business background

[0071] Device: The user shares the documents and screens they use in their work and explains the specific work background and challenges to the support staff.

[0072] Support staff's device: Understand the job content and listen to the user's explanation while taking notes.

[0073] 5. Providing optimal prompts

[0074] Support staff terminal: Based on the user's explanation, it devises a prompt suitable for the generative AI and presents it to the user.

[0075] User's device: Accept suggestions and make any necessary corrections or questions.

[0076] 6. Meeting Recording

[0077] Server: After the meeting ends, the logs, chat history, shared files, and generated prompts are saved in a database.

[0078] Terminal: User can access the recording later and download it as needed.

[0079] In this way, the system helps users effectively utilize generative AI to improve business efficiency.

[0080] The processing flow will be explained below.

[0081] Sign up and log in

[0082] Step 1: View the sign-up page

[0083] Device: User visits web app and clicks "Sign up" button.

[0084] Server: Sends the sign-up page to the browser and displays it.

[0085] Step 2: Enter your sign-up information

[0086] On the device: The user fills in a form with their username, email address, password, and confirmation password.

[0087] Step 3: Submit your sign-up information

[0088] Terminal: The user clicks the "Register" button and sends the information to the server.

[0089] Step 4: Verify and save your sign-up information

[0090] Server: Checks the entered information for consistency and duplication, and if there are no problems, saves it to the database.

[0091] Server: Sends a confirmation email to the user.

[0092] Step 5: Display the login page

[0093] Server: After successful sign-up, display the login page to the user.

[0094] Step 6: Enter your login details

[0095] Device: The user enters their email address and password and clicks the "Login" button.

[0096] Step 7: Verify your login details

[0097] Server: The entered email address and password are verified against the database.

[0098] Step 8: View the Dashboard

[0099] Server: If authentication is successful, display the user's dashboard page.

[0100] Web Meeting Reservation

[0101] Step 1: View the booking page

[0102] On your device: The user accesses the "Book a Meeting" section of their dashboard.

[0103] Server: Sends data to display the reservation page.

[0104] Step 2: Select a reservation slot

[0105] Terminal: The user selects the desired date and time from the calendar.

[0106] Step 3: Confirm and submit your booking

[0107] Terminal: The user clicks the "Confirm Reservation" button and sends the information to the server.

[0108] Step 4: Save your reservation

[0109] Server: Saves the entered reservation information in a database and sends a reservation notification to the support staff.

[0110] Step 5: View the reservation confirmation message

[0111] Server: Display a confirmation message to the user that the reservation was successful.

[0112] Joining a web meeting

[0113] Step 1: Meeting start notification

[0114] Server: Send a reminder to users and support staff 10 minutes before the meeting start time.

[0115] Step 2: Generate a meetings link

[0116] Server: Generates and sends secure web meeting links to users and support personnel.

[0117] Step 3: Join the meeting

[0118] Terminal: The user and support staff click on the link sent to join the web meeting.

[0119] Step 4: Start Screen Sharing

[0120] Terminal: The user uses the screen sharing function to show the current work status to the support staff.

[0121] User's business background

[0122] Step 1: Oral explanation of the business background

[0123] Terminal: The user verbally explains the specific background and issues of the work to the support staff.

[0124] Step 2: Share materials

[0125] Terminal: The user provides relevant materials and documents to the support staff via screen sharing or file transfer.

[0126] Providing optimal prompts

[0127] Step 1: Create a Prompt

[0128] Support staff terminal: Generates optimal prompt suggestions based on the user's explanation.

[0129] Step 2: Propose a Prompt

[0130] Terminal: The support staff presents the proposed prompt to the user and explains it to them.

[0131] Step 3: Receive feedback and make corrections

[0132] On the device: The user provides feedback on the proposed prompt and requests revisions if necessary.

[0133] Support staff device: Receive feedback and revise the proposed prompt.

[0134] Meeting recording

[0135] Step 1: Save the meeting recording

[0136] Server: Stores meeting logs (chat history, shared files, generated prompts) in a database.

[0137] Step 2: Notification and Access to Records

[0138] Server: After the meeting ends, notify the user that the recording has been saved and provide an access link.

[0139] Step 3: Review and download the recording

[0140] On your device: Users can click the link provided to log in, view the meeting recording, and download it if necessary.

[0141] In this way, specific processing is carried out at each step, and the system provides assistance to users to make the most of generated AI and improve business efficiency.

[0142] Example 1

[0143] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0144] In today's work environment, users need appropriate support to efficiently utilize generative AI to perform their work. However, if users lack the specific background information or know-how to devise prompts, it can be difficult to maximize the effectiveness of generative AI. Furthermore, the process of sharing the status and progress of work with support staff in real time and receiving specific advice must be streamlined. Furthermore, it is also important to properly store meeting records so they can be referenced later.

[0145] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0146] In this invention, the server includes a means for a user to create and authenticate an account, a means for the user to select a date and time based on the user's work and schedule an online conference, and a means for a support staff member to participate in the online conference and generate optimal prompts based on the work materials shared by the user. This allows the user to receive advice from the appropriate support staff member and efficiently utilize the generation AI, significantly improving work efficiency. Furthermore, by properly storing online conference records and dialogue history, users can enjoy the benefit of making them easier to reference later.

[0147] "Creating an account and authenticating" refers to the process in which a user first registers (signs up) with the system and the system verifies the user's authenticity based on the information entered.

[0148] "Selecting a date and time to reserve an online meeting" refers to the process by which a user selects a specific date and time using the calendar function within the system and reserves the system to hold an online meeting at that date and time.

[0149] "Participating in an online meeting" refers to the process in which a user and a support staff member connect online using a web link generated at a specified date and time to hold a meeting.

[0150] "Generating optimal prompt sentences based on business materials" refers to the process in which a support staff member creates and provides effective prompts to the generation AI based on the business-related materials and information shared with the support staff by the user.

[0151] "Storing on a network server" refers to the process of securely storing online meeting records, conversation histories, shared data, etc. on a remote data storage system.

[0152] "Explaining the work background and sharing information on the screen in real time" refers to the process in which a user explains the details and background of the work while sharing the screen of their computer or device with a support staff member in real time during an online meeting.

[0153] "Storing online meeting logs, dialogue history, shared data, and generated prompts in a management system" refers to the process of storing all records and data related to online meetings in a dedicated management system for future reference.

[0154] This invention is a system designed to allow users to improve work efficiency by utilizing generated AI. Specific embodiments of the invention are described below.

[0155] overview

[0156] This system helps users create an account through a web application, schedule a business meeting, and participate in an online meeting, and then receive appropriate prompt advice from support staff, thereby improving work efficiency.

[0157] Hardware and Software Configuration

[0158] server

[0159] The server is a backend system for user authentication, conference booking, online conference link generation, and meeting record storage. Specifically, it has the following configuration:

[0160] Database: Use a database (e.g. PostgreSQL) to store user information, booking information, and meeting records.

[0161] Application server: Uses a backend framework (e.g., Node.js, Django) that provides user authentication and conference management functions.

[0162] Mail Server: Use the mail server to send account confirmation emails and meeting reminder notifications.

[0163] Terminal

[0164] The terminal is where users and support staff access and operate the web application. It consists of:

[0165] Browser: The user uses a web browser (e.g., GOOGLE CHROME®, Firefox) to access the web application.

[0166] Screen sharing software: Use software (e.g., Zoom, MICROSOFT® TEAMS®) that allows users to share their screens with support personnel in real time.

[0167] Explanation of program processing

[0168] Sign up and log in

[0169] Device: The user accesses the web application using a browser, clicks the "Sign Up" button, enters their username, email address, and password, and clicks "Register." The server stores the entered information in a database and sends a confirmation email to the user. The user clicks the link in the email and their account is activated. The user then enters their email address and password on the login page and clicks "Login." The server performs authentication, and if successful, displays the dashboard to the user.

[0170] Examples:

[0171] A user accesses the web app in a browser, enters "testuser", "testuser@example.com", and "password123", and clicks the register button. The server stores this information in a database and sends a confirmation email to "testuser@example.com".

[0172] Web Meeting Reservation

[0173] Terminal: The user selects the desired date and time from the calendar in the dashboard and clicks the "Reserve" button. The server saves the reservation information in the database and sends a reservation notification to the support staff.

[0174] Examples:

[0175] The user selects "October 15, 2023, 3:00 PM" on the dashboard calendar and clicks the "Book" button. The server saves this information in the database and notifies the support staff by email.

[0176] Joining a web meeting

[0177] Terminal: When the meeting start time arrives, users and support staff click the provided link to join the web meeting. The server sends reminder notifications and generates and provides a secure web meeting link to participants.

[0178] Examples:

[0179] Five minutes before the meeting starts, the server sends a reminder to the user and support staff and provides them with a meeting link, which they click to join.

[0180] User's business background

[0181] Device: The user uses the screen sharing function to share the work situation and related documents with the support staff. The support staff understands the work situation based on the user's explanation and takes notes.

[0182] Examples:

[0183] The user clicks the screen sharing button in the browser and displays the Excel file used in work to the support staff.

[0184] Providing optimal prompts

[0185] Support staff terminal: The support staff will devise a prompt suitable for the generation AI based on the user's work background. The user will review the proposed prompt and make any necessary corrections or questions.

[0186] Specific prompt examples:

[0187] Suggest business prompts such as "Automatically generate invoices."

[0188] Meeting recording

[0189] Server: After the meeting, the logs, chat history, shared files, and generated prompts are stored in a database. Users can access the recordings later and download them as needed.

[0190] Examples:

[0191] The server stores meeting records, interaction history, and shared files in a database, which users can download by accessing the history section of their dashboard.

[0192] As described above, by using this system, users can effectively utilize generative AI to optimize their business processes. With the help of support staff, they can quickly create effective prompts, significantly improving work efficiency.

[0193] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0194] Step 1:

[0195] Create an account (sign up)

[0196] Device: The user visits the web application and enters their username, email address, and password into the sign-up form.

[0197] Specific behavior:

[0198] The user enters "testuser", "testuser@example.com", and "password123" and clicks the Register button.

[0199] Server: The server stores the entered username, email address, and password in a database and sends a confirmation email to the user. Based on the user's input, the server stores the information in a database and sends a confirmation email.

[0200] Input: Username, Email Address, Password

[0201] Data processing: Save user information in a database and request the mail server to send a confirmation email

[0202] Output: User information stored in the database, confirmation email sent to the user

[0203] Step 2:

[0204] Account authentication (login)

[0205] Device: After signing up, the user enters their email address and password on the login page and clicks "Login."

[0206] Specific behavior:

[0207] The user enters "testuser@example.com" and "password123" and clicks the login button.

[0208] Server: The server checks the input information against the database and performs authentication. If successful, it displays the dashboard to the user.

[0209] Input: Email address, Password

[0210] Data processing: Retrieving user information from the database and matching it with the input information

[0211] Output: Authentication result. If successful, the user's dashboard screen will be displayed.

[0212] Step 3:

[0213] Web Meeting Reservation

[0214] Terminal: The user selects the desired date and time using the calendar function in the dashboard and clicks the "Book" button.

[0215] Specific behavior:

[0216] The user selects "October 15, 2023, 3:00 PM" from the calendar and clicks the reservation button.

[0217] Server: The server saves the reservation information in a database and sends a reservation notification to the support staff. It saves the information to the database and notifies the staff based on the user's input.

[0218] Input: Reservation date and time

[0219] Data processing: Save the reservation information to the database and call the notification system to send reservation notifications to support staff.

[0220] Output: Booking information stored in the database, booking notification sent to support staff

[0221] Step 4:

[0222] Joining a web meeting

[0223] Server: Sends reminders to users and support staff before the meeting starts and generates a secure web meeting link.

[0224] Input: Reservation information

[0225] Data processing: generating and sending reminder notifications, generating web meeting links

[0226] Output: Reminder sent, generated web meeting link

[0227] Terminal: Users and support staff click on the provided link to join the web meeting.

[0228] Specific behavior:

[0229] Five minutes before the meeting starts, the user and support staff will check the notification and click the provided link to join.

[0230] Step 5:

[0231] Business background explanation

[0232] Device: Users can use the screen sharing function to share work status and related documents with support staff.

[0233] Specific behavior:

[0234] The user clicks the screen sharing button in the browser and displays an Excel file used for work.

[0235] Support staff's device: The support staff understands the work situation based on the information shared on the screen and the user's explanation, and takes notes.

[0236] Input: Screen sharing information, user's verbal description

[0237] Data processing: Support staff understanding and recording

[0238] Output: Support personnel notes

[0239] Step 6:

[0240] Providing optimal prompts

[0241] Support staff terminal: The support staff will devise a prompt suitable for the generation AI based on the user's work background and present it to the user.

[0242] Input: User's business background information

[0243] Data processing: Prompt generation

[0244] Output: Suggested prompt text

[0245] Example prompt:

[0246] Suggest business prompts such as "Automatically generate invoices."

[0247] User's device: The user reviews the suggested prompt and makes any necessary corrections or questions.

[0248] Specific behavior:

[0249] The user reviews the prompts from the support staff and, if necessary, requests to "change the date format."

[0250] Step 7:

[0251] Meeting recording

[0252] Server: After the meeting ends, the log, chat history, shared files, and generated prompts are saved in a database.

[0253] Input: Meeting logs, chat history, shared files, generated prompts

[0254] Data processing: Record preservation processing

[0255] Output: Records stored in a database

[0256] Terminal: User can access the recording later and download it if needed.

[0257] Specific behavior:

[0258] After a meeting ends, users can access the History section of their dashboard and download a specific meeting recording.

[0259] (Application example 1)

[0260] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0261] To improve efficiency and productivity in food delivery operations, there is a need for technology that allows users to collaborate with support staff in real time and provides optimal routes using generative AI models. However, conventional systems have issues with being unable to efficiently perform a series of tasks, such as web meetings, recording work, and providing optimal routes.

[0262] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0263] In this invention, the server includes a means for users to sign up and log in, a means for users to book web meetings based on their tasks, a means for support staff to participate in the web meetings and provide optimal prompts based on the user's work background, a means for saving meeting records on the server, a means for users to manage delivery task reservations, a means for providing optimal route suggestions using a generative AI model, and a means for inputting feedback after the task is completed. This significantly improves the efficiency and productivity of food delivery tasks, and enables users to effectively use generative AI models to optimize their tasks.

[0264] "User" means a person who uses the system to improve business efficiency.

[0265] "Signing up" is the process of creating a new account.

[0266] "Login" means the act of authenticating to access an existing account and use the system.

[0267] A "web meeting" is an online conference held over the Internet.

[0268] "Reservation" is the act of setting up a web meeting for a specific date and time.

[0269] A "support staff member" is someone whose role is to support users in optimizing their operations.

[0270] A "prompt" is an input sentence that gives instructions to a generative AI model.

[0271] "Recording" refers to saving the progress and content of a meeting in a format that can be referenced later.

[0272] A "server" is a computer system that stores and processes data.

[0273] "Delivery services" refers to the general business of delivering goods or services to designated locations.

[0274] A "generative AI model" is an artificial intelligence algorithm that makes inferences and predictions based on data.

[0275] A "route proposal" is a plan that suggests an efficient route for delivery work.

[0276] "Feedback" is the act of recording evaluations and opinions after a task is completed.

[0277] This invention is a system that uses generative AI models to help improve the efficiency of food delivery operations. The system involves a series of processes in which users sign up and log in, book web meetings, and share their work status with support staff in real time.

[0278] 1. Sign up and login

[0279] The server stores the user's input information in a database and performs authentication. For example, when signing up, the user enters their username, email address, and password, and after receiving a verification email, clicks the link to log in.

[0280] 2. Delivery business reservation management

[0281] The user's device is used to manage delivery schedules. The server uses the Google (registered trademark) Calendar API or similar to store reservation information in a cloud database and notifies support staff.

[0282] 3. Participating in a web meeting

[0283] Using WebRTC, the user and support staff join a web meeting at the specified time. The server sends a reminder notification and generates a secure meeting link.

[0284] 4. Explanation of business background and screen sharing

[0285] Users can use the screen sharing function to show support staff the documents and progress they are using in their work, making it easier for support staff to understand the user's work background.

[0286] 5. Providing optimal route suggestions

[0287] The support staff terminal uses a generative AI model based on the user's work background to provide optimal route suggestions. For example, the generative AI can suggest optimal routes using prompts such as: "Please suggest the optimal route based on the delivery address and customer address."

[0288] 6. Feedback after the job is completed

[0289] After completing a delivery task, users can enter feedback using their devices. This feedback is stored on the server and can be used to improve future operations.

[0290] 7. Meeting Recording

[0291] The server stores meeting logs, chat history, shared files, and generated prompts in a database such as Firebase Firestore, allowing for future review and continuous improvement.

[0292] For example, a user who wants to streamline delivery operations during lunchtime can enter the following: "We have a high volume of deliveries during lunchtime. Please suggest the best route and most efficient delivery method."

[0293] Generative AI analyzes data to create optimal route plans in real time and notify users' smartphones, thereby significantly improving the efficiency of food delivery operations.

[0294] The system can be realized using specific hardware and software such as Firebase Authentication, Google Calendar API, WebRTC, OpenAI® GPT-3®, and Firebase Firestore.

[0295] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0296] Step 1:

[0297] A user signs up.

[0298] Input: Username, Email Address, Password

[0299] Data processing: The terminal collects the user's input information and sends it to the server.

[0300] Output: The server stores the information in a database and sends a verification email.

[0301] How it works: A user opens the web app, enters the required information, and clicks "Register." The server receives the information, writes it to the database, and then sends a verification email.

[0302] Step 2:

[0303] The user logs in.

[0304] Input: Email address, password

[0305] Data processing: The server validates and authenticates the input information.

[0306] Output: Login success message and transition to the dashboard screen

[0307] What happens: The user enters their email address and password on the login page and clicks "Login." The server verifies the information and, if correct, displays the dashboard screen.

[0308] Step 3:

[0309] A user books a delivery job.

[0310] Input: Scheduled delivery date and time, detailed information

[0311] Data processing: The server saves the reservation information and updates the calendar using the Google Calendar API.

[0312] Output: Reservation confirmation and notification to support staff

[0313] What it does: The user selects the desired date and time on the dashboard and clicks "Book." The server saves the reservation information, updates the calendar using the Google Calendar API, and notifies the support staff.

[0314] Step 4:

[0315] A user joins a web meeting.

[0316] Input: Meeting time, link

[0317] Data processing: The server sends reminder notifications and generates secure meeting links over WebRTC.

[0318] Output: Reminders, Meeting Links

[0319] Specific operation: When the meeting start time arrives, the user and support staff click the designated link to join the web meeting. The server sends a notification and generates and distributes the link.

[0320] Step 5:

[0321] The user explains the background of the work to the support staff and shares the screen.

[0322] Input: Business documents, progress status

[0323] Data processing: The device shares the screen and communicates the work status to the support staff.

[0324] Output: Shared work documents, progress explanation

[0325] Specific operation: The user uses the screen sharing function to show the support staff the materials and progress, and the support staff deepens their understanding and takes notes.

[0326] Step 6:

[0327] Support staff use generative AI models to provide optimal route suggestions.

[0328] Input: User's business background information, delivery route

[0329] Data processing: A generative AI model calculates and suggests efficient routes.

[0330] Output: Optimal route plan

[0331] Specific operation: The support staff inputs prompts into the AI ​​generator based on the information obtained from the user and suggests the optimal route. Example prompt: "Please suggest the optimal route based on the delivery address and customer address."

[0332] Step 7:

[0333] The user enters feedback after completing the delivery job.

[0334] Input: Feedback

[0335] Data processing: The server stores the feedback information.

[0336] Output: Saved feedback data

[0337] What it does: A user enters and submits feedback using the web app. The server stores the information in a database.

[0338] Step 8:

[0339] The server stores the meeting records.

[0340] Input: Meeting logs, chat history, shared files, generated prompt ideas

[0341] Data processing: The server stores the data in a database.

[0342] Output: Saved meeting data

[0343] Specific operation: After the meeting ends, the server saves the logs, chat history, shared files, and generated prompt ideas in a database such as Firebase Firestore.

[0344] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0345] This invention is a system that provides support to users to improve work efficiency using generative AI, and aims to provide more effective support by combining it with an emotion engine that recognizes and responds to user emotions. The system involves a series of processes in which users sign up and log in, book a web meeting, explain the work background through screen sharing with a support staff member, and receive optimal prompt suggestions. In addition, the emotion engine collects user emotion data, adjusts prompt suggestions based on that data, and stores records.

[0346] Program processing overview

[0347] 1. Sign up and login

[0348] Device: The user accesses the web app and creates an account (signs up) or logs in with an existing account.

[0349] Server: Stores the information entered by the user in a database, performs authentication, and sends a confirmation email to the user once sign-up is complete.

[0350] 2. Schedule a web meeting

[0351] Terminal: Users can see available slots on their dashboard and select the desired date and time for the meeting.

[0352] Server: Saves reservation information in a database and notifies support staff.

[0353] 3. Participating in a web meeting

[0354] Terminal: When the meeting start time arrives, the user and support staff click on the provided link to join the web meeting.

[0355] Server: Sends reminders and generates and provides meeting links to participants.

[0356] 4. Explanation of the user's business background

[0357] Terminal: The user can use the screen sharing function to display the current status of the work to the support staff, provide verbal explanations, and share relevant materials and documents.

[0358] Support staff's terminal: The support staff understands the current situation based on the user's explanation and takes notes.

[0359] 5. Operation of the Emotion Engine

[0360] Device: Uses a camera and microphone to analyze the user's facial expressions and voice, and collects emotional data from them.

[0361] Server: The emotion engine processes the collected data in real time and recognizes the user's emotions.

[0362] 6. Providing optimal prompts

[0363] Support staff device: Based on the user's work background and emotional data, the device creates prompt ideas suitable for the generating AI and proposes them to the user.

[0364] User's device: The user receives the suggestions and asks questions or requests corrections as needed.

[0365] 7. Meeting Recording

[0366] Server: Stores meeting logs, chat history, shared files, sentiment data, and generated prompt ideas in a database.

[0367] Device: Users can review recordings after the meeting and download them if necessary.

[0368] Specific examples

[0369] 1. User signs up and logs in

[0370] On the device: The user opens the web app, clicks the "Sign Up" button, enters their username, email address, and password, and clicks "Register."

[0371] Server: Stores the entered information in a database and sends a confirmation email to the user. When the user clicks on the link in the email, the login page is displayed.

[0372] 2. A user books a web meeting

[0373] On the device: The user goes to the "Book a Meeting" section of their dashboard, selects the desired date and time from the calendar, clicks the "Book" button and receives a confirmation message.

[0374] Server: Saves reservation information in a database and sends reservation notifications to support staff.

[0375] 3. The user and the support staff join a web meeting

[0376] On your device: The user opens the web app at the designated time and clicks the "Join Meeting" button. The support staff also joins.

[0377] Server: Generates secure web meeting links and sends reminder notifications.

[0378] 4. User explains the business background

[0379] Device: The user shares the documents and screens they use in their work and explains the specific work background and challenges to the support staff.

[0380] Support staff's device: Understand the job content and listen to the user's explanation while taking notes.

[0381] 5. Operation of the Emotion Engine

[0382] Device: Captures the user's face with a camera, picks up their voice with a microphone, and collects emotional data in real time.

[0383] Server: Analyzes the collected data and determines the user's emotional state.

[0384] 6. Providing optimal prompts

[0385] Support staff device: Taking into account the user's work background and emotional state, the system creates prompt proposals suitable for the generative AI and presents them to the user.

[0386] User's device: The user accepts the suggestions and makes any necessary corrections or questions.

[0387] 7. Meeting Recording

[0388] Server: After the meeting ends, the logs, chat history, shared files, emotion data, and generated prompts are stored in a database.

[0389] On-device: Users can review saved records later and download them as needed.

[0390] In this way, the system helps users maximize the use of generative AI and emotion recognition to improve work efficiency.

[0391] The processing flow will be explained below.

[0392] Sign up and log in

[0393] Step 1: View the sign-up page

[0394] Device: User visits web app and clicks "Sign up" button.

[0395] Server: Sends the sign-up page to the browser and displays it.

[0396] Step 2: Enter your sign-up information

[0397] On the device: The user fills in a form with their username, email address, password, and confirmation password.

[0398] Step 3: Submit your sign-up information

[0399] Terminal: The user clicks the "Register" button and sends the information to the server.

[0400] Step 4: Verify and save your sign-up information

[0401] Server: Checks the entered information for consistency and duplication, and if there are no problems, saves it to the database.

[0402] Server: Sends a confirmation email to the user.

[0403] Step 5: Display the login page

[0404] Server: After successful sign-up, display the login page to the user.

[0405] Step 6: Enter your login details

[0406] Device: The user enters their email address and password and clicks the "Login" button.

[0407] Step 7: Verify your login details

[0408] Server: The entered email address and password are verified against the database.

[0409] Step 8: View the Dashboard

[0410] Server: If authentication is successful, display the user's dashboard page.

[0411] Web Meeting Reservation

[0412] Step 1: View the booking page

[0413] On your device: The user accesses the "Book a Meeting" section of their dashboard.

[0414] Server: Sends data to display the reservation page.

[0415] Step 2: Select a reservation slot

[0416] Terminal: The user selects the desired date and time from the calendar.

[0417] Step 3: Confirm and submit your booking

[0418] Terminal: The user clicks the "Confirm Reservation" button and sends the information to the server.

[0419] Step 4: Save your reservation

[0420] Server: Saves the entered reservation information in a database and sends a reservation notification to the support staff.

[0421] Step 5: View the reservation confirmation message

[0422] Server: Display a confirmation message to the user that the reservation was successful.

[0423] Joining a web meeting

[0424] Step 1: Meeting start notification

[0425] Server: Send a reminder to users and support staff 10 minutes before the meeting start time.

[0426] Step 2: Generate a meetings link

[0427] Server: Generates and sends secure web meeting links to users and support personnel.

[0428] Step 3: Join the meeting

[0429] Terminal: The user and support staff click on the link sent to join the web meeting.

[0430] Step 4: Start Screen Sharing

[0431] Terminal: The user uses the screen sharing function to show the current work status to the support staff.

[0432] User's business background

[0433] Step 1: Oral explanation of the business background

[0434] Terminal: The user verbally explains the specific background and issues of the work to the support staff.

[0435] Step 2: Share materials

[0436] Terminal: The user provides relevant materials and documents to the support staff via screen sharing or file transfer.

[0437] Emotion Engine Operation

[0438] Step 1: Collecting emotion data

[0439] Device: Emotional data is collected by taking a picture of the user's face with a camera and picking up their voice with a microphone.

[0440] Step 2: Analyze the emotion data

[0441] Server: Analyzes collected data in real time and determines the user's emotional state.

[0442] Providing optimal prompts

[0443] Step 1: Create a Prompt

[0444] Support staff device: Based on the user's work background and emotional data, the device creates a prompt suitable for the generating AI and presents it to the user.

[0445] Step 2: Propose a Prompt

[0446] Terminal: The support staff presents the proposed prompt to the user and explains it to them.

[0447] Step 3: Receive feedback and make corrections

[0448] On the device: The user provides feedback on the proposed prompt and requests revisions if necessary.

[0449] Support staff device: Receive feedback and revise the proposed prompt.

[0450] Meeting recording

[0451] Step 1: Save the meeting recording

[0452] Server: Stores meeting logs (conversation records, screen sharing content, emotion data, and generated prompts) in a database.

[0453] Step 2: Notification and Access to Records

[0454] Server: After the meeting ends, notify the user that the recording has been saved and provide an access link.

[0455] Step 3: Review and download the recording

[0456] On your device: Users can click the link provided to log in, view the meeting recording, and download it if necessary.

[0457] In this way, specific actions are performed at each processing step, and the system helps users maximize the use of generative AI and emotion recognition to improve work efficiency.

[0458] Example 2

[0459] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0460] Conventional business support systems have difficulty providing optimal instructions based on the user's work context, and do not take the user's emotions into consideration when providing support. As a result, there are problems with user satisfaction and work efficiency declining.

[0461] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0462] In this invention, the server includes a means for a user to sign up and log in, a means for a user to reserve an online meeting based on the user's work, a means for a support person to participate in the online meeting and provide optimal instructions based on the user's work background, a means for analyzing the user's facial expressions and voice and collecting emotional data, a means for processing the collected emotional data in real time and recognizing the user's emotions, and a means for saving a record of the meeting on the server. This makes it possible to provide appropriate instructions according to the user's work background and support that takes the user's emotions into consideration.

[0463] "Signing up" is the process by which a user creates a new account and registers with the system.

[0464] "Logging in" is the process of accessing a system using existing account information.

[0465] An "online meeting" is a conference held over the Internet, allowing participants to communicate in real time.

[0466] "Support Personnel" means personnel who provide technical and business support to Users.

[0467] "Instructions" are texts created using generative AI models that provide specific suggestions or guidelines to help users with their work.

[0468] "Facial Expression" refers to the muscle movements and other physical characteristics of a user's face that indicate an emotional state.

[0469] "Voice" refers to the voice produced by the user, and is data used to determine the user's emotional state using the tone, pitch, volume, etc.

[0470] "Emotional data" is information that expresses the user's emotional state as numerical or categorical data.

[0471] "Real-time processing" means processing data immediately after it is generated, providing results with little to no delay.

[0472] A "meeting recording" is a series of records containing digital data, such as conversations, chats, and shared files, that take place during an online meeting.

[0473] This invention is a support system that utilizes a generative AI model to help users improve their work efficiency, and also combines it with an emotion engine that recognizes and responds to user emotions.

[0474] The system includes the following means:

[0475] 1. Sign-up and login methods

[0476] Device: A user accesses a web app that uses a JavaScript framework to sign up and log in. Specifically, the user enters their username, email address, and password, and submits the sign-up form.

[0477] Server: Build a backend API using Node.js and Express, store this information in a MySQL database, and use an SMTP server to send confirmation emails, for example, sendgrid.

[0478] 2. Online Meeting Reservation Methods

[0479] On the device: Users access a dashboard powered by React.js and use a calendar component such as FullCalendar.js to book online meetings.

[0480] Server: Reservation information is stored in a MySQL database, and the server sends reservation confirmation emails to users and support staff.

[0481] 3. How to join an online meeting

[0482] Device: Users and support staff log in to the web app at the specified date and time and join through the API of online meeting tools such as GOOGLE MEET (registered trademark) and Zoom.

[0483] Server: Sends reminders near the scheduled time and generates a secure meeting link.

[0484] 4. Means of explaining the business background

[0485] Device: During online meetings, users can use Zoom's screen sharing function to explain the current work background and related materials to support staff.

[0486] Support staff's device: The support staff listens to the user's explanation and takes notes of important points using a note-taking app (e.g., OneNote) on their PC.

[0487] 5. Emotion Engine Operation Method

[0488] Device: The user's face is photographed with a webcam and their voice is collected with a microphone. A facial expression recognition model and voice analysis tool using TENSORFLOW(registered trademark).js are used.

[0489] Server: Uses a Python sentiment analysis library (e.g., emotion_recognition) to process the collected data in real time and recognize user emotions.

[0490] 6. Optimal Means of Providing Instructions

[0491] Support staff terminal: The support staff uses a generative AI model (e.g., Chat GPT) to generate instructions based on the user's work background and emotional data.

[0492] User's device: Provide the generated instructions to the user, correcting or asking questions as needed.

[0493] Examples of prompts:

[0494] "I want to use a generative AI model to generate optimal prompt suggestions. My user is preparing a presentation for a new marketing campaign and is particularly short on ideas for slide design. He's feeling stressed. What would be the best prompt?"

[0495] 7. Meeting Record Keeping Method

[0496] Server: After the online meeting ends, the logs, communication history, shared files, emotional data, and generated instructions are stored in a MySQL database.

[0497] On your device: Users can view saved recordings in the "Meeting History" section of the web app and download them if necessary.

[0498] In this way, the present invention provides specific assistance to users to make the most of generative AI models and emotion recognition to improve work efficiency.

[0499] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0500] Step 1:

[0501] Sign up and log in

[0502] On the device: The user opens the web app and clicks the "Sign up" button. The user enters their username, email address, and password and clicks "Register."

[0503] Enter your username, email address, and password.

[0504] Output: Signup request data.

[0505] Server: Receives the sign-up request data, stores the entered information in a MySQL database, and then sends a confirmation email using an SMTP server.

[0506] Input: Signup request data.

[0507] Output: Database saved results and confirmation email.

[0508] On the device: When the user clicks on the link in the confirmation email, they are authenticated and directed to a login page.

[0509] Enter: Click on the confirmation link.

[0510] Output: Login page displayed.

[0511] Step 2:

[0512] Schedule an online meeting

[0513] On the device: The user opens the dashboard, selects the desired date and time from the calendar, and clicks the "Book" button.

[0514] Input: Desired date and time.

[0515] Output: Booking request data.

[0516] Server: Receives the reservation request data, stores the reservation information in a MySQL database, and then sends a reservation confirmation email to the user and support staff.

[0517] Input: Booking request data.

[0518] Output: Database saved results and confirmation email.

[0519] Step 3:

[0520] Joining an online meeting

[0521] Device: The user and support staff open the web app at the designated time and click the "Join Meeting" button.

[0522] Input: Meeting time, Join button click.

[0523] Output: Meeting join request.

[0524] Server: Sends reminders before the specified time and generates a secure meeting link using Google Meet or Zoom APIs.

[0525] Input: Meeting join request.

[0526] Output: Reminder and meeting link.

[0527] Step 4:

[0528] Business background explanation

[0529] Terminal: The user uses the screen sharing function to show the support staff the current task and related materials, and provides verbal explanations.

[0530] Input: Start screen sharing, business documents.

[0531] Output: Screen sharing data.

[0532] Support staff's device: The support staff listens to the user's explanation and takes notes of important points using a note-taking app on their PC.

[0533] Input: Screen share data.

[0534] Output: Note data.

[0535] Step 5:

[0536] Emotion Engine Operation

[0537] Device: The user's face is photographed with a webcam and their voice is collected with a microphone. TensorFlow.js is used to capture facial expression and voice data in real time.

[0538] Input: Video and audio data.

[0539] Output: facial expression data, voice data.

[0540] Server: Sends the collected data to a Python sentiment analysis library to recognize the user's emotional state.

[0541] Input: facial expression data, voice data.

[0542] Output: Emotional state data.

[0543] Step 6:

[0544] Providing optimal instructions

[0545] Support staff device: Using the user's work context and emotional data, prompts are input into a generative AI model (e.g., Chat GPT) to generate optimal instructions.

[0546] Input: Work background data, emotional state data.

[0547] Output: The generated directive.

[0548] User's device: Provide the generated instructions to the user. If the user is not satisfied with the content, they can request corrections or ask questions.

[0549] Input: The generated instruction.

[0550] Output: User feedback.

[0551] Step 7:

[0552] Meeting Recording

[0553] Server: After the meeting ends, the logs, communication history, shared files, emotional data, and generated instructions are saved in a MySQL database.

[0554] Input: Meeting logs, communication history, shared files, emotion data, generated instructions.

[0555] Output: Database save results.

[0556] On your device: Users can view and optionally download recordings from the "Meeting History" section of the web app.

[0557] Input: User's record confirmation request.

[0558] Output: Meeting recording data.

[0559] (Application example 2)

[0560] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0561] Conventional business support systems lack the ability to utilize generative AI to provide prompts to users so that they can receive efficient support in their work. Furthermore, they lack the ability to provide flexible support based on the user's emotional state, which can increase stress and reduce work efficiency. Furthermore, they lack a system for monitoring the operating status of factory robots in real time and quickly detect and respond to abnormalities, which can lead to production line stagnation.

[0562] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0563] In this invention, the server includes a means for users to sign up and log in, a means for users to reserve web meetings based on their work, a means for support staff to participate in the web meetings and provide optimal prompts based on the user's work background, a means for saving meeting records on the server, a means for collecting user emotion data using an emotion engine and adjusting prompt proposals based on that data, a means for monitoring the operating status of factory robots, and a means for sending notifications when an abnormality is detected. This allows users to use generative AI and emotion recognition to improve work efficiency, and enables them to monitor the operating status of factory robots in real time to quickly detect and respond to abnormalities.

[0564] "User" means an individual or corporation that uses the system to receive business support.

[0565] "Sign-up and login" refers to the process by which a user creates an account and accesses the system with an existing account.

[0566] A "web meeting" is a meeting conducted over the Internet between a user and a support staff member using screen sharing and audio / video calls.

[0567] A "prompt" is a specific instruction or suggestion that a generative AI model provides to a user.

[0568] A "support staff member" is a person who provides business support and advice to users.

[0569] "Server" means a computer system that stores and manages user data and meeting records.

[0570] An "emotion engine" is software that analyzes a user's facial expressions and voice to recognize their emotional state.

[0571] "Emotional Data" refers to information about a user's emotions collected and analyzed by the Emotion Engine.

[0572] A "factory robot" is a robot used to perform automated tasks in a factory.

[0573] "Operation status" refers to the status of a factory robot, indicating whether it is operating normally or whether an error or abnormality has occurred.

[0574] "Anomaly detection" is the process of detecting abnormalities when they occur in the operation of factory robots.

[0575] A "notification" is a message sent by the system to notify the user of an abnormality or event.

[0576] A "log" is a detailed record of web meetings and activity within the system.

[0577] "Chat history" means a record of text communications that take place during a web meeting.

[0578] "Shared files" refers to documents and materials shared between the user and the support staff during a web meeting.

[0579] This invention provides a support system for improving business efficiency by combining generative AI and an emotion recognition engine. In particular, it can monitor the operating status of factory robots in real time and quickly detect and respond to abnormalities. Below, we will explain in detail the embodiments of this system.

[0580] This support system consists of the following means:

[0581] 1. How users sign up and log in

[0582] Device: The user uses the smartphone app to create an account (sign up) or log in with an existing account.

[0583] Server: Store the user's input information in a database and use a service such as Firebase Authentication for authentication.

[0584] 2. A way for users to schedule business-based web meetings

[0585] On the device: Users can check available slots on the app dashboard and book a web meeting at the desired date and time.

[0586] Server: Saves user reservation information in a database and notifies support staff using the Google Calendar API, etc.

[0587] 3. A way for support staff to join a web meeting and provide the most appropriate prompts based on the user's work context

[0588] Device: When the web meeting starts, the user and support staff click the specified link to join. The screen sharing function is also used so that the user can explain the background of the work and the support staff can take notes.

[0589] Server: Generates secure web meeting links and sends reminder notifications.

[0590] 4. Using an emotion engine to gather user sentiment data and tailor prompts based on that data

[0591] Device: The camera captures the user's face, the microphone picks up their voice, and collects emotional data in real time.

[0592] Server: Analyzes emotional data using Microsoft Azure (registered trademark) Emotion API, etc., and determines the user's emotional state.

[0593] 5. A means of monitoring the operation status of factory robots

[0594] Terminal: Factory workers can check the status of running robots on a dashboard and select the robot with the problem.

[0595] Server: Updates the robot's operating status data in real time and sends notifications if an abnormality is detected.

[0596] 6. How to send notifications when an anomaly is detected

[0597] Server: When an abnormality is detected, a notification is sent immediately and an alert is issued to the user, which encourages a prompt response.

[0598] 7. How to store meeting records on a server

[0599] Server: Use a database service such as Firebase Firestore to store meeting logs, chat history, shared files, and generated prompt ideas in a database.

[0600] On your device: Users can review the recording after the meeting and download it if needed.

[0601] This system allows users to make the most of generative AI and emotion recognition to improve work efficiency, while also enabling them to monitor the operating status of factory robots in real time and quickly detect and respond to abnormalities.

[0602] Specific examples

[0603] Users sign up and log in

[0604] Enter your username, email address, and password, then click "Register."

[0605] A confirmation email will be sent to you and a link will take you to the login page.

[0606] Prompt Sentence Examples

[0607] "User's work context: Monitoring the operation status of a robot in a factory. User's emotional state: Angry (due to frequent errors). Desired output from the generative AI model: Suggest specific steps to improve work efficiency and quickly resolve issues."

[0608] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0609] Specific flow of processing steps

[0610] Step 1:

[0611] User sign-up and login

[0612] Input: A user opens the smartphone app and enters their username, email address, and password.

[0613] Processing: The device sends the entered information to Firebase Authentication, and the server performs authentication. If this is a new registration, the server sends a confirmation email to the user and waits for them to click the confirmation link.

[0614] Output: The server generates an authentication token and displays the user's target page.

[0615] What happens: The user receives a confirmation email and clicks on the link to authenticate their account. After successfully logging in, the dashboard will be displayed.

[0616] Step 2:

[0617] Users book web meetings

[0618] Input: User sees available slots on the app dashboard and selects their preferred time slot.

[0619] Processing: The device sends the reservation information to the server via the Google Calendar API. The server saves the reservation information in the database and notifies the support staff.

[0620] Output: A notification of the meeting reservation will be sent to the support staff. The reservation information will be added to Google Calendar.

[0621] Specific operation: The user selects the desired time and clicks the "Book" button. The server displays a confirmation message and adds the appointment to the support staff's calendar.

[0622] Step 3:

[0623] The user and the support staff join a web meeting

[0624] Input: When the web meeting starts, the user and support staff will click the specified link on their devices.

[0625] Processing: The server generates a secure web meeting link and sends a reminder. The user and support staff use screen sharing to explain the work background and take notes.

[0626] Output: The web meeting starts and both parties are connected correctly.

[0627] Specific operation: The server generates a meeting link, and the user and support staff join. The user shares materials, and the support staff takes notes while receiving explanations.

[0628] Step 4:

[0629] Emotional data collection and analysis using an emotion engine

[0630] Input: User's facial video and audio data are collected via camera and microphone.

[0631] Processing: The device sends the collected data to the Microsoft Azure Emotion API, where the server analyzes it and determines the user's emotional state.

[0632] Output: The user's emotional data is updated in real time and displayed to the support staff.

[0633] Specific operation: Data is collected in real time using a camera and microphone, and an emotion engine analyzes the user's state of joy, anger, sadness, etc.

[0634] Step 5:

[0635] Providing optimal prompts

[0636] Input: The support staff's device acquires the user's work background and emotional data.

[0637] Processing: Based on this data, the support staff sends the optimal prompt proposal to the generative AI model (e.g., OpenAI API). The generative AI model generates an appropriate prompt proposal and returns it to the support staff's device.

[0638] Output: The generated prompt proposal is displayed on the support staff's terminal. A suggestion is made to the user.

[0639] Specific operation: The support staff requests the AI ​​to generate prompts that best suit the user's situation, and then explains the generated prompts to the user.

[0640] Step 6:

[0641] Meeting recording

[0642] Input: Meeting logs, chat history, shared files, and generated prompt ideas are sent to the server.

[0643] Processing: The server stores this data in a database such as Firebase Firestore.

[0644] Output: Records are stored in a database for users to review and download at a later time.

[0645] Specific behavior: After the meeting ends, the server automatically saves logs and other records, allowing users to view details of past meetings.

[0646] In this way, the system helps users maximize the use of generative AI and emotion recognition to improve work efficiency.

[0647] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0648] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0649] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0650] [Second embodiment]

[0651] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0652] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0653] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0654] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0655] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0656] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0657] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0658] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0659] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0660] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0661] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0662] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0663] This invention is a system that provides support to users to improve work efficiency by utilizing generative AI. The system involves a series of processes in which users sign up and log in, book a web meeting, and receive optimal prompt suggestions through screen sharing with a support staff member.

[0664] Program processing overview

[0665] 1. Sign up and login

[0666] Device: The user accesses the web app and creates an account (signs up) or logs in with an existing account.

[0667] Server: Stores the information entered by the user in a database, performs authentication, and sends a confirmation email to the user once sign-up is complete.

[0668] 2. Schedule a web meeting

[0669] Terminal: Users can see available slots on their dashboard and select the desired date and time for the meeting.

[0670] Server: Saves reservation information in a database and notifies support staff.

[0671] 3. Participating in a web meeting

[0672] Terminal: When the meeting start time arrives, the user and support staff click on the provided link to join the web meeting.

[0673] Server: Sends reminders and generates and provides meeting links to participants.

[0674] 4. Explanation of the user's business background

[0675] Terminal: The user can use the screen sharing function to display the current status of the work to the support staff, provide verbal explanations, and share relevant materials and documents.

[0676] Support staff's terminal: The support staff understands the current situation based on the user's explanation and takes notes.

[0677] 5. Providing optimal prompts

[0678] Support staff device: Based on the user's work background, the system creates prompt ideas suitable for the generation AI and proposes them to the user.

[0679] User's device: The user receives the suggestions and asks questions or requests corrections as needed.

[0680] 6. Meeting Recording

[0681] Server: Stores meeting logs, chat history, shared files, and generated prompts in a database.

[0682] Device: Users can review recordings after the meeting and download them if necessary.

[0683] Specific examples

[0684] 1. User signs up and logs in

[0685] On the device: The user opens the web app, clicks the "Sign Up" button, enters their username, email address, and password, and clicks "Register."

[0686] Server: Stores the entered information in a database and sends a confirmation email to the user. When the user clicks on the link in the email, the login page is displayed.

[0687] 2. A user books a web meeting

[0688] On the device: The user goes to the "Book a Meeting" section of their dashboard, selects the desired date and time from the calendar, clicks the "Book" button and receives a confirmation message.

[0689] Server: Saves reservation information in a database and sends reservation notifications to support staff.

[0690] 3. The user and the support staff join a web meeting

[0691] On your device: The user opens the web app at the designated time and clicks the "Join Meeting" button. The support staff also joins.

[0692] Server: Generates secure web meeting links and sends reminder notifications.

[0693] 4. User explains the business background

[0694] Device: The user shares the documents and screens they use in their work and explains the specific work background and challenges to the support staff.

[0695] Support staff's device: Understand the job content and listen to the user's explanation while taking notes.

[0696] 5. Providing optimal prompts

[0697] Support staff terminal: Based on the user's explanation, it devises a prompt suitable for the generative AI and presents it to the user.

[0698] User's device: Accept suggestions and make any necessary corrections or questions.

[0699] 6. Meeting Recording

[0700] Server: After the meeting ends, the logs, chat history, shared files, and generated prompts are saved in a database.

[0701] Terminal: User can access the recording later and download it as needed.

[0702] In this way, the system helps users effectively utilize generative AI to improve business efficiency.

[0703] The processing flow will be explained below.

[0704] Sign up and log in

[0705] Step 1: View the sign-up page

[0706] Device: User visits web app and clicks "Sign up" button.

[0707] Server: Sends the sign-up page to the browser and displays it.

[0708] Step 2: Enter your sign-up information

[0709] On the device: The user fills in a form with their username, email address, password, and confirmation password.

[0710] Step 3: Submit your sign-up information

[0711] Terminal: The user clicks the "Register" button and sends the information to the server.

[0712] Step 4: Verify and save your sign-up information

[0713] Server: Checks the entered information for consistency and duplication, and if there are no problems, saves it to the database.

[0714] Server: Sends a confirmation email to the user.

[0715] Step 5: Display the login page

[0716] Server: After successful sign-up, display the login page to the user.

[0717] Step 6: Enter your login details

[0718] Device: The user enters their email address and password and clicks the "Login" button.

[0719] Step 7: Verify your login details

[0720] Server: The entered email address and password are verified against the database.

[0721] Step 8: View the Dashboard

[0722] Server: If authentication is successful, display the user's dashboard page.

[0723] Web Meeting Reservation

[0724] Step 1: View the booking page

[0725] On your device: The user accesses the "Book a Meeting" section of their dashboard.

[0726] Server: Sends data to display the reservation page.

[0727] Step 2: Select a reservation slot

[0728] Terminal: The user selects the desired date and time from the calendar.

[0729] Step 3: Confirm and submit your booking

[0730] Terminal: The user clicks the "Confirm Reservation" button and sends the information to the server.

[0731] Step 4: Save your reservation

[0732] Server: Saves the entered reservation information in a database and sends a reservation notification to the support staff.

[0733] Step 5: View the reservation confirmation message

[0734] Server: Display a confirmation message to the user that the reservation was successful.

[0735] Joining a web meeting

[0736] Step 1: Meeting start notification

[0737] Server: Send a reminder to users and support staff 10 minutes before the meeting start time.

[0738] Step 2: Generate a meetings link

[0739] Server: Generates and sends secure web meeting links to users and support personnel.

[0740] Step 3: Join the meeting

[0741] Terminal: The user and support staff click on the link sent to join the web meeting.

[0742] Step 4: Start Screen Sharing

[0743] Terminal: The user uses the screen sharing function to show the current work status to the support staff.

[0744] User's business background

[0745] Step 1: Oral explanation of the business background

[0746] Terminal: The user verbally explains the specific background and issues of the work to the support staff.

[0747] Step 2: Share materials

[0748] Terminal: The user provides relevant materials and documents to the support staff via screen sharing or file transfer.

[0749] Providing optimal prompts

[0750] Step 1: Create a Prompt

[0751] Support staff terminal: Generates optimal prompt suggestions based on the user's explanation.

[0752] Step 2: Propose a Prompt

[0753] Terminal: The support staff presents the proposed prompt to the user and explains it to them.

[0754] Step 3: Receive feedback and make corrections

[0755] On the device: The user provides feedback on the proposed prompt and requests revisions if necessary.

[0756] Support staff device: Receive feedback and revise the proposed prompt.

[0757] Meeting recording

[0758] Step 1: Save the meeting recording

[0759] Server: Stores meeting logs (chat history, shared files, generated prompts) in a database.

[0760] Step 2: Notification and Access to Records

[0761] Server: After the meeting ends, notify the user that the recording has been saved and provide an access link.

[0762] Step 3: Review and download the recording

[0763] On your device: Users can click the link provided to log in, view the meeting recording, and download it if necessary.

[0764] In this way, specific processing is carried out at each step, and the system provides assistance to users to make the most of generated AI and improve business efficiency.

[0765] Example 1

[0766] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0767] In today's work environment, users need appropriate support to efficiently utilize generative AI to perform their work. However, if users lack the specific background information or know-how to devise prompts, it can be difficult to maximize the effectiveness of generative AI. Furthermore, the process of sharing the status and progress of work with support staff in real time and receiving specific advice must be streamlined. Furthermore, it is also important to properly store meeting records so they can be referenced later.

[0768] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0769] In this invention, the server includes a means for a user to create and authenticate an account, a means for the user to select a date and time based on the user's work and schedule an online conference, and a means for a support staff member to participate in the online conference and generate optimal prompts based on the work materials shared by the user. This allows the user to receive advice from the appropriate support staff member and efficiently utilize the generation AI, significantly improving work efficiency. Furthermore, by properly storing online conference records and dialogue history, users can enjoy the benefit of making them easier to reference later.

[0770] "Creating an account and authenticating" refers to the process in which a user first registers (signs up) with the system and the system verifies the user's authenticity based on the information entered.

[0771] "Selecting a date and time to reserve an online meeting" refers to the process by which a user selects a specific date and time using the calendar function within the system and reserves the system to hold an online meeting at that date and time.

[0772] "Participating in an online meeting" refers to the process in which a user and a support staff member connect online using a web link generated at a specified date and time to hold a meeting.

[0773] "Generating optimal prompt sentences based on business materials" refers to the process in which a support staff member creates and provides effective prompts to the generation AI based on the business-related materials and information shared with the support staff by the user.

[0774] "Storing on a network server" refers to the process of securely storing online meeting records, conversation histories, shared data, etc. on a remote data storage system.

[0775] "Explaining the work background and sharing information on the screen in real time" refers to the process in which a user explains the details and background of the work while sharing the screen of their computer or device with a support staff member in real time during an online meeting.

[0776] "Storing online meeting logs, dialogue history, shared data, and generated prompts in a management system" refers to the process of storing all records and data related to online meetings in a dedicated management system for future reference.

[0777] This invention is a system designed to allow users to improve work efficiency by utilizing generated AI. Specific embodiments of the invention are described below.

[0778] overview

[0779] This system helps users create an account through a web application, schedule a business meeting, and participate in an online meeting, and then receive appropriate prompt advice from support staff, thereby improving work efficiency.

[0780] Hardware and Software Configuration

[0781] server

[0782] The server is a backend system for user authentication, conference booking, online conference link generation, and meeting record storage. Specifically, it has the following configuration:

[0783] Database: Use a database (e.g. PostgreSQL) to store user information, booking information, and meeting records.

[0784] Application server: Uses a backend framework (e.g., Node.js, Django) that provides user authentication and conference management functions.

[0785] Mail Server: Use the mail server to send account confirmation emails and meeting reminder notifications.

[0786] Terminal

[0787] The terminal is where users and support staff access and operate the web application. It consists of:

[0788] Browser: The user uses a web browser (e.g., Google Chrome, Firefox) to access the web application.

[0789] Screen sharing software: Use software (e.g., Zoom, Microsoft Teams) that allows users to share their screens with support personnel in real time.

[0790] Explanation of program processing

[0791] Sign up and log in

[0792] Device: The user accesses the web application using a browser, clicks the "Sign Up" button, enters their username, email address, and password, and clicks "Register." The server stores the entered information in a database and sends a confirmation email to the user. The user clicks the link in the email and their account is activated. The user then enters their email address and password on the login page and clicks "Login." The server performs authentication, and if successful, displays the dashboard to the user.

[0793] Examples:

[0794] A user accesses the web app in a browser, enters "testuser", "testuser@example.com", and "password123", and clicks the register button. The server stores this information in a database and sends a confirmation email to "testuser@example.com".

[0795] Web Meeting Reservation

[0796] Terminal: The user selects the desired date and time from the calendar in the dashboard and clicks the "Reserve" button. The server saves the reservation information in the database and sends a reservation notification to the support staff.

[0797] Examples:

[0798] The user selects "October 15, 2023, 3:00 PM" on the dashboard calendar and clicks the "Book" button. The server saves this information in the database and notifies the support staff by email.

[0799] Joining a web meeting

[0800] Terminal: When the meeting start time arrives, users and support staff click the provided link to join the web meeting. The server sends reminder notifications and generates and provides a secure web meeting link to participants.

[0801] Examples:

[0802] Five minutes before the meeting starts, the server sends a reminder to the user and support staff and provides them with a meeting link, which they click to join.

[0803] User's business background

[0804] Device: The user uses the screen sharing function to share the work situation and related documents with the support staff. The support staff understands the work situation based on the user's explanation and takes notes.

[0805] Examples:

[0806] The user clicks the screen sharing button in the browser and displays the Excel file used in work to the support staff.

[0807] Providing optimal prompts

[0808] Support staff terminal: The support staff will devise a prompt suitable for the generation AI based on the user's work background. The user will review the proposed prompt and make any necessary corrections or questions.

[0809] Specific prompt examples:

[0810] Suggest business prompts such as "Automatically generate invoices."

[0811] Meeting recording

[0812] Server: After the meeting, the logs, chat history, shared files, and generated prompts are stored in a database. Users can access the recordings later and download them as needed.

[0813] Examples:

[0814] The server stores meeting records, interaction history, and shared files in a database, which users can download by accessing the history section of their dashboard.

[0815] As described above, by using this system, users can effectively utilize generative AI to optimize their business processes. With the help of support staff, they can quickly create effective prompts, significantly improving work efficiency.

[0816] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0817] Step 1:

[0818] Create an account (sign up)

[0819] Device: The user visits the web application and enters their username, email address, and password into the sign-up form.

[0820] Specific behavior:

[0821] The user enters "testuser", "testuser@example.com", and "password123" and clicks the Register button.

[0822] Server: The server stores the entered username, email address, and password in a database and sends a confirmation email to the user. Based on the user's input, the server stores the information in a database and sends a confirmation email.

[0823] Input: Username, Email Address, Password

[0824] Data processing: Save user information in a database and request the mail server to send a confirmation email

[0825] Output: User information stored in the database, confirmation email sent to the user

[0826] Step 2:

[0827] Account authentication (login)

[0828] Device: After signing up, the user enters their email address and password on the login page and clicks "Login."

[0829] Specific behavior:

[0830] The user enters "testuser@example.com" and "password123" and clicks the login button.

[0831] Server: The server checks the input information against the database and performs authentication. If successful, it displays the dashboard to the user.

[0832] Input: Email address, Password

[0833] Data processing: Retrieving user information from the database and matching it with the input information

[0834] Output: Authentication result. If successful, the user's dashboard screen will be displayed.

[0835] Step 3:

[0836] Web Meeting Reservation

[0837] Terminal: The user selects the desired date and time using the calendar function in the dashboard and clicks the "Book" button.

[0838] Specific behavior:

[0839] The user selects "October 15, 2023, 3:00 PM" from the calendar and clicks the reservation button.

[0840] Server: The server saves the reservation information in a database and sends a reservation notification to the support staff. It saves the information to the database and notifies the staff based on the user's input.

[0841] Input: Reservation date and time

[0842] Data processing: Save the reservation information to the database and call the notification system to send reservation notifications to support staff.

[0843] Output: Booking information stored in the database, booking notification sent to support staff

[0844] Step 4:

[0845] Joining a web meeting

[0846] Server: Sends reminders to users and support staff before the meeting starts and generates a secure web meeting link.

[0847] Input: Reservation information

[0848] Data processing: generating and sending reminder notifications, generating web meeting links

[0849] Output: Reminder sent, generated web meeting link

[0850] Terminal: Users and support staff click on the provided link to join the web meeting.

[0851] Specific behavior:

[0852] Five minutes before the meeting starts, the user and support staff will check the notification and click the provided link to join.

[0853] Step 5:

[0854] Business background explanation

[0855] Device: Users can use the screen sharing function to share work status and related documents with support staff.

[0856] Specific behavior:

[0857] The user clicks the screen sharing button in the browser and displays an Excel file used for work.

[0858] Support staff's device: The support staff understands the work situation based on the information shared on the screen and the user's explanation, and takes notes.

[0859] Input: Screen sharing information, user's verbal description

[0860] Data processing: Support staff understanding and recording

[0861] Output: Support personnel notes

[0862] Step 6:

[0863] Providing optimal prompts

[0864] Support staff terminal: The support staff will devise a prompt suitable for the generation AI based on the user's work background and present it to the user.

[0865] Input: User's business background information

[0866] Data processing: Prompt generation

[0867] Output: Suggested prompt text

[0868] Example prompt:

[0869] Suggest business prompts such as "Automatically generate invoices."

[0870] User's device: The user reviews the suggested prompt and makes any necessary corrections or questions.

[0871] Specific behavior:

[0872] The user reviews the prompts from the support staff and, if necessary, requests to "change the date format."

[0873] Step 7:

[0874] Meeting recording

[0875] Server: After the meeting ends, the log, chat history, shared files, and generated prompts are saved in a database.

[0876] Input: Meeting logs, chat history, shared files, generated prompts

[0877] Data processing: Record preservation processing

[0878] Output: Records stored in a database

[0879] Terminal: User can access the recording later and download it if needed.

[0880] Specific behavior:

[0881] After a meeting ends, users can access the History section of their dashboard and download a specific meeting recording.

[0882] (Application example 1)

[0883] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0884] To improve efficiency and productivity in food delivery operations, there is a need for technology that allows users to collaborate with support staff in real time and provides optimal routes using generative AI models. However, conventional systems have issues with being unable to efficiently perform a series of tasks, such as web meetings, recording work, and providing optimal routes.

[0885] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0886] In this invention, the server includes a means for users to sign up and log in, a means for users to book web meetings based on their tasks, a means for support staff to participate in the web meetings and provide optimal prompts based on the user's work background, a means for saving meeting records on the server, a means for users to manage delivery task reservations, a means for providing optimal route suggestions using a generative AI model, and a means for inputting feedback after the task is completed. This significantly improves the efficiency and productivity of food delivery tasks, and enables users to effectively use generative AI models to optimize their tasks.

[0887] "User" means a person who uses the system to improve business efficiency.

[0888] "Signing up" is the process of creating a new account.

[0889] "Login" means the act of authenticating to access an existing account and use the system.

[0890] A "web meeting" is an online conference held over the Internet.

[0891] "Reservation" is the act of setting up a web meeting for a specific date and time.

[0892] A "support staff member" is someone whose role is to support users in optimizing their operations.

[0893] A "prompt" is an input sentence that gives instructions to a generative AI model.

[0894] "Recording" refers to saving the progress and content of a meeting in a format that can be referenced later.

[0895] A "server" is a computer system that stores and processes data.

[0896] "Delivery services" refers to the general business of delivering goods or services to designated locations.

[0897] A "generative AI model" is an artificial intelligence algorithm that makes inferences and predictions based on data.

[0898] A "route proposal" is a plan that suggests an efficient route for delivery work.

[0899] "Feedback" is the act of recording evaluations and opinions after a task is completed.

[0900] This invention is a system that uses generative AI models to help improve the efficiency of food delivery operations. The system involves a series of processes in which users sign up and log in, book web meetings, and share their work status with support staff in real time.

[0901] 1. Sign up and login

[0902] The server stores the user's input information in a database and performs authentication. For example, when signing up, the user enters their username, email address, and password, and after receiving a verification email, clicks the link to log in.

[0903] 2. Delivery business reservation management

[0904] The user's device is used to manage delivery schedules. The server uses the Google Calendar API to store reservation information in a cloud database and notify support staff.

[0905] 3. Participating in a web meeting

[0906] Using WebRTC, the user and support staff join a web meeting at the specified time. The server sends a reminder notification and generates a secure meeting link.

[0907] 4. Explanation of business background and screen sharing

[0908] Users can use the screen sharing function to show support staff the documents and progress they are using in their work, making it easier for support staff to understand the user's work background.

[0909] 5. Providing optimal route suggestions

[0910] The support staff terminal uses a generative AI model based on the user's work background to provide optimal route suggestions. For example, the generative AI can suggest optimal routes using prompts such as: "Please suggest the optimal route based on the delivery address and customer address."

[0911] 6. Feedback after the job is completed

[0912] After completing a delivery task, users can enter feedback using their devices. This feedback is stored on the server and can be used to improve future operations.

[0913] 7. Meeting Recording

[0914] The server stores meeting logs, chat history, shared files, and generated prompts in a database such as Firebase Firestore, allowing for future review and continuous improvement.

[0915] For example, a user who wants to streamline delivery operations during lunchtime can enter the following: "We have a high volume of deliveries during lunchtime. Please suggest the best route and most efficient delivery method."

[0916] Generative AI analyzes data to create optimal route plans in real time and notify users' smartphones, thereby significantly improving the efficiency of food delivery operations.

[0917] The system can be realized using specific hardware and software such as Firebase Authentication, Google Calendar API, WebRTC, OpenAI GPT-3, and Firebase Firestore.

[0918] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0919] Step 1:

[0920] A user signs up.

[0921] Input: Username, Email Address, Password

[0922] Data processing: The terminal collects the user's input information and sends it to the server.

[0923] Output: The server stores the information in a database and sends a verification email.

[0924] How it works: A user opens the web app, enters the required information, and clicks "Register." The server receives the information, writes it to the database, and then sends a verification email.

[0925] Step 2:

[0926] The user logs in.

[0927] Input: Email address, password

[0928] Data processing: The server validates and authenticates the input information.

[0929] Output: Login success message and transition to the dashboard screen

[0930] What happens: The user enters their email address and password on the login page and clicks "Login." The server verifies the information and, if correct, displays the dashboard screen.

[0931] Step 3:

[0932] A user books a delivery job.

[0933] Input: Scheduled delivery date and time, detailed information

[0934] Data processing: The server saves the reservation information and updates the calendar using the Google Calendar API.

[0935] Output: Reservation confirmation and notification to support staff

[0936] What it does: The user selects the desired date and time on the dashboard and clicks "Book." The server saves the reservation information, updates the calendar using the Google Calendar API, and notifies the support staff.

[0937] Step 4:

[0938] A user joins a web meeting.

[0939] Input: Meeting time, link

[0940] Data processing: The server sends reminder notifications and generates secure meeting links over WebRTC.

[0941] Output: Reminders, Meeting Links

[0942] Specific operation: When the meeting start time arrives, the user and support staff click the designated link to join the web meeting. The server sends a notification and generates and distributes the link.

[0943] Step 5:

[0944] The user explains the background of the work to the support staff and shares the screen.

[0945] Input: Business documents, progress status

[0946] Data processing: The device shares the screen and communicates the work status to the support staff.

[0947] Output: Shared work documents, progress explanation

[0948] Specific operation: The user uses the screen sharing function to show the support staff the materials and progress, and the support staff deepens their understanding and takes notes.

[0949] Step 6:

[0950] Support staff use generative AI models to provide optimal route suggestions.

[0951] Input: User's business background information, delivery route

[0952] Data processing: A generative AI model calculates and suggests efficient routes.

[0953] Output: Optimal route plan

[0954] Specific operation: The support staff inputs prompts into the AI ​​generator based on the information obtained from the user and suggests the optimal route. Example prompt: "Please suggest the optimal route based on the delivery address and customer address."

[0955] Step 7:

[0956] The user enters feedback after completing the delivery job.

[0957] Input: Feedback

[0958] Data processing: The server stores the feedback information.

[0959] Output: Saved feedback data

[0960] What it does: A user enters and submits feedback using the web app. The server stores the information in a database.

[0961] Step 8:

[0962] The server stores the meeting records.

[0963] Input: Meeting logs, chat history, shared files, generated prompt ideas

[0964] Data processing: The server stores the data in a database.

[0965] Output: Saved meeting data

[0966] Specific operation: After the meeting ends, the server saves the logs, chat history, shared files, and generated prompt ideas in a database such as Firebase Firestore.

[0967] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0968] This invention is a system that provides support to users to improve work efficiency using generative AI, and aims to provide more effective support by combining it with an emotion engine that recognizes and responds to user emotions. The system involves a series of processes in which users sign up and log in, book a web meeting, explain the work background through screen sharing with a support staff member, and receive optimal prompt suggestions. In addition, the emotion engine collects user emotion data, adjusts prompt suggestions based on that data, and stores records.

[0969] Program processing overview

[0970] 1. Sign up and login

[0971] Device: The user accesses the web app and creates an account (signs up) or logs in with an existing account.

[0972] Server: Stores the information entered by the user in a database, performs authentication, and sends a confirmation email to the user once sign-up is complete.

[0973] 2. Schedule a web meeting

[0974] Terminal: Users can see available slots on their dashboard and select the desired date and time for the meeting.

[0975] Server: Saves reservation information in a database and notifies support staff.

[0976] 3. Participating in a web meeting

[0977] Terminal: When the meeting start time arrives, the user and support staff click on the provided link to join the web meeting.

[0978] Server: Sends reminders and generates and provides meeting links to participants.

[0979] 4. Explanation of the user's business background

[0980] Terminal: The user can use the screen sharing function to display the current status of the work to the support staff, provide verbal explanations, and share relevant materials and documents.

[0981] Support staff's terminal: The support staff understands the current situation based on the user's explanation and takes notes.

[0982] 5. Operation of the Emotion Engine

[0983] Device: Uses a camera and microphone to analyze the user's facial expressions and voice, and collects emotional data from them.

[0984] Server: The emotion engine processes the collected data in real time and recognizes the user's emotions.

[0985] 6. Providing optimal prompts

[0986] Support staff device: Based on the user's work background and emotional data, the device creates prompt ideas suitable for the generating AI and proposes them to the user.

[0987] User's device: The user receives the suggestions and asks questions or requests corrections as needed.

[0988] 7. Meeting Recording

[0989] Server: Stores meeting logs, chat history, shared files, sentiment data, and generated prompt ideas in a database.

[0990] Device: Users can review recordings after the meeting and download them if necessary.

[0991] Specific examples

[0992] 1. User signs up and logs in

[0993] On the device: The user opens the web app, clicks the "Sign Up" button, enters their username, email address, and password, and clicks "Register."

[0994] Server: Stores the entered information in a database and sends a confirmation email to the user. When the user clicks on the link in the email, the login page is displayed.

[0995] 2. A user books a web meeting

[0996] On the device: The user goes to the "Book a Meeting" section of their dashboard, selects the desired date and time from the calendar, clicks the "Book" button and receives a confirmation message.

[0997] Server: Saves reservation information in a database and sends reservation notifications to support staff.

[0998] 3. The user and the support staff join a web meeting

[0999] On your device: The user opens the web app at the designated time and clicks the "Join Meeting" button. The support staff also joins.

[1000] Server: Generates secure web meeting links and sends reminder notifications.

[1001] 4. User explains the business background

[1002] Device: The user shares the documents and screens they use in their work and explains the specific work background and challenges to the support staff.

[1003] Support staff's device: Understand the job content and listen to the user's explanation while taking notes.

[1004] 5. Operation of the Emotion Engine

[1005] Device: Captures the user's face with a camera, picks up their voice with a microphone, and collects emotional data in real time.

[1006] Server: Analyzes the collected data and determines the user's emotional state.

[1007] 6. Providing optimal prompts

[1008] Support staff device: Taking into account the user's work background and emotional state, the system creates prompt proposals suitable for the generative AI and presents them to the user.

[1009] User's device: The user accepts the suggestions and makes any necessary corrections or questions.

[1010] 7. Meeting Recording

[1011] Server: After the meeting ends, the logs, chat history, shared files, emotion data, and generated prompts are stored in a database.

[1012] On-device: Users can review saved records later and download them as needed.

[1013] In this way, the system helps users maximize the use of generative AI and emotion recognition to improve work efficiency.

[1014] The processing flow will be explained below.

[1015] Sign up and log in

[1016] Step 1: View the sign-up page

[1017] Device: User visits web app and clicks "Sign up" button.

[1018] Server: Sends the sign-up page to the browser and displays it.

[1019] Step 2: Enter your sign-up information

[1020] On the device: The user fills in a form with their username, email address, password, and confirmation password.

[1021] Step 3: Submit your sign-up information

[1022] Terminal: The user clicks the "Register" button and sends the information to the server.

[1023] Step 4: Verify and save your sign-up information

[1024] Server: Checks the entered information for consistency and duplication, and if there are no problems, saves it to the database.

[1025] Server: Sends a confirmation email to the user.

[1026] Step 5: Display the login page

[1027] Server: After successful sign-up, display the login page to the user.

[1028] Step 6: Enter your login details

[1029] Device: The user enters their email address and password and clicks the "Login" button.

[1030] Step 7: Verify your login details

[1031] Server: The entered email address and password are verified against the database.

[1032] Step 8: View the Dashboard

[1033] Server: If authentication is successful, display the user's dashboard page.

[1034] Web Meeting Reservation

[1035] Step 1: View the booking page

[1036] On your device: The user accesses the "Book a Meeting" section of their dashboard.

[1037] Server: Sends data to display the reservation page.

[1038] Step 2: Select a reservation slot

[1039] Terminal: The user selects the desired date and time from the calendar.

[1040] Step 3: Confirm and submit your booking

[1041] Terminal: The user clicks the "Confirm Reservation" button and sends the information to the server.

[1042] Step 4: Save your reservation

[1043] Server: Saves the entered reservation information in a database and sends a reservation notification to the support staff.

[1044] Step 5: View the reservation confirmation message

[1045] Server: Display a confirmation message to the user that the reservation was successful.

[1046] Joining a web meeting

[1047] Step 1: Meeting start notification

[1048] Server: Send a reminder to users and support staff 10 minutes before the meeting start time.

[1049] Step 2: Generate a meetings link

[1050] Server: Generates and sends secure web meeting links to users and support personnel.

[1051] Step 3: Join the meeting

[1052] Terminal: The user and support staff click on the link sent to join the web meeting.

[1053] Step 4: Start Screen Sharing

[1054] Terminal: The user uses the screen sharing function to show the current work status to the support staff.

[1055] User's business background

[1056] Step 1: Oral explanation of the business background

[1057] Terminal: The user verbally explains the specific background and issues of the work to the support staff.

[1058] Step 2: Share materials

[1059] Terminal: The user provides relevant materials and documents to the support staff via screen sharing or file transfer.

[1060] Emotion Engine Operation

[1061] Step 1: Collecting emotion data

[1062] Device: Emotional data is collected by taking a picture of the user's face with a camera and picking up their voice with a microphone.

[1063] Step 2: Analyze the emotion data

[1064] Server: Analyzes collected data in real time and determines the user's emotional state.

[1065] Providing optimal prompts

[1066] Step 1: Create a Prompt

[1067] Support staff device: Based on the user's work background and emotional data, the device creates a prompt suitable for the generating AI and presents it to the user.

[1068] Step 2: Propose a Prompt

[1069] Terminal: The support staff presents the proposed prompt to the user and explains it to them.

[1070] Step 3: Receive feedback and make corrections

[1071] On the device: The user provides feedback on the proposed prompt and requests revisions if necessary.

[1072] Support staff device: Receive feedback and revise the proposed prompt.

[1073] Meeting recording

[1074] Step 1: Save the meeting recording

[1075] Server: Stores meeting logs (conversation records, screen sharing content, emotion data, and generated prompts) in a database.

[1076] Step 2: Notification and Access to Records

[1077] Server: After the meeting ends, notify the user that the recording has been saved and provide an access link.

[1078] Step 3: Review and download the recording

[1079] On your device: Users can click the link provided to log in, view the meeting recording, and download it if necessary.

[1080] In this way, specific actions are performed at each processing step, and the system helps users maximize the use of generative AI and emotion recognition to improve work efficiency.

[1081] Example 2

[1082] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1083] Conventional business support systems have difficulty providing optimal instructions based on the user's work context, and do not take the user's emotions into consideration when providing support. As a result, there are problems with user satisfaction and work efficiency declining.

[1084] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1085] In this invention, the server includes a means for a user to sign up and log in, a means for a user to reserve an online meeting based on the user's work, a means for a support person to participate in the online meeting and provide optimal instructions based on the user's work background, a means for analyzing the user's facial expressions and voice and collecting emotional data, a means for processing the collected emotional data in real time and recognizing the user's emotions, and a means for saving a record of the meeting on the server. This makes it possible to provide appropriate instructions according to the user's work background and support that takes the user's emotions into consideration.

[1086] "Signing up" is the process by which a user creates a new account and registers with the system.

[1087] "Logging in" is the process of accessing a system using existing account information.

[1088] An "online meeting" is a conference held over the Internet, allowing participants to communicate in real time.

[1089] "Support Personnel" means personnel who provide technical and business support to Users.

[1090] "Instructions" are texts created using generative AI models that provide specific suggestions or guidelines to help users with their work.

[1091] "Facial Expression" refers to the muscle movements and other physical characteristics of a user's face that indicate an emotional state.

[1092] "Voice" refers to the voice produced by the user, and is data used to determine the user's emotional state using the tone, pitch, volume, etc.

[1093] "Emotional data" is information that expresses the user's emotional state as numerical or categorical data.

[1094] "Real-time processing" means processing data immediately after it is generated, providing results with little to no delay.

[1095] A "meeting recording" is a series of records containing digital data, such as conversations, chats, and shared files, that take place during an online meeting.

[1096] This invention is a support system that utilizes a generative AI model to help users improve their work efficiency, and also combines it with an emotion engine that recognizes and responds to user emotions.

[1097] The system includes the following means:

[1098] 1. Sign-up and login methods

[1099] Device: A user visits a web app using a JavaScript framework to sign up and log in. Specifically, the user enters a username, email address, and password, and submits the sign-up form.

[1100] Server: Build a backend API using Node.js and Express, store this information in a MySQL database, and use an SMTP server to send confirmation emails, for example, sendgrid.

[1101] 2. Online Meeting Reservation Methods

[1102] On the device: Users access a dashboard powered by React.js and use a calendar component such as FullCalendar.js to book online meetings.

[1103] Server: Reservation information is stored in a MySQL database, and the server sends reservation confirmation emails to users and support staff.

[1104] 3. How to join an online meeting

[1105] Device: Users and support staff log in to the web app at the specified date and time and join through the API of online meeting tools such as Google Meet and Zoom.

[1106] Server: Sends reminders near the scheduled time and generates a secure meeting link.

[1107] 4. Means of explaining the business background

[1108] Device: During online meetings, users can use Zoom's screen sharing function to explain the current work background and related materials to support staff.

[1109] Support staff's device: The support staff listens to the user's explanation and takes notes of important points using a note-taking app (e.g., OneNote) on their PC.

[1110] 5. Emotion Engine Operation Method

[1111] Device: The user's face is photographed with a webcam and their voice is collected with a microphone. A facial expression recognition model and voice analysis tool using TensorFlow.js are used.

[1112] Server: Uses a Python sentiment analysis library (e.g., emotion_recognition) to process the collected data in real time and recognize user emotions.

[1113] 6. Optimal Means of Providing Instructions

[1114] Support staff terminal: The support staff uses a generative AI model (e.g., Chat GPT) to generate instructions based on the user's work background and emotional data.

[1115] User's device: Provide the generated instructions to the user, correcting or asking questions as needed.

[1116] Examples of prompts:

[1117] "I want to use a generative AI model to generate optimal prompt suggestions. My user is preparing a presentation for a new marketing campaign and is particularly short on ideas for slide design. He's feeling stressed. What would be the best prompt?"

[1118] 7. Meeting Record Keeping Method

[1119] Server: After the online meeting ends, the logs, communication history, shared files, emotional data, and generated instructions are stored in a MySQL database.

[1120] On your device: Users can view saved recordings in the "Meeting History" section of the web app and download them if necessary.

[1121] In this way, the present invention provides specific assistance to users to make the most of generative AI models and emotion recognition to improve work efficiency.

[1122] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1123] Step 1:

[1124] Sign up and log in

[1125] On the device: The user opens the web app and clicks the "Sign up" button. The user enters their username, email address, and password and clicks "Register."

[1126] Enter your username, email address, and password.

[1127] Output: Signup request data.

[1128] Server: Receives the sign-up request data, stores the entered information in a MySQL database, and then sends a confirmation email using an SMTP server.

[1129] Input: Signup request data.

[1130] Output: Database saved results and confirmation email.

[1131] On the device: When the user clicks on the link in the confirmation email, they are authenticated and directed to a login page.

[1132] Enter: Click on the confirmation link.

[1133] Output: Login page displayed.

[1134] Step 2:

[1135] Schedule an online meeting

[1136] On the device: The user opens the dashboard, selects the desired date and time from the calendar, and clicks the "Book" button.

[1137] Input: Desired date and time.

[1138] Output: Booking request data.

[1139] Server: Receives the reservation request data, stores the reservation information in a MySQL database, and then sends a reservation confirmation email to the user and support staff.

[1140] Input: Booking request data.

[1141] Output: Database saved results and confirmation email.

[1142] Step 3:

[1143] Joining an online meeting

[1144] Device: The user and support staff open the web app at the designated time and click the "Join Meeting" button.

[1145] Input: Meeting time, Join button click.

[1146] Output: Meeting join request.

[1147] Server: Sends reminders before the specified time and generates a secure meeting link using Google Meet or Zoom APIs.

[1148] Input: Meeting join request.

[1149] Output: Reminder and meeting link.

[1150] Step 4:

[1151] Business background explanation

[1152] Terminal: The user uses the screen sharing function to show the support staff the current task and related materials, and provides verbal explanations.

[1153] Input: Start screen sharing, business documents.

[1154] Output: Screen sharing data.

[1155] Support staff's device: The support staff listens to the user's explanation and takes notes of important points using a note-taking app on their PC.

[1156] Input: Screen share data.

[1157] Output: Note data.

[1158] Step 5:

[1159] Emotion Engine Operation

[1160] Device: The user's face is photographed with a webcam and their voice is collected with a microphone. TensorFlow.js is used to capture facial expression and voice data in real time.

[1161] Input: Video and audio data.

[1162] Output: facial expression data, voice data.

[1163] Server: Sends the collected data to a Python sentiment analysis library to recognize the user's emotional state.

[1164] Input: facial expression data, voice data.

[1165] Output: Emotional state data.

[1166] Step 6:

[1167] Providing optimal instructions

[1168] Support staff device: Using the user's work context and emotional data, prompts are input into a generative AI model (e.g., Chat GPT) to generate optimal instructions.

[1169] Input: Work background data, emotional state data.

[1170] Output: The generated directive.

[1171] User's device: Provide the generated instructions to the user. If the user is not satisfied with the content, they can request corrections or ask questions.

[1172] Input: The generated instruction.

[1173] Output: User feedback.

[1174] Step 7:

[1175] Meeting Recording

[1176] Server: After the meeting ends, the logs, communication history, shared files, emotional data, and generated instructions are saved in a MySQL database.

[1177] Input: Meeting logs, communication history, shared files, emotion data, generated instructions.

[1178] Output: Database save results.

[1179] On your device: Users can view and optionally download recordings from the "Meeting History" section of the web app.

[1180] Input: User's record confirmation request.

[1181] Output: Meeting recording data.

[1182] (Application example 2)

[1183] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1184] Conventional business support systems lack the ability to utilize generative AI to provide prompts to users so that they can receive efficient support in their work. Furthermore, they lack the ability to provide flexible support based on the user's emotional state, which can increase stress and reduce work efficiency. Furthermore, they lack a system for monitoring the operating status of factory robots in real time and quickly detect and respond to abnormalities, which can lead to production line stagnation.

[1185] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1186] In this invention, the server includes a means for users to sign up and log in, a means for users to reserve web meetings based on their work, a means for support staff to participate in the web meetings and provide optimal prompts based on the user's work background, a means for saving meeting records on the server, a means for collecting user emotion data using an emotion engine and adjusting prompt proposals based on that data, a means for monitoring the operating status of factory robots, and a means for sending notifications when an abnormality is detected. This allows users to use generative AI and emotion recognition to improve work efficiency, and enables them to monitor the operating status of factory robots in real time to quickly detect and respond to abnormalities.

[1187] "User" means an individual or corporation that uses the system to receive business support.

[1188] "Sign-up and login" refers to the process by which a user creates an account and accesses the system with an existing account.

[1189] A "web meeting" is a meeting conducted over the Internet between a user and a support staff member using screen sharing and audio / video calls.

[1190] A "prompt" is a specific instruction or suggestion that a generative AI model provides to a user.

[1191] A "support staff member" is a person who provides business support and advice to users.

[1192] "Server" means a computer system that stores and manages user data and meeting records.

[1193] An "emotion engine" is software that analyzes a user's facial expressions and voice to recognize their emotional state.

[1194] "Emotional Data" refers to information about a user's emotions collected and analyzed by the Emotion Engine.

[1195] A "factory robot" is a robot used to perform automated tasks in a factory.

[1196] "Operation status" refers to the status of a factory robot, indicating whether it is operating normally or whether an error or abnormality has occurred.

[1197] "Anomaly detection" is the process of detecting abnormalities when they occur in the operation of factory robots.

[1198] A "notification" is a message sent by the system to notify the user of an abnormality or event.

[1199] A "log" is a detailed record of web meetings and activity within the system.

[1200] "Chat history" means a record of text communications that take place during a web meeting.

[1201] "Shared files" refers to documents and materials shared between the user and the support staff during a web meeting.

[1202] This invention provides a support system for improving business efficiency by combining generative AI and an emotion recognition engine. In particular, it can monitor the operating status of factory robots in real time and quickly detect and respond to abnormalities. Below, we will explain in detail the embodiments of this system.

[1203] This support system consists of the following means:

[1204] 1. How users sign up and log in

[1205] Device: The user uses the smartphone app to create an account (sign up) or log in with an existing account.

[1206] Server: Store the user's input information in a database and use a service such as Firebase Authentication for authentication.

[1207] 2. A way for users to schedule business-based web meetings

[1208] On the device: Users can check available slots on the app dashboard and book a web meeting at the desired date and time.

[1209] Server: Saves user reservation information in a database and notifies support staff using the Google Calendar API, etc.

[1210] 3. A way for support staff to join a web meeting and provide the most appropriate prompts based on the user's work context

[1211] Device: When the web meeting starts, the user and support staff click the specified link to join. The screen sharing function is also used so that the user can explain the background of the work and the support staff can take notes.

[1212] Server: Generates secure web meeting links and sends reminder notifications.

[1213] 4. Using an emotion engine to gather user sentiment data and tailor prompts based on that data

[1214] Device: The camera captures the user's face, the microphone picks up their voice, and collects emotional data in real time.

[1215] Server: Analyzes emotional data using Microsoft Azure Emotion API and other tools to determine the user's emotional state.

[1216] 5. A means of monitoring the operation status of factory robots

[1217] Terminal: Factory workers can check the status of running robots on a dashboard and select the robot with the problem.

[1218] Server: Updates the robot's operating status data in real time and sends notifications if an abnormality is detected.

[1219] 6. How to send notifications when an anomaly is detected

[1220] Server: When an abnormality is detected, a notification is sent immediately and an alert is issued to the user, which encourages a prompt response.

[1221] 7. How to store meeting records on a server

[1222] Server: Use a database service such as Firebase Firestore to store meeting logs, chat history, shared files, and generated prompt ideas in a database.

[1223] On your device: Users can review the recording after the meeting and download it if needed.

[1224] This system allows users to make the most of generative AI and emotion recognition to improve work efficiency, while also enabling them to monitor the operating status of factory robots in real time and quickly detect and respond to abnormalities.

[1225] Specific examples

[1226] Users sign up and log in

[1227] Enter your username, email address, and password, then click "Register."

[1228] A confirmation email will be sent to you and a link will take you to the login page.

[1229] Prompt Sentence Examples

[1230] "User's work context: Monitoring the operation status of a robot in a factory. User's emotional state: Angry (due to frequent errors). Desired output from the generative AI model: Suggest specific steps to improve work efficiency and quickly resolve issues."

[1231] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1232] Specific flow of processing steps

[1233] Step 1:

[1234] User sign-up and login

[1235] Input: A user opens the smartphone app and enters their username, email address, and password.

[1236] Processing: The device sends the entered information to Firebase Authentication, and the server performs authentication. If this is a new registration, the server sends a confirmation email to the user and waits for them to click the confirmation link.

[1237] Output: The server generates an authentication token and displays the user's target page.

[1238] What happens: The user receives a confirmation email and clicks on the link to authenticate their account. After successfully logging in, the dashboard will be displayed.

[1239] Step 2:

[1240] Users book web meetings

[1241] Input: User sees available slots on the app dashboard and selects their preferred time slot.

[1242] Processing: The device sends the reservation information to the server via the Google Calendar API. The server saves the reservation information in the database and notifies the support staff.

[1243] Output: A notification of the meeting reservation will be sent to the support staff. The reservation information will be added to Google Calendar.

[1244] Specific operation: The user selects the desired time and clicks the "Book" button. The server displays a confirmation message and adds the appointment to the support staff's calendar.

[1245] Step 3:

[1246] The user and the support staff join a web meeting

[1247] Input: When the web meeting starts, the user and support staff will click the specified link on their devices.

[1248] Processing: The server generates a secure web meeting link and sends a reminder. The user and support staff use screen sharing to explain the work background and take notes.

[1249] Output: The web meeting starts and both parties are connected correctly.

[1250] Specific operation: The server generates a meeting link, and the user and support staff join. The user shares materials, and the support staff takes notes while receiving explanations.

[1251] Step 4:

[1252] Emotional data collection and analysis using an emotion engine

[1253] Input: User's facial video and audio data are collected via camera and microphone.

[1254] Processing: The device sends the collected data to the Microsoft Azure Emotion API, where the server analyzes it and determines the user's emotional state.

[1255] Output: The user's emotional data is updated in real time and displayed to the support staff.

[1256] Specific operation: Data is collected in real time using a camera and microphone, and an emotion engine analyzes the user's state of joy, anger, sadness, etc.

[1257] Step 5:

[1258] Providing optimal prompts

[1259] Input: The support staff's device acquires the user's work background and emotional data.

[1260] Processing: Based on this data, the support staff sends the optimal prompt proposal to the generative AI model (e.g., OpenAI API). The generative AI model generates an appropriate prompt proposal and returns it to the support staff's device.

[1261] Output: The generated prompt proposal is displayed on the support staff's terminal. A suggestion is made to the user.

[1262] Specific operation: The support staff requests the AI ​​to generate prompts that best suit the user's situation, and then explains the generated prompts to the user.

[1263] Step 6:

[1264] Meeting recording

[1265] Input: Meeting logs, chat history, shared files, and generated prompt ideas are sent to the server.

[1266] Processing: The server stores this data in a database such as Firebase Firestore.

[1267] Output: Records are stored in a database for users to review and download at a later time.

[1268] Specific behavior: After the meeting ends, the server automatically saves logs and other records, allowing users to view details of past meetings.

[1269] In this way, the system helps users maximize the use of generative AI and emotion recognition to improve work efficiency.

[1270] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1271] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1272] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1273] [Third embodiment]

[1274] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1275] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1276] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1277] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1278] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1279] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1280] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1281] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1282] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1283] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1284] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1285] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1286] This invention is a system that provides support to users to improve work efficiency by utilizing generative AI. The system involves a series of processes in which users sign up and log in, book a web meeting, and receive optimal prompt suggestions through screen sharing with a support staff member.

[1287] Program processing overview

[1288] 1. Sign up and login

[1289] Device: The user accesses the web app and creates an account (signs up) or logs in with an existing account.

[1290] Server: Stores the information entered by the user in a database, performs authentication, and sends a confirmation email to the user once sign-up is complete.

[1291] 2. Schedule a web meeting

[1292] Terminal: Users can see available slots on their dashboard and select the desired date and time for the meeting.

[1293] Server: Saves reservation information in a database and notifies support staff.

[1294] 3. Participating in a web meeting

[1295] Terminal: When the meeting start time arrives, the user and support staff click on the provided link to join the web meeting.

[1296] Server: Sends reminders and generates and provides meeting links to participants.

[1297] 4. Explanation of the user's business background

[1298] Terminal: The user can use the screen sharing function to display the current status of the work to the support staff, provide verbal explanations, and share relevant materials and documents.

[1299] Support staff's terminal: The support staff understands the current situation based on the user's explanation and takes notes.

[1300] 5. Providing optimal prompts

[1301] Support staff device: Based on the user's work background, the system creates prompt ideas suitable for the generation AI and proposes them to the user.

[1302] User's device: The user receives the suggestions and asks questions or requests corrections as needed.

[1303] 6. Meeting Recording

[1304] Server: Stores meeting logs, chat history, shared files, and generated prompts in a database.

[1305] Device: Users can review recordings after the meeting and download them if necessary.

[1306] Specific examples

[1307] 1. User signs up and logs in

[1308] On the device: The user opens the web app, clicks the "Sign Up" button, enters their username, email address, and password, and clicks "Register."

[1309] Server: Stores the entered information in a database and sends a confirmation email to the user. When the user clicks on the link in the email, the login page is displayed.

[1310] 2. A user books a web meeting

[1311] On the device: The user goes to the "Book a Meeting" section of their dashboard, selects the desired date and time from the calendar, clicks the "Book" button and receives a confirmation message.

[1312] Server: Saves reservation information in a database and sends reservation notifications to support staff.

[1313] 3. The user and the support staff join a web meeting

[1314] On your device: The user opens the web app at the designated time and clicks the "Join Meeting" button. The support staff also joins.

[1315] Server: Generates secure web meeting links and sends reminder notifications.

[1316] 4. User explains the business background

[1317] Device: The user shares the documents and screens they use in their work and explains the specific work background and challenges to the support staff.

[1318] Support staff's device: Understand the job content and listen to the user's explanation while taking notes.

[1319] 5. Providing optimal prompts

[1320] Support staff terminal: Based on the user's explanation, it devises a prompt suitable for the generative AI and presents it to the user.

[1321] User's device: Accept suggestions and make any necessary corrections or questions.

[1322] 6. Meeting Recording

[1323] Server: After the meeting ends, the logs, chat history, shared files, and generated prompts are saved in a database.

[1324] Terminal: User can access the recording later and download it as needed.

[1325] In this way, the system helps users effectively utilize generative AI to improve business efficiency.

[1326] The processing flow will be explained below.

[1327] Sign up and log in

[1328] Step 1: View the sign-up page

[1329] Device: User visits web app and clicks "Sign up" button.

[1330] Server: Sends the sign-up page to the browser and displays it.

[1331] Step 2: Enter your sign-up information

[1332] On the device: The user fills in a form with their username, email address, password, and confirmation password.

[1333] Step 3: Submit your sign-up information

[1334] Terminal: The user clicks the "Register" button and sends the information to the server.

[1335] Step 4: Verify and save your sign-up information

[1336] Server: Checks the entered information for consistency and duplication, and if there are no problems, saves it to the database.

[1337] Server: Sends a confirmation email to the user.

[1338] Step 5: Display the login page

[1339] Server: After successful sign-up, display the login page to the user.

[1340] Step 6: Enter your login details

[1341] Device: The user enters their email address and password and clicks the "Login" button.

[1342] Step 7: Verify your login details

[1343] Server: The entered email address and password are verified against the database.

[1344] Step 8: View the Dashboard

[1345] Server: If authentication is successful, display the user's dashboard page.

[1346] Web Meeting Reservation

[1347] Step 1: View the booking page

[1348] On your device: The user accesses the "Book a Meeting" section of their dashboard.

[1349] Server: Sends data to display the reservation page.

[1350] Step 2: Select a reservation slot

[1351] Terminal: The user selects the desired date and time from the calendar.

[1352] Step 3: Confirm and submit your booking

[1353] Terminal: The user clicks the "Confirm Reservation" button and sends the information to the server.

[1354] Step 4: Save your reservation

[1355] Server: Saves the entered reservation information in a database and sends a reservation notification to the support staff.

[1356] Step 5: View the reservation confirmation message

[1357] Server: Display a confirmation message to the user that the reservation was successful.

[1358] Joining a web meeting

[1359] Step 1: Meeting start notification

[1360] Server: Send a reminder to users and support staff 10 minutes before the meeting start time.

[1361] Step 2: Generate a meetings link

[1362] Server: Generates and sends secure web meeting links to users and support personnel.

[1363] Step 3: Join the meeting

[1364] Terminal: The user and support staff click on the link sent to join the web meeting.

[1365] Step 4: Start Screen Sharing

[1366] Terminal: The user uses the screen sharing function to show the current work status to the support staff.

[1367] User's business background

[1368] Step 1: Oral explanation of the business background

[1369] Terminal: The user verbally explains the specific background and issues of the work to the support staff.

[1370] Step 2: Share materials

[1371] Terminal: The user provides relevant materials and documents to the support staff via screen sharing or file transfer.

[1372] Providing optimal prompts

[1373] Step 1: Create a Prompt

[1374] Support staff terminal: Generates optimal prompt suggestions based on the user's explanation.

[1375] Step 2: Propose a Prompt

[1376] Terminal: The support staff presents the proposed prompt to the user and explains it to them.

[1377] Step 3: Receive feedback and make corrections

[1378] On the device: The user provides feedback on the proposed prompt and requests revisions if necessary.

[1379] Support staff device: Receive feedback and revise the proposed prompt.

[1380] Meeting recording

[1381] Step 1: Save the meeting recording

[1382] Server: Stores meeting logs (chat history, shared files, generated prompts) in a database.

[1383] Step 2: Notification and Access to Records

[1384] Server: After the meeting ends, notify the user that the recording has been saved and provide an access link.

[1385] Step 3: Review and download the recording

[1386] On your device: Users can click the link provided to log in, view the meeting recording, and download it if necessary.

[1387] In this way, specific processing is carried out at each step, and the system provides assistance to users to make the most of generated AI and improve business efficiency.

[1388] Example 1

[1389] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1390] In today's work environment, users need appropriate support to efficiently utilize generative AI to perform their work. However, if users lack the specific background information or know-how to devise prompts, it can be difficult to maximize the effectiveness of generative AI. Furthermore, the process of sharing the status and progress of work with support staff in real time and receiving specific advice must be streamlined. Furthermore, it is also important to properly store meeting records so they can be referenced later.

[1391] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1392] In this invention, the server includes a means for a user to create and authenticate an account, a means for the user to select a date and time based on the user's work and schedule an online conference, and a means for a support staff member to participate in the online conference and generate optimal prompts based on the work materials shared by the user. This allows the user to receive advice from the appropriate support staff member and efficiently utilize the generation AI, significantly improving work efficiency. Furthermore, by properly storing online conference records and dialogue history, users can enjoy the benefit of making them easier to reference later.

[1393] "Creating an account and authenticating" refers to the process in which a user first registers (signs up) with the system and the system verifies the user's authenticity based on the information entered.

[1394] "Selecting a date and time to reserve an online meeting" refers to the process by which a user selects a specific date and time using the calendar function within the system and reserves the system to hold an online meeting at that date and time.

[1395] "Participating in an online meeting" refers to the process in which a user and a support staff member connect online using a web link generated at a specified date and time to hold a meeting.

[1396] "Generating optimal prompt sentences based on business materials" refers to the process in which a support staff member creates and provides effective prompts to the generation AI based on the business-related materials and information shared with the support staff by the user.

[1397] "Storing on a network server" refers to the process of securely storing online meeting records, conversation histories, shared data, etc. on a remote data storage system.

[1398] "Explaining the work background and sharing information on the screen in real time" refers to the process in which a user explains the details and background of the work while sharing the screen of their computer or device with a support staff member in real time during an online meeting.

[1399] "Storing online meeting logs, dialogue history, shared data, and generated prompts in a management system" refers to the process of storing all records and data related to online meetings in a dedicated management system for future reference.

[1400] This invention is a system designed to allow users to improve work efficiency by utilizing generated AI. Specific embodiments of the invention are described below.

[1401] overview

[1402] This system helps users create an account through a web application, schedule a business meeting, and participate in an online meeting, and then receive appropriate prompt advice from support staff, thereby improving work efficiency.

[1403] Hardware and Software Configuration

[1404] server

[1405] The server is a backend system for user authentication, conference booking, online conference link generation, and meeting record storage. Specifically, it has the following configuration:

[1406] Database: Use a database (e.g. PostgreSQL) to store user information, booking information, and meeting records.

[1407] Application server: Uses a backend framework (e.g., Node.js, Django) that provides user authentication and conference management functions.

[1408] Mail Server: Use the mail server to send account confirmation emails and meeting reminder notifications.

[1409] Terminal

[1410] The terminal is where users and support staff access and operate the web application. It consists of:

[1411] Browser: The user uses a web browser (e.g., Google Chrome, Firefox) to access the web application.

[1412] Screen sharing software: Use software (e.g., Zoom, Microsoft Teams) that allows users to share their screens with support personnel in real time.

[1413] Explanation of program processing

[1414] Sign up and log in

[1415] Device: The user accesses the web application using a browser, clicks the "Sign Up" button, enters their username, email address, and password, and clicks "Register." The server stores the entered information in a database and sends a confirmation email to the user. The user clicks the link in the email and their account is activated. The user then enters their email address and password on the login page and clicks "Login." The server performs authentication, and if successful, displays the dashboard to the user.

[1416] Examples:

[1417] A user accesses the web app in a browser, enters "testuser", "testuser@example.com", and "password123", and clicks the register button. The server stores this information in a database and sends a confirmation email to "testuser@example.com".

[1418] Web Meeting Reservation

[1419] Terminal: The user selects the desired date and time from the calendar in the dashboard and clicks the "Reserve" button. The server saves the reservation information in the database and sends a reservation notification to the support staff.

[1420] Examples:

[1421] The user selects "October 15, 2023, 3:00 PM" on the dashboard calendar and clicks the "Book" button. The server saves this information in the database and notifies the support staff by email.

[1422] Joining a web meeting

[1423] Terminal: When the meeting start time arrives, users and support staff click the provided link to join the web meeting. The server sends reminder notifications and generates and provides a secure web meeting link to participants.

[1424] Examples:

[1425] Five minutes before the meeting starts, the server sends a reminder to the user and support staff and provides them with a meeting link, which they click to join.

[1426] User's business background

[1427] Device: The user uses the screen sharing function to share the work situation and related documents with the support staff. The support staff understands the work situation based on the user's explanation and takes notes.

[1428] Examples:

[1429] The user clicks the screen sharing button in the browser and displays the Excel file used in work to the support staff.

[1430] Providing optimal prompts

[1431] Support staff terminal: The support staff will devise a prompt suitable for the generation AI based on the user's work background. The user will review the proposed prompt and make any necessary corrections or questions.

[1432] Specific prompt examples:

[1433] Suggest business prompts such as "Automatically generate invoices."

[1434] Meeting recording

[1435] Server: After the meeting, the logs, chat history, shared files, and generated prompts are stored in a database. Users can access the recordings later and download them as needed.

[1436] Examples:

[1437] The server stores meeting records, interaction history, and shared files in a database, which users can download by accessing the history section of their dashboard.

[1438] As described above, by using this system, users can effectively utilize generative AI to optimize their business processes. With the help of support staff, they can quickly create effective prompts, significantly improving work efficiency.

[1439] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1440] Step 1:

[1441] Create an account (sign up)

[1442] Device: The user visits the web application and enters their username, email address, and password into the sign-up form.

[1443] Specific behavior:

[1444] The user enters "testuser", "testuser@example.com", and "password123" and clicks the Register button.

[1445] Server: The server stores the entered username, email address, and password in a database and sends a confirmation email to the user. Based on the user's input, the server stores the information in a database and sends a confirmation email.

[1446] Input: Username, Email Address, Password

[1447] Data processing: Save user information in a database and request the mail server to send a confirmation email

[1448] Output: User information stored in the database, confirmation email sent to the user

[1449] Step 2:

[1450] Account authentication (login)

[1451] Device: After signing up, the user enters their email address and password on the login page and clicks "Login."

[1452] Specific behavior:

[1453] The user enters "testuser@example.com" and "password123" and clicks the login button.

[1454] Server: The server checks the input information against the database and performs authentication. If successful, it displays the dashboard to the user.

[1455] Input: Email address, Password

[1456] Data processing: Retrieving user information from the database and matching it with the input information

[1457] Output: Authentication result. If successful, the user's dashboard screen will be displayed.

[1458] Step 3:

[1459] Web Meeting Reservation

[1460] Terminal: The user selects the desired date and time using the calendar function in the dashboard and clicks the "Book" button.

[1461] Specific behavior:

[1462] The user selects "October 15, 2023, 3:00 PM" from the calendar and clicks the reservation button.

[1463] Server: The server saves the reservation information in a database and sends a reservation notification to the support staff. It saves the information to the database and notifies the staff based on the user's input.

[1464] Input: Reservation date and time

[1465] Data processing: Save the reservation information to the database and call the notification system to send reservation notifications to support staff.

[1466] Output: Booking information stored in the database, booking notification sent to support staff

[1467] Step 4:

[1468] Joining a web meeting

[1469] Server: Sends reminders to users and support staff before the meeting starts and generates a secure web meeting link.

[1470] Input: Reservation information

[1471] Data processing: generating and sending reminder notifications, generating web meeting links

[1472] Output: Reminder sent, generated web meeting link

[1473] Terminal: Users and support staff click on the provided link to join the web meeting.

[1474] Specific behavior:

[1475] Five minutes before the meeting starts, the user and support staff will check the notification and click the provided link to join.

[1476] Step 5:

[1477] Business background explanation

[1478] Device: Users can use the screen sharing function to share work status and related documents with support staff.

[1479] Specific behavior:

[1480] The user clicks the screen sharing button in the browser and displays an Excel file used for work.

[1481] Support staff's device: The support staff understands the work situation based on the information shared on the screen and the user's explanation, and takes notes.

[1482] Input: Screen sharing information, user's verbal description

[1483] Data processing: Support staff understanding and recording

[1484] Output: Support personnel notes

[1485] Step 6:

[1486] Providing optimal prompts

[1487] Support staff terminal: The support staff will devise a prompt suitable for the generation AI based on the user's work background and present it to the user.

[1488] Input: User's business background information

[1489] Data processing: Prompt generation

[1490] Output: Suggested prompt text

[1491] Example prompt:

[1492] Suggest business prompts such as "Automatically generate invoices."

[1493] User's device: The user reviews the suggested prompt and makes any necessary corrections or questions.

[1494] Specific behavior:

[1495] The user reviews the prompts from the support staff and, if necessary, requests to "change the date format."

[1496] Step 7:

[1497] Meeting recording

[1498] Server: After the meeting ends, the log, chat history, shared files, and generated prompts are saved in a database.

[1499] Input: Meeting logs, chat history, shared files, generated prompts

[1500] Data processing: Record preservation processing

[1501] Output: Records stored in a database

[1502] Terminal: User can access the recording later and download it if needed.

[1503] Specific behavior:

[1504] After a meeting ends, users can access the History section of their dashboard and download a specific meeting recording.

[1505] (Application example 1)

[1506] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1507] To improve efficiency and productivity in food delivery operations, there is a need for technology that allows users to collaborate with support staff in real time and provides optimal routes using generative AI models. However, conventional systems have issues with being unable to efficiently perform a series of tasks, such as web meetings, recording work, and providing optimal routes.

[1508] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1509] In this invention, the server includes a means for users to sign up and log in, a means for users to book web meetings based on their tasks, a means for support staff to participate in the web meetings and provide optimal prompts based on the user's work background, a means for saving meeting records on the server, a means for users to manage delivery task reservations, a means for providing optimal route suggestions using a generative AI model, and a means for inputting feedback after the task is completed. This significantly improves the efficiency and productivity of food delivery tasks, and enables users to effectively use generative AI models to optimize their tasks.

[1510] "User" means a person who uses the system to improve business efficiency.

[1511] "Signing up" is the process of creating a new account.

[1512] "Login" means the act of authenticating to access an existing account and use the system.

[1513] A "web meeting" is an online conference held over the Internet.

[1514] "Reservation" is the act of setting up a web meeting for a specific date and time.

[1515] A "support staff member" is someone whose role is to support users in optimizing their operations.

[1516] A "prompt" is an input sentence that gives instructions to a generative AI model.

[1517] "Recording" refers to saving the progress and content of a meeting in a format that can be referenced later.

[1518] A "server" is a computer system that stores and processes data.

[1519] "Delivery services" refers to the general business of delivering goods or services to designated locations.

[1520] A "generative AI model" is an artificial intelligence algorithm that makes inferences and predictions based on data.

[1521] A "route proposal" is a plan that suggests an efficient route for delivery work.

[1522] "Feedback" is the act of recording evaluations and opinions after a task is completed.

[1523] This invention is a system that uses generative AI models to help improve the efficiency of food delivery operations. The system involves a series of processes in which users sign up and log in, book web meetings, and share their work status with support staff in real time.

[1524] 1. Sign up and login

[1525] The server stores the user's input information in a database and performs authentication. For example, when signing up, the user enters their username, email address, and password, and after receiving a verification email, clicks the link to log in.

[1526] 2. Delivery business reservation management

[1527] The user's device is used to manage delivery schedules. The server uses the Google Calendar API to store reservation information in a cloud database and notify support staff.

[1528] 3. Participating in a web meeting

[1529] Using WebRTC, the user and support staff join a web meeting at the specified time. The server sends a reminder notification and generates a secure meeting link.

[1530] 4. Explanation of business background and screen sharing

[1531] Users can use the screen sharing function to show support staff the documents and progress they are using in their work, making it easier for support staff to understand the user's work background.

[1532] 5. Providing optimal route suggestions

[1533] The support staff terminal uses a generative AI model based on the user's work background to provide optimal route suggestions. For example, the generative AI can suggest optimal routes using prompts such as: "Please suggest the optimal route based on the delivery address and customer address."

[1534] 6. Feedback after the job is completed

[1535] After completing a delivery task, users can enter feedback using their devices. This feedback is stored on the server and can be used to improve future operations.

[1536] 7. Meeting Recording

[1537] The server stores meeting logs, chat history, shared files, and generated prompts in a database such as Firebase Firestore, allowing for future review and continuous improvement.

[1538] For example, a user who wants to streamline delivery operations during lunchtime can enter the following: "We have a high volume of deliveries during lunchtime. Please suggest the best route and most efficient delivery method."

[1539] Generative AI analyzes data to create optimal route plans in real time and notify users' smartphones, thereby significantly improving the efficiency of food delivery operations.

[1540] The system can be realized using specific hardware and software such as Firebase Authentication, Google Calendar API, WebRTC, OpenAI GPT-3, and Firebase Firestore.

[1541] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1542] Step 1:

[1543] A user signs up.

[1544] Input: Username, Email Address, Password

[1545] Data processing: The terminal collects the user's input information and sends it to the server.

[1546] Output: The server stores the information in a database and sends a verification email.

[1547] How it works: A user opens the web app, enters the required information, and clicks "Register." The server receives the information, writes it to the database, and then sends a verification email.

[1548] Step 2:

[1549] The user logs in.

[1550] Input: Email address, password

[1551] Data processing: The server validates and authenticates the input information.

[1552] Output: Login success message and transition to the dashboard screen

[1553] What happens: The user enters their email address and password on the login page and clicks "Login." The server verifies the information and, if correct, displays the dashboard screen.

[1554] Step 3:

[1555] A user books a delivery job.

[1556] Input: Scheduled delivery date and time, detailed information

[1557] Data processing: The server saves the reservation information and updates the calendar using the Google Calendar API.

[1558] Output: Reservation confirmation and notification to support staff

[1559] What it does: The user selects the desired date and time on the dashboard and clicks "Book." The server saves the reservation information, updates the calendar using the Google Calendar API, and notifies the support staff.

[1560] Step 4:

[1561] A user joins a web meeting.

[1562] Input: Meeting time, link

[1563] Data processing: The server sends reminder notifications and generates secure meeting links over WebRTC.

[1564] Output: Reminders, Meeting Links

[1565] Specific operation: When the meeting start time arrives, the user and support staff click the designated link to join the web meeting. The server sends a notification and generates and distributes the link.

[1566] Step 5:

[1567] The user explains the background of the work to the support staff and shares the screen.

[1568] Input: Business documents, progress status

[1569] Data processing: The device shares the screen and communicates the work status to the support staff.

[1570] Output: Shared work documents, progress explanation

[1571] Specific operation: The user uses the screen sharing function to show the support staff the materials and progress, and the support staff deepens their understanding and takes notes.

[1572] Step 6:

[1573] Support staff use generative AI models to provide optimal route suggestions.

[1574] Input: User's business background information, delivery route

[1575] Data processing: A generative AI model calculates and suggests efficient routes.

[1576] Output: Optimal route plan

[1577] Specific operation: The support staff inputs prompts into the AI ​​generator based on the information obtained from the user and suggests the optimal route. Example prompt: "Please suggest the optimal route based on the delivery address and customer address."

[1578] Step 7:

[1579] The user enters feedback after completing the delivery job.

[1580] Input: Feedback

[1581] Data processing: The server stores the feedback information.

[1582] Output: Saved feedback data

[1583] What it does: A user enters and submits feedback using the web app. The server stores the information in a database.

[1584] Step 8:

[1585] The server stores the meeting records.

[1586] Input: Meeting logs, chat history, shared files, generated prompt ideas

[1587] Data processing: The server stores the data in a database.

[1588] Output: Saved meeting data

[1589] Specific operation: After the meeting ends, the server saves the logs, chat history, shared files, and generated prompt ideas in a database such as Firebase Firestore.

[1590] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1591] This invention is a system that provides support to users to improve work efficiency using generative AI, and aims to provide more effective support by combining it with an emotion engine that recognizes and responds to user emotions. The system involves a series of processes in which users sign up and log in, book a web meeting, explain the work background through screen sharing with a support staff member, and receive optimal prompt suggestions. In addition, the emotion engine collects user emotion data, adjusts prompt suggestions based on that data, and stores records.

[1592] Program processing overview

[1593] 1. Sign up and login

[1594] Device: The user accesses the web app and creates an account (signs up) or logs in with an existing account.

[1595] Server: Stores the information entered by the user in a database, performs authentication, and sends a confirmation email to the user once sign-up is complete.

[1596] 2. Schedule a web meeting

[1597] Terminal: Users can see available slots on their dashboard and select the desired date and time for the meeting.

[1598] Server: Saves reservation information in a database and notifies support staff.

[1599] 3. Participating in a web meeting

[1600] Terminal: When the meeting start time arrives, the user and support staff click on the provided link to join the web meeting.

[1601] Server: Sends reminders and generates and provides meeting links to participants.

[1602] 4. Explanation of the user's business background

[1603] Terminal: The user can use the screen sharing function to display the current status of the work to the support staff, provide verbal explanations, and share relevant materials and documents.

[1604] Support staff's terminal: The support staff understands the current situation based on the user's explanation and takes notes.

[1605] 5. Operation of the Emotion Engine

[1606] Device: Uses a camera and microphone to analyze the user's facial expressions and voice, and collects emotional data from them.

[1607] Server: The emotion engine processes the collected data in real time and recognizes the user's emotions.

[1608] 6. Providing optimal prompts

[1609] Support staff device: Based on the user's work background and emotional data, the device creates prompt ideas suitable for the generating AI and proposes them to the user.

[1610] User's device: The user receives the suggestions and asks questions or requests corrections as needed.

[1611] 7. Meeting Recording

[1612] Server: Stores meeting logs, chat history, shared files, sentiment data, and generated prompt ideas in a database.

[1613] Device: Users can review recordings after the meeting and download them if necessary.

[1614] Specific examples

[1615] 1. User signs up and logs in

[1616] On the device: The user opens the web app, clicks the "Sign Up" button, enters their username, email address, and password, and clicks "Register."

[1617] Server: Stores the entered information in a database and sends a confirmation email to the user. When the user clicks on the link in the email, the login page is displayed.

[1618] 2. A user books a web meeting

[1619] On the device: The user goes to the "Book a Meeting" section of their dashboard, selects the desired date and time from the calendar, clicks the "Book" button and receives a confirmation message.

[1620] Server: Saves reservation information in a database and sends reservation notifications to support staff.

[1621] 3. The user and the support staff join a web meeting

[1622] On your device: The user opens the web app at the designated time and clicks the "Join Meeting" button. The support staff also joins.

[1623] Server: Generates secure web meeting links and sends reminder notifications.

[1624] 4. User explains the business background

[1625] Device: The user shares the documents and screens they use in their work and explains the specific work background and challenges to the support staff.

[1626] Support staff's device: Understand the job content and listen to the user's explanation while taking notes.

[1627] 5. Operation of the Emotion Engine

[1628] Device: Captures the user's face with a camera, picks up their voice with a microphone, and collects emotional data in real time.

[1629] Server: Analyzes the collected data and determines the user's emotional state.

[1630] 6. Providing optimal prompts

[1631] Support staff device: Taking into account the user's work background and emotional state, the system creates prompt proposals suitable for the generative AI and presents them to the user.

[1632] User's device: The user accepts the suggestions and makes any necessary corrections or questions.

[1633] 7. Meeting Recording

[1634] Server: After the meeting ends, the logs, chat history, shared files, emotion data, and generated prompts are stored in a database.

[1635] On-device: Users can review saved records later and download them as needed.

[1636] In this way, the system helps users maximize the use of generative AI and emotion recognition to improve work efficiency.

[1637] The processing flow will be explained below.

[1638] Sign up and log in

[1639] Step 1: View the sign-up page

[1640] Device: User visits web app and clicks "Sign up" button.

[1641] Server: Sends the sign-up page to the browser and displays it.

[1642] Step 2: Enter your sign-up information

[1643] On the device: The user fills in a form with their username, email address, password, and confirmation password.

[1644] Step 3: Submit your sign-up information

[1645] Terminal: The user clicks the "Register" button and sends the information to the server.

[1646] Step 4: Verify and save your sign-up information

[1647] Server: Checks the entered information for consistency and duplication, and if there are no problems, saves it to the database.

[1648] Server: Sends a confirmation email to the user.

[1649] Step 5: Display the login page

[1650] Server: After successful sign-up, display the login page to the user.

[1651] Step 6: Enter your login details

[1652] Device: The user enters their email address and password and clicks the "Login" button.

[1653] Step 7: Verify your login details

[1654] Server: The entered email address and password are verified against the database.

[1655] Step 8: View the Dashboard

[1656] Server: If authentication is successful, display the user's dashboard page.

[1657] Web Meeting Reservation

[1658] Step 1: View the booking page

[1659] On your device: The user accesses the "Book a Meeting" section of their dashboard.

[1660] Server: Sends data to display the reservation page.

[1661] Step 2: Select a reservation slot

[1662] Terminal: The user selects the desired date and time from the calendar.

[1663] Step 3: Confirm and submit your booking

[1664] Terminal: The user clicks the "Confirm Reservation" button and sends the information to the server.

[1665] Step 4: Save your reservation

[1666] Server: Saves the entered reservation information in a database and sends a reservation notification to the support staff.

[1667] Step 5: View the reservation confirmation message

[1668] Server: Display a confirmation message to the user that the reservation was successful.

[1669] Joining a web meeting

[1670] Step 1: Meeting start notification

[1671] Server: Send a reminder to users and support staff 10 minutes before the meeting start time.

[1672] Step 2: Generate a meetings link

[1673] Server: Generates and sends secure web meeting links to users and support personnel.

[1674] Step 3: Join the meeting

[1675] Terminal: The user and support staff click on the link sent to join the web meeting.

[1676] Step 4: Start Screen Sharing

[1677] Terminal: The user uses the screen sharing function to show the current work status to the support staff.

[1678] User's business background

[1679] Step 1: Oral explanation of the business background

[1680] Terminal: The user verbally explains the specific background and issues of the work to the support staff.

[1681] Step 2: Share materials

[1682] Terminal: The user provides relevant materials and documents to the support staff via screen sharing or file transfer.

[1683] Emotion Engine Operation

[1684] Step 1: Collecting emotion data

[1685] Device: Emotional data is collected by taking a picture of the user's face with a camera and picking up their voice with a microphone.

[1686] Step 2: Analyze the emotion data

[1687] Server: Analyzes collected data in real time and determines the user's emotional state.

[1688] Providing optimal prompts

[1689] Step 1: Create a Prompt

[1690] Support staff device: Based on the user's work background and emotional data, the device creates a prompt suitable for the generating AI and presents it to the user.

[1691] Step 2: Propose a Prompt

[1692] Terminal: The support staff presents the proposed prompt to the user and explains it to them.

[1693] Step 3: Receive feedback and make corrections

[1694] On the device: The user provides feedback on the proposed prompt and requests revisions if necessary.

[1695] Support staff device: Receive feedback and revise the proposed prompt.

[1696] Meeting recording

[1697] Step 1: Save the meeting recording

[1698] Server: Stores meeting logs (conversation records, screen sharing content, emotion data, and generated prompts) in a database.

[1699] Step 2: Notification and Access to Records

[1700] Server: After the meeting ends, notify the user that the recording has been saved and provide an access link.

[1701] Step 3: Review and download the recording

[1702] On your device: Users can click the link provided to log in, view the meeting recording, and download it if necessary.

[1703] In this way, specific actions are performed at each processing step, and the system helps users maximize the use of generative AI and emotion recognition to improve work efficiency.

[1704] Example 2

[1705] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1706] Conventional business support systems have difficulty providing optimal instructions based on the user's work context, and do not take the user's emotions into consideration when providing support. As a result, there are problems with user satisfaction and work efficiency declining.

[1707] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1708] In this invention, the server includes a means for a user to sign up and log in, a means for a user to reserve an online meeting based on the user's work, a means for a support person to participate in the online meeting and provide optimal instructions based on the user's work background, a means for analyzing the user's facial expressions and voice and collecting emotional data, a means for processing the collected emotional data in real time and recognizing the user's emotions, and a means for saving a record of the meeting on the server. This makes it possible to provide appropriate instructions according to the user's work background and support that takes the user's emotions into consideration.

[1709] "Signing up" is the process by which a user creates a new account and registers with the system.

[1710] "Logging in" is the process of accessing a system using existing account information.

[1711] An "online meeting" is a conference held over the Internet, allowing participants to communicate in real time.

[1712] "Support Personnel" means personnel who provide technical and business support to Users.

[1713] "Instructions" are texts created using generative AI models that provide specific suggestions or guidelines to help users with their work.

[1714] "Facial Expression" refers to the muscle movements and other physical characteristics of a user's face that indicate an emotional state.

[1715] "Voice" refers to the voice produced by the user, and is data used to determine the user's emotional state using the tone, pitch, volume, etc.

[1716] "Emotional data" is information that expresses the user's emotional state as numerical or categorical data.

[1717] "Real-time processing" means processing data immediately after it is generated, providing results with little to no delay.

[1718] A "meeting recording" is a series of records containing digital data, such as conversations, chats, and shared files, that take place during an online meeting.

[1719] This invention is a support system that utilizes a generative AI model to help users improve their work efficiency, and also combines it with an emotion engine that recognizes and responds to user emotions.

[1720] The system includes the following means:

[1721] 1. Sign-up and login methods

[1722] Device: A user visits a web app using a JavaScript framework to sign up and log in. Specifically, the user enters a username, email address, and password, and submits the sign-up form.

[1723] Server: Build a backend API using Node.js and Express, store this information in a MySQL database, and use an SMTP server to send confirmation emails, for example, sendgrid.

[1724] 2. Online Meeting Reservation Methods

[1725] On the device: Users access a dashboard powered by React.js and use a calendar component such as FullCalendar.js to book online meetings.

[1726] Server: Reservation information is stored in a MySQL database, and the server sends reservation confirmation emails to users and support staff.

[1727] 3. How to join an online meeting

[1728] Device: Users and support staff log in to the web app at the specified date and time and join through the API of online meeting tools such as Google Meet and Zoom.

[1729] Server: Sends reminders near the scheduled time and generates a secure meeting link.

[1730] 4. Means of explaining the business background

[1731] Device: During online meetings, users can use Zoom's screen sharing function to explain the current work background and related materials to support staff.

[1732] Support staff's device: The support staff listens to the user's explanation and takes notes of important points using a note-taking app (e.g., OneNote) on their PC.

[1733] 5. Emotion Engine Operation Method

[1734] Device: The user's face is photographed with a webcam and their voice is collected with a microphone. A facial expression recognition model and voice analysis tool using TensorFlow.js are used.

[1735] Server: Uses a Python sentiment analysis library (e.g., emotion_recognition) to process the collected data in real time and recognize user emotions.

[1736] 6. Optimal Means of Providing Instructions

[1737] Support staff terminal: The support staff uses a generative AI model (e.g., Chat GPT) to generate instructions based on the user's work background and emotional data.

[1738] User's device: Provide the generated instructions to the user, correcting or asking questions as needed.

[1739] Examples of prompts:

[1740] "I want to use a generative AI model to generate optimal prompt suggestions. My user is preparing a presentation for a new marketing campaign and is particularly short on ideas for slide design. He's feeling stressed. What would be the best prompt?"

[1741] 7. Meeting Record Keeping Method

[1742] Server: After the online meeting ends, the logs, communication history, shared files, emotional data, and generated instructions are stored in a MySQL database.

[1743] On your device: Users can view saved recordings in the "Meeting History" section of the web app and download them if necessary.

[1744] In this way, the present invention provides specific assistance to users to make the most of generative AI models and emotion recognition to improve work efficiency.

[1745] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1746] Step 1:

[1747] Sign up and log in

[1748] On the device: The user opens the web app and clicks the "Sign up" button. The user enters their username, email address, and password and clicks "Register."

[1749] Enter your username, email address, and password.

[1750] Output: Signup request data.

[1751] Server: Receives the sign-up request data, stores the entered information in a MySQL database, and then sends a confirmation email using an SMTP server.

[1752] Input: Signup request data.

[1753] Output: Database saved results and confirmation email.

[1754] On the device: When the user clicks on the link in the confirmation email, they are authenticated and directed to a login page.

[1755] Enter: Click on the confirmation link.

[1756] Output: Login page displayed.

[1757] Step 2:

[1758] Schedule an online meeting

[1759] On the device: The user opens the dashboard, selects the desired date and time from the calendar, and clicks the "Book" button.

[1760] Input: Desired date and time.

[1761] Output: Booking request data.

[1762] Server: Receives the reservation request data, stores the reservation information in a MySQL database, and then sends a reservation confirmation email to the user and support staff.

[1763] Input: Booking request data.

[1764] Output: Database saved results and confirmation email.

[1765] Step 3:

[1766] Joining an online meeting

[1767] Device: The user and support staff open the web app at the designated time and click the "Join Meeting" button.

[1768] Input: Meeting time, Join button click.

[1769] Output: Meeting join request.

[1770] Server: Sends reminders before the specified time and generates a secure meeting link using Google Meet or Zoom APIs.

[1771] Input: Meeting join request.

[1772] Output: Reminder and meeting link.

[1773] Step 4:

[1774] Business background explanation

[1775] Terminal: The user uses the screen sharing function to show the support staff the current task and related materials, and provides verbal explanations.

[1776] Input: Start screen sharing, business documents.

[1777] Output: Screen sharing data.

[1778] Support staff's device: The support staff listens to the user's explanation and takes notes of important points using a note-taking app on their PC.

[1779] Input: Screen share data.

[1780] Output: Note data.

[1781] Step 5:

[1782] Emotion Engine Operation

[1783] Device: The user's face is photographed with a webcam and their voice is collected with a microphone. TensorFlow.js is used to capture facial expression and voice data in real time.

[1784] Input: Video and audio data.

[1785] Output: facial expression data, voice data.

[1786] Server: Sends the collected data to a Python sentiment analysis library to recognize the user's emotional state.

[1787] Input: facial expression data, voice data.

[1788] Output: Emotional state data.

[1789] Step 6:

[1790] Providing optimal instructions

[1791] Support staff device: Using the user's work context and emotional data, prompts are input into a generative AI model (e.g., Chat GPT) to generate optimal instructions.

[1792] Input: Work background data, emotional state data.

[1793] Output: The generated directive.

[1794] User's device: Provide the generated instructions to the user. If the user is not satisfied with the content, they can request corrections or ask questions.

[1795] Input: The generated instruction.

[1796] Output: User feedback.

[1797] Step 7:

[1798] Meeting Recording

[1799] Server: After the meeting ends, the logs, communication history, shared files, emotional data, and generated instructions are saved in a MySQL database.

[1800] Input: Meeting logs, communication history, shared files, emotion data, generated instructions.

[1801] Output: Database save results.

[1802] On your device: Users can view and optionally download recordings from the "Meeting History" section of the web app.

[1803] Input: User's record confirmation request.

[1804] Output: Meeting recording data.

[1805] (Application example 2)

[1806] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1807] Conventional business support systems lack the ability to utilize generative AI to provide prompts to users so that they can receive efficient support in their work. Furthermore, they lack the ability to provide flexible support based on the user's emotional state, which can increase stress and reduce work efficiency. Furthermore, they lack a system for monitoring the operating status of factory robots in real time and quickly detect and respond to abnormalities, which can lead to production line stagnation.

[1808] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1809] In this invention, the server includes a means for users to sign up and log in, a means for users to reserve web meetings based on their work, a means for support staff to participate in the web meetings and provide optimal prompts based on the user's work background, a means for saving meeting records on the server, a means for collecting user emotion data using an emotion engine and adjusting prompt proposals based on that data, a means for monitoring the operating status of factory robots, and a means for sending notifications when an abnormality is detected. This allows users to use generative AI and emotion recognition to improve work efficiency, and enables them to monitor the operating status of factory robots in real time to quickly detect and respond to abnormalities.

[1810] "User" means an individual or corporation that uses the system to receive business support.

[1811] "Sign-up and login" refers to the process by which a user creates an account and accesses the system with an existing account.

[1812] A "web meeting" is a meeting conducted over the Internet between a user and a support staff member using screen sharing and audio / video calls.

[1813] A "prompt" is a specific instruction or suggestion that a generative AI model provides to a user.

[1814] A "support staff member" is a person who provides business support and advice to users.

[1815] "Server" means a computer system that stores and manages user data and meeting records.

[1816] An "emotion engine" is software that analyzes a user's facial expressions and voice to recognize their emotional state.

[1817] "Emotional Data" refers to information about a user's emotions collected and analyzed by the Emotion Engine.

[1818] A "factory robot" is a robot used to perform automated tasks in a factory.

[1819] "Operation status" refers to the status of a factory robot, indicating whether it is operating normally or whether an error or abnormality has occurred.

[1820] "Anomaly detection" is the process of detecting abnormalities when they occur in the operation of factory robots.

[1821] A "notification" is a message sent by the system to notify the user of an abnormality or event.

[1822] A "log" is a detailed record of web meetings and activity within the system.

[1823] "Chat history" means a record of text communications that take place during a web meeting.

[1824] "Shared files" refers to documents and materials shared between the user and the support staff during a web meeting.

[1825] This invention provides a support system for improving business efficiency by combining generative AI and an emotion recognition engine. In particular, it can monitor the operating status of factory robots in real time and quickly detect and respond to abnormalities. Below, we will explain in detail the embodiments of this system.

[1826] This support system consists of the following means:

[1827] 1. How users sign up and log in

[1828] Device: The user uses the smartphone app to create an account (sign up) or log in with an existing account.

[1829] Server: Store the user's input information in a database and use a service such as Firebase Authentication for authentication.

[1830] 2. A way for users to schedule business-based web meetings

[1831] On the device: Users can check available slots on the app dashboard and book a web meeting at the desired date and time.

[1832] Server: Saves user reservation information in a database and notifies support staff using the Google Calendar API, etc.

[1833] 3. A way for support staff to join a web meeting and provide the most appropriate prompts based on the user's work context

[1834] Device: When the web meeting starts, the user and support staff click the specified link to join. The screen sharing function is also used so that the user can explain the background of the work and the support staff can take notes.

[1835] Server: Generates secure web meeting links and sends reminder notifications.

[1836] 4. Using an emotion engine to gather user sentiment data and tailor prompts based on that data

[1837] Device: The camera captures the user's face, the microphone picks up their voice, and collects emotional data in real time.

[1838] Server: Analyzes emotional data using Microsoft Azure Emotion API and other tools to determine the user's emotional state.

[1839] 5. A means of monitoring the operation status of factory robots

[1840] Terminal: Factory workers can check the status of running robots on a dashboard and select the robot with the problem.

[1841] Server: Updates the robot's operating status data in real time and sends notifications if an abnormality is detected.

[1842] 6. How to send notifications when an anomaly is detected

[1843] Server: When an abnormality is detected, a notification is sent immediately and an alert is issued to the user, which encourages a prompt response.

[1844] 7. How to store meeting records on a server

[1845] Server: Use a database service such as Firebase Firestore to store meeting logs, chat history, shared files, and generated prompt ideas in a database.

[1846] On your device: Users can review the recording after the meeting and download it if needed.

[1847] This system allows users to make the most of generative AI and emotion recognition to improve work efficiency, while also enabling them to monitor the operating status of factory robots in real time and quickly detect and respond to abnormalities.

[1848] Specific examples

[1849] Users sign up and log in

[1850] Enter your username, email address, and password, then click "Register."

[1851] A confirmation email will be sent to you and a link will take you to the login page.

[1852] Prompt Sentence Examples

[1853] "User's work context: Monitoring the operation status of a robot in a factory. User's emotional state: Angry (due to frequent errors). Desired output from the generative AI model: Suggest specific steps to improve work efficiency and quickly resolve issues."

[1854] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1855] Specific flow of processing steps

[1856] Step 1:

[1857] User sign-up and login

[1858] Input: A user opens the smartphone app and enters their username, email address, and password.

[1859] Processing: The device sends the entered information to Firebase Authentication, and the server performs authentication. If this is a new registration, the server sends a confirmation email to the user and waits for them to click the confirmation link.

[1860] Output: The server generates an authentication token and displays the user's target page.

[1861] What happens: The user receives a confirmation email and clicks on the link to authenticate their account. After successfully logging in, the dashboard will be displayed.

[1862] Step 2:

[1863] Users book web meetings

[1864] Input: User sees available slots on the app dashboard and selects their preferred time slot.

[1865] Processing: The device sends the reservation information to the server via the Google Calendar API. The server saves the reservation information in the database and notifies the support staff.

[1866] Output: A notification of the meeting reservation will be sent to the support staff. The reservation information will be added to Google Calendar.

[1867] Specific operation: The user selects the desired time and clicks the "Book" button. The server displays a confirmation message and adds the appointment to the support staff's calendar.

[1868] Step 3:

[1869] The user and the support staff join a web meeting

[1870] Input: When the web meeting starts, the user and support staff will click the specified link on their devices.

[1871] Processing: The server generates a secure web meeting link and sends a reminder. The user and support staff use screen sharing to explain the work background and take notes.

[1872] Output: The web meeting starts and both parties are connected correctly.

[1873] Specific operation: The server generates a meeting link, and the user and support staff join. The user shares materials, and the support staff takes notes while receiving explanations.

[1874] Step 4:

[1875] Emotional data collection and analysis using an emotion engine

[1876] Input: User's facial video and audio data are collected via camera and microphone.

[1877] Processing: The device sends the collected data to the Microsoft Azure Emotion API, where the server analyzes it and determines the user's emotional state.

[1878] Output: The user's emotional data is updated in real time and displayed to the support staff.

[1879] Specific operation: Data is collected in real time using a camera and microphone, and an emotion engine analyzes the user's state of joy, anger, sadness, etc.

[1880] Step 5:

[1881] Providing optimal prompts

[1882] Input: The support staff's device acquires the user's work background and emotional data.

[1883] Processing: Based on this data, the support staff sends the optimal prompt proposal to the generative AI model (e.g., OpenAI API). The generative AI model generates an appropriate prompt proposal and returns it to the support staff's device.

[1884] Output: The generated prompt proposal is displayed on the support staff's terminal. A suggestion is made to the user.

[1885] Specific operation: The support staff requests the AI ​​to generate prompts that best suit the user's situation, and then explains the generated prompts to the user.

[1886] Step 6:

[1887] Meeting recording

[1888] Input: Meeting logs, chat history, shared files, and generated prompt ideas are sent to the server.

[1889] Processing: The server stores this data in a database such as Firebase Firestore.

[1890] Output: Records are stored in a database for users to review and download at a later time.

[1891] Specific behavior: After the meeting ends, the server automatically saves logs and other records, allowing users to view details of past meetings.

[1892] In this way, the system helps users maximize the use of generative AI and emotion recognition to improve work efficiency.

[1893] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1894] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1895] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1896] [Fourth embodiment]

[1897] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1898] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1899] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1900] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1901] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1902] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1903] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1904] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1905] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1906] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1907] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1908] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1909] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1910] This invention is a system that provides support to users to improve work efficiency by utilizing generative AI. The system involves a series of processes in which users sign up and log in, book a web meeting, and receive optimal prompt suggestions through screen sharing with a support staff member.

[1911] Program processing overview

[1912] 1. Sign up and login

[1913] Device: The user accesses the web app and creates an account (signs up) or logs in with an existing account.

[1914] Server: Stores the information entered by the user in a database, performs authentication, and sends a confirmation email to the user once sign-up is complete.

[1915] 2. Schedule a web meeting

[1916] Terminal: Users can see available slots on their dashboard and select the desired date and time for the meeting.

[1917] Server: Saves reservation information in a database and notifies support staff.

[1918] 3. Participating in a web meeting

[1919] Terminal: When the meeting start time arrives, the user and support staff click on the provided link to join the web meeting.

[1920] Server: Sends reminders and generates and provides meeting links to participants.

[1921] 4. Explanation of the user's business background

[1922] Terminal: The user can use the screen sharing function to display the current status of the work to the support staff, provide verbal explanations, and share relevant materials and documents.

[1923] Support staff's terminal: The support staff understands the current situation based on the user's explanation and takes notes.

[1924] 5. Providing optimal prompts

[1925] Support staff device: Based on the user's work background, the system creates prompt ideas suitable for the generation AI and proposes them to the user.

[1926] User's device: The user receives the suggestions and asks questions or requests corrections as needed.

[1927] 6. Meeting Recording

[1928] Server: Stores meeting logs, chat history, shared files, and generated prompts in a database.

[1929] Device: Users can review recordings after the meeting and download them if necessary.

[1930] Specific examples

[1931] 1. User signs up and logs in

[1932] On the device: The user opens the web app, clicks the "Sign Up" button, enters their username, email address, and password, and clicks "Register."

[1933] Server: Stores the entered information in a database and sends a confirmation email to the user. When the user clicks on the link in the email, the login page is displayed.

[1934] 2. A user books a web meeting

[1935] On the device: The user goes to the "Book a Meeting" section of their dashboard, selects the desired date and time from the calendar, clicks the "Book" button and receives a confirmation message.

[1936] Server: Saves reservation information in a database and sends reservation notifications to support staff.

[1937] 3. The user and the support staff join a web meeting

[1938] On your device: The user opens the web app at the designated time and clicks the "Join Meeting" button. The support staff also joins.

[1939] Server: Generates secure web meeting links and sends reminder notifications.

[1940] 4. User explains the business background

[1941] Device: The user shares the documents and screens they use in their work and explains the specific work background and challenges to the support staff.

[1942] Support staff's device: Understand the job content and listen to the user's explanation while taking notes.

[1943] 5. Providing optimal prompts

[1944] Support staff terminal: Based on the user's explanation, it devises a prompt suitable for the generative AI and presents it to the user.

[1945] User's device: Accept suggestions and make any necessary corrections or questions.

[1946] 6. Meeting Recording

[1947] Server: After the meeting ends, the logs, chat history, shared files, and generated prompts are saved in a database.

[1948] Terminal: User can access the recording later and download it as needed.

[1949] In this way, the system helps users effectively utilize generative AI to improve business efficiency.

[1950] The processing flow will be explained below.

[1951] Sign up and log in

[1952] Step 1: View the sign-up page

[1953] Device: User visits web app and clicks "Sign up" button.

[1954] Server: Sends the sign-up page to the browser and displays it.

[1955] Step 2: Enter your sign-up information

[1956] On the device: The user fills in a form with their username, email address, password, and confirmation password.

[1957] Step 3: Submit your sign-up information

[1958] Terminal: The user clicks the "Register" button and sends the information to the server.

[1959] Step 4: Verify and save your sign-up information

[1960] Server: Checks the entered information for consistency and duplication, and if there are no problems, saves it to the database.

[1961] Server: Sends a confirmation email to the user.

[1962] Step 5: Display the login page

[1963] Server: After successful sign-up, display the login page to the user.

[1964] Step 6: Enter your login details

[1965] Device: The user enters their email address and password and clicks the "Login" button.

[1966] Step 7: Verify your login details

[1967] Server: The entered email address and password are verified against the database.

[1968] Step 8: View the Dashboard

[1969] Server: If authentication is successful, display the user's dashboard page.

[1970] Web Meeting Reservation

[1971] Step 1: View the booking page

[1972] On your device: The user accesses the "Book a Meeting" section of their dashboard.

[1973] Server: Sends data to display the reservation page.

[1974] Step 2: Select a reservation slot

[1975] Terminal: The user selects the desired date and time from the calendar.

[1976] Step 3: Confirm and submit your booking

[1977] Terminal: The user clicks the "Confirm Reservation" button and sends the information to the server.

[1978] Step 4: Save your reservation

[1979] Server: Saves the entered reservation information in a database and sends a reservation notification to the support staff.

[1980] Step 5: View the reservation confirmation message

[1981] Server: Display a confirmation message to the user that the reservation was successful.

[1982] Joining a web meeting

[1983] Step 1: Meeting start notification

[1984] Server: Send a reminder to users and support staff 10 minutes before the meeting start time.

[1985] Step 2: Generate a meetings link

[1986] Server: Generates and sends secure web meeting links to users and support personnel.

[1987] Step 3: Join the meeting

[1988] Terminal: The user and support staff click on the link sent to join the web meeting.

[1989] Step 4: Start Screen Sharing

[1990] Terminal: The user uses the screen sharing function to show the current work status to the support staff.

[1991] User's business background

[1992] Step 1: Oral explanation of the business background

[1993] Terminal: The user verbally explains the specific background and issues of the work to the support staff.

[1994] Step 2: Share materials

[1995] Terminal: The user provides relevant materials and documents to the support staff via screen sharing or file transfer.

[1996] Providing optimal prompts

[1997] Step 1: Create a Prompt

[1998] Support staff terminal: Generates optimal prompt suggestions based on the user's explanation.

[1999] Step 2: Propose a Prompt

[2000] Terminal: The support staff presents the proposed prompt to the user and explains it to them.

[2001] Step 3: Receive feedback and make corrections

[2002] On the device: The user provides feedback on the proposed prompt and requests revisions if necessary.

[2003] Support staff device: Receive feedback and revise the proposed prompt.

[2004] Meeting recording

[2005] Step 1: Save the meeting recording

[2006] Server: Stores meeting logs (chat history, shared files, generated prompts) in a database.

[2007] Step 2: Notification and Access to Records

[2008] Server: After the meeting ends, notify the user that the recording has been saved and provide an access link.

[2009] Step 3: Review and download the recording

[2010] On your device: Users can click the link provided to log in, view the meeting recording, and download it if necessary.

[2011] In this way, specific processing is carried out at each step, and the system provides assistance to users to make the most of generated AI and improve business efficiency.

[2012] Example 1

[2013] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2014] In today's work environment, users need appropriate support to efficiently utilize generative AI to perform their work. However, if users lack the specific background information or know-how to devise prompts, it can be difficult to maximize the effectiveness of generative AI. Furthermore, the process of sharing the status and progress of work with support staff in real time and receiving specific advice must be streamlined. Furthermore, it is also important to properly store meeting records so they can be referenced later.

[2015] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[2016] In this invention, the server includes a means for a user to create and authenticate an account, a means for the user to select a date and time based on the user's work and schedule an online conference, and a means for a support staff member to participate in the online conference and generate optimal prompts based on the work materials shared by the user. This allows the user to receive advice from the appropriate support staff member and efficiently utilize the generation AI, significantly improving work efficiency. Furthermore, by properly storing online conference records and dialogue history, users can enjoy the benefit of making them easier to reference later.

[2017] "Creating an account and authenticating" refers to the process in which a user first registers (signs up) with the system and the system verifies the user's authenticity based on the information entered.

[2018] "Selecting a date and time to reserve an online meeting" refers to the process by which a user selects a specific date and time using the calendar function within the system and reserves the system to hold an online meeting at that date and time.

[2019] "Participating in an online meeting" refers to the process in which a user and a support staff member connect online using a web link generated at a specified date and time to hold a meeting.

[2020] "Generating optimal prompt sentences based on business materials" refers to the process in which a support staff member creates and provides effective prompts to the generation AI based on the business-related materials and information shared with the support staff by the user.

[2021] "Storing on a network server" refers to the process of securely storing online meeting records, conversation histories, shared data, etc. on a remote data storage system.

[2022] "Explaining the work background and sharing information on the screen in real time" refers to the process in which a user explains the details and background of the work while sharing the screen of their computer or device with a support staff member in real time during an online meeting.

[2023] "Storing online meeting logs, dialogue history, shared data, and generated prompts in a management system" refers to the process of storing all records and data related to online meetings in a dedicated management system for future reference.

[2024] This invention is a system designed to allow users to improve work efficiency by utilizing generated AI. Specific embodiments of the invention are described below.

[2025] overview

[2026] This system helps users create an account through a web application, schedule a business meeting, and participate in an online meeting, and then receive appropriate prompt advice from support staff, thereby improving work efficiency.

[2027] Hardware and Software Configuration

[2028] server

[2029] The server is a backend system for user authentication, conference booking, online conference link generation, and meeting record storage. Specifically, it has the following configuration:

[2030] Database: Use a database (e.g. PostgreSQL) to store user information, booking information, and meeting records.

[2031] Application server: Uses a backend framework (e.g., Node.js, Django) that provides user authentication and conference management functions.

[2032] Mail Server: Use the mail server to send account confirmation emails and meeting reminder notifications.

[2033] Terminal

[2034] The terminal is where users and support staff access and operate the web application. It consists of:

[2035] Browser: The user uses a web browser (e.g., Google Chrome, Firefox) to access the web application.

[2036] Screen sharing software: Use software (e.g., Zoom, Microsoft Teams) that allows users to share their screens with support personnel in real time.

[2037] Explanation of program processing

[2038] Sign up and log in

[2039] Device: The user accesses the web application using a browser, clicks the "Sign Up" button, enters their username, email address, and password, and clicks "Register." The server stores the entered information in a database and sends a confirmation email to the user. The user clicks the link in the email and their account is activated. The user then enters their email address and password on the login page and clicks "Login." The server performs authentication, and if successful, displays the dashboard to the user.

[2040] Examples:

[2041] A user accesses the web app in a browser, enters "testuser", "testuser@example.com", and "password123", and clicks the register button. The server stores this information in a database and sends a confirmation email to "testuser@example.com".

[2042] Web Meeting Reservation

[2043] Terminal: The user selects the desired date and time from the calendar in the dashboard and clicks the "Reserve" button. The server saves the reservation information in the database and sends a reservation notification to the support staff.

[2044] Examples:

[2045] The user selects "October 15, 2023, 3:00 PM" on the dashboard calendar and clicks the "Book" button. The server saves this information in the database and notifies the support staff by email.

[2046] Joining a web meeting

[2047] Terminal: When the meeting start time arrives, users and support staff click the provided link to join the web meeting. The server sends reminder notifications and generates and provides a secure web meeting link to participants.

[2048] Examples:

[2049] Five minutes before the meeting starts, the server sends a reminder to the user and support staff and provides them with a meeting link, which they click to join.

[2050] User's business background

[2051] Device: The user uses the screen sharing function to share the work situation and related documents with the support staff. The support staff understands the work situation based on the user's explanation and takes notes.

[2052] Examples:

[2053] The user clicks the screen sharing button in the browser and displays the Excel file used in work to the support staff.

[2054] Providing optimal prompts

[2055] Support staff terminal: The support staff will devise a prompt suitable for the generation AI based on the user's work background. The user will review the proposed prompt and make any necessary corrections or questions.

[2056] Specific prompt examples:

[2057] Suggest business prompts such as "Automatically generate invoices."

[2058] Meeting recording

[2059] Server: After the meeting, the logs, chat history, shared files, and generated prompts are stored in a database. Users can access the recordings later and download them as needed.

[2060] Examples:

[2061] The server stores meeting records, interaction history, and shared files in a database, which users can download by accessing the history section of their dashboard.

[2062] As described above, by using this system, users can effectively utilize generative AI to optimize their business processes. With the help of support staff, they can quickly create effective prompts, significantly improving work efficiency.

[2063] The flow of the identification process in the first embodiment will be described with reference to FIG.

[2064] Step 1:

[2065] Create an account (sign up)

[2066] Device: The user visits the web application and enters their username, email address, and password into the sign-up form.

[2067] Specific behavior:

[2068] The user enters "testuser", "testuser@example.com", and "password123" and clicks the Register button.

[2069] Server: The server stores the entered username, email address, and password in a database and sends a confirmation email to the user. Based on the user's input, the server stores the information in a database and sends a confirmation email.

[2070] Input: Username, Email Address, Password

[2071] Data processing: Save user information in a database and request the mail server to send a confirmation email

[2072] Output: User information stored in the database, confirmation email sent to the user

[2073] Step 2:

[2074] Account authentication (login)

[2075] Device: After signing up, the user enters their email address and password on the login page and clicks "Login."

[2076] Specific behavior:

[2077] The user enters "testuser@example.com" and "password123" and clicks the login button.

[2078] Server: The server checks the input information against the database and performs authentication. If successful, it displays the dashboard to the user.

[2079] Input: Email address, Password

[2080] Data processing: Retrieving user information from the database and matching it with the input information

[2081] Output: Authentication result. If successful, the user's dashboard screen will be displayed.

[2082] Step 3:

[2083] Web Meeting Reservation

[2084] Terminal: The user selects the desired date and time using the calendar function in the dashboard and clicks the "Book" button.

[2085] Specific behavior:

[2086] The user selects "October 15, 2023, 3:00 PM" from the calendar and clicks the reservation button.

[2087] Server: The server saves the reservation information in a database and sends a reservation notification to the support staff. It saves the information to the database and notifies the staff based on the user's input.

[2088] Input: Reservation date and time

[2089] Data processing: Save the reservation information to the database and call the notification system to send reservation notifications to support staff.

[2090] Output: Booking information stored in the database, booking notification sent to support staff

[2091] Step 4:

[2092] Joining a web meeting

[2093] Server: Sends reminders to users and support staff before the meeting starts and generates a secure web meeting link.

[2094] Input: Reservation information

[2095] Data processing: generating and sending reminder notifications, generating web meeting links

[2096] Output: Reminder sent, generated web meeting link

[2097] Terminal: Users and support staff click on the provided link to join the web meeting.

[2098] Specific behavior:

[2099] Five minutes before the meeting starts, the user and support staff will check the notification and click the provided link to join.

[2100] Step 5:

[2101] Business background explanation

[2102] Device: Users can use the screen sharing function to share work status and related documents with support staff.

[2103] Specific behavior:

[2104] The user clicks the screen sharing button in the browser and displays an Excel file used for work.

[2105] Support staff's device: The support staff understands the work situation based on the information shared on the screen and the user's explanation, and takes notes.

[2106] Input: Screen sharing information, user's verbal description

[2107] Data processing: Support staff understanding and recording

[2108] Output: Support personnel notes

[2109] Step 6:

[2110] Providing optimal prompts

[2111] Support staff terminal: The support staff will devise a prompt suitable for the generation AI based on the user's work background and present it to the user.

[2112] Input: User's business background information

[2113] Data processing: Prompt generation

[2114] Output: Suggested prompt text

[2115] Example prompt:

[2116] Suggest business prompts such as "Automatically generate invoices."

[2117] User's device: The user reviews the suggested prompt and makes any necessary corrections or questions.

[2118] Specific behavior:

[2119] The user reviews the prompts from the support staff and, if necessary, requests to "change the date format."

[2120] Step 7:

[2121] Meeting recording

[2122] Server: After the meeting ends, the log, chat history, shared files, and generated prompts are saved in a database.

[2123] Input: Meeting logs, chat history, shared files, generated prompts

[2124] Data processing: Record preservation processing

[2125] Output: Records stored in a database

[2126] Terminal: User can access the recording later and download it if needed.

[2127] Specific behavior:

[2128] After a meeting ends, users can access the History section of their dashboard and download a specific meeting recording.

[2129] (Application example 1)

[2130] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2131] To improve efficiency and productivity in food delivery operations, there is a need for technology that allows users to collaborate with support staff in real time and provides optimal routes using generative AI models. However, conventional systems have issues with being unable to efficiently perform a series of tasks, such as web meetings, recording work, and providing optimal routes.

[2132] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2133] In this invention, the server includes a means for users to sign up and log in, a means for users to book web meetings based on their tasks, a means for support staff to participate in the web meetings and provide optimal prompts based on the user's work background, a means for saving meeting records on the server, a means for users to manage delivery task reservations, a means for providing optimal route suggestions using a generative AI model, and a means for inputting feedback after the task is completed. This significantly improves the efficiency and productivity of food delivery tasks, and enables users to effectively use generative AI models to optimize their tasks.

[2134] "User" means a person who uses the system to improve business efficiency.

[2135] "Signing up" is the process of creating a new account.

[2136] "Login" means the act of authenticating to access an existing account and use the system.

[2137] A "web meeting" is an online conference held over the Internet.

[2138] "Reservation" is the act of setting up a web meeting for a specific date and time.

[2139] A "support staff member" is someone whose role is to support users in optimizing their operations.

[2140] A "prompt" is an input sentence that gives instructions to a generative AI model.

[2141] "Recording" refers to saving the progress and content of a meeting in a format that can be referenced later.

[2142] A "server" is a computer system that stores and processes data.

[2143] "Delivery services" refers to the general business of delivering goods or services to designated locations.

[2144] A "generative AI model" is an artificial intelligence algorithm that makes inferences and predictions based on data.

[2145] A "route proposal" is a plan that suggests an efficient route for delivery work.

[2146] "Feedback" is the act of recording evaluations and opinions after a task is completed.

[2147] This invention is a system that uses generative AI models to help improve the efficiency of food delivery operations. The system involves a series of processes in which users sign up and log in, book web meetings, and share their work status with support staff in real time.

[2148] 1. Sign up and login

[2149] The server stores the user's input information in a database and performs authentication. For example, when signing up, the user enters their username, email address, and password, and after receiving a verification email, clicks the link to log in.

[2150] 2. Delivery business reservation management

[2151] The user's device is used to manage delivery schedules. The server uses the Google Calendar API to store reservation information in a cloud database and notify support staff.

[2152] 3. Participating in a web meeting

[2153] Using WebRTC, the user and support staff join a web meeting at the specified time. The server sends a reminder notification and generates a secure meeting link.

[2154] 4. Explanation of business background and screen sharing

[2155] Users can use the screen sharing function to show support staff the documents and progress they are using in their work, making it easier for support staff to understand the user's work background.

[2156] 5. Providing optimal route suggestions

[2157] The support staff terminal uses a generative AI model based on the user's work background to provide optimal route suggestions. For example, the generative AI can suggest optimal routes using prompts such as: "Please suggest the optimal route based on the delivery address and customer address."

[2158] 6. Feedback after the job is completed

[2159] After completing a delivery task, users can enter feedback using their devices. This feedback is stored on the server and can be used to improve future operations.

[2160] 7. Meeting Recording

[2161] The server stores meeting logs, chat history, shared files, and generated prompts in a database such as Firebase Firestore, allowing for future review and continuous improvement.

[2162] For example, a user who wants to streamline delivery operations during lunchtime can enter the following: "We have a high volume of deliveries during lunchtime. Please suggest the best route and most efficient delivery method."

[2163] Generative AI analyzes data to create optimal route plans in real time and notify users' smartphones, thereby significantly improving the efficiency of food delivery operations.

[2164] The system can be realized using specific hardware and software such as Firebase Authentication, Google Calendar API, WebRTC, OpenAI GPT-3, and Firebase Firestore.

[2165] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2166] Step 1:

[2167] A user signs up.

[2168] Input: Username, Email Address, Password

[2169] Data processing: The terminal collects the user's input information and sends it to the server.

[2170] Output: The server stores the information in a database and sends a verification email.

[2171] How it works: A user opens the web app, enters the required information, and clicks "Register." The server receives the information, writes it to the database, and then sends a verification email.

[2172] Step 2:

[2173] The user logs in.

[2174] Input: Email address, password

[2175] Data processing: The server validates and authenticates the input information.

[2176] Output: Login success message and transition to the dashboard screen

[2177] What happens: The user enters their email address and password on the login page and clicks "Login." The server verifies the information and, if correct, displays the dashboard screen.

[2178] Step 3:

[2179] A user books a delivery job.

[2180] Input: Scheduled delivery date and time, detailed information

[2181] Data processing: The server saves the reservation information and updates the calendar using the Google Calendar API.

[2182] Output: Reservation confirmation and notification to support staff

[2183] What it does: The user selects the desired date and time on the dashboard and clicks "Book." The server saves the reservation information, updates the calendar using the Google Calendar API, and notifies the support staff.

[2184] Step 4:

[2185] A user joins a web meeting.

[2186] Input: Meeting time, link

[2187] Data processing: The server sends reminder notifications and generates secure meeting links over WebRTC.

[2188] Output: Reminders, Meeting Links

[2189] Specific operation: When the meeting start time arrives, the user and support staff click the designated link to join the web meeting. The server sends a notification and generates and distributes the link.

[2190] Step 5:

[2191] The user explains the background of the work to the support staff and shares the screen.

[2192] Input: Business documents, progress status

[2193] Data processing: The device shares the screen and communicates the work status to the support staff.

[2194] Output: Shared work documents, progress explanation

[2195] Specific operation: The user uses the screen sharing function to show the support staff the materials and progress, and the support staff deepens their understanding and takes notes.

[2196] Step 6:

[2197] Support staff use generative AI models to provide optimal route suggestions.

[2198] Input: User's business background information, delivery route

[2199] Data processing: A generative AI model calculates and suggests efficient routes.

[2200] Output: Optimal route plan

[2201] Specific operation: The support staff inputs prompts into the AI ​​generator based on the information obtained from the user and suggests the optimal route. Example prompt: "Please suggest the optimal route based on the delivery address and customer address."

[2202] Step 7:

[2203] The user enters feedback after completing the delivery job.

[2204] Input: Feedback

[2205] Data processing: The server stores the feedback information.

[2206] Output: Saved feedback data

[2207] What it does: A user enters and submits feedback using the web app. The server stores the information in a database.

[2208] Step 8:

[2209] The server stores the meeting records.

[2210] Input: Meeting logs, chat history, shared files, generated prompt ideas

[2211] Data processing: The server stores the data in a database.

[2212] Output: Saved meeting data

[2213] Specific operation: After the meeting ends, the server saves the logs, chat history, shared files, and generated prompt ideas in a database such as Firebase Firestore.

[2214] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2215] This invention is a system that provides support to users to improve work efficiency using generative AI, and aims to provide more effective support by combining it with an emotion engine that recognizes and responds to user emotions. The system involves a series of processes in which users sign up and log in, book a web meeting, explain the work background through screen sharing with a support staff member, and receive optimal prompt suggestions. In addition, the emotion engine collects user emotion data, adjusts prompt suggestions based on that data, and stores records.

[2216] Program processing overview

[2217] 1. Sign up and login

[2218] Device: The user accesses the web app and creates an account (signs up) or logs in with an existing account.

[2219] Server: Stores the information entered by the user in a database, performs authentication, and sends a confirmation email to the user once sign-up is complete.

[2220] 2. Schedule a web meeting

[2221] Terminal: Users can see available slots on their dashboard and select the desired date and time for the meeting.

[2222] Server: Saves reservation information in a database and notifies support staff.

[2223] 3. Participating in a web meeting

[2224] Terminal: When the meeting start time arrives, the user and support staff click on the provided link to join the web meeting.

[2225] Server: Sends reminders and generates and provides meeting links to participants.

[2226] 4. Explanation of the user's business background

[2227] Terminal: The user can use the screen sharing function to display the current status of the work to the support staff, provide verbal explanations, and share relevant materials and documents.

[2228] Support staff's terminal: The support staff understands the current situation based on the user's explanation and takes notes.

[2229] 5. Operation of the Emotion Engine

[2230] Device: Uses a camera and microphone to analyze the user's facial expressions and voice, and collects emotional data from them.

[2231] Server: The emotion engine processes the collected data in real time and recognizes the user's emotions.

[2232] 6. Providing optimal prompts

[2233] Support staff device: Based on the user's work background and emotional data, the device creates prompt ideas suitable for the generating AI and proposes them to the user.

[2234] User's device: The user receives the suggestions and asks questions or requests corrections as needed.

[2235] 7. Meeting Recording

[2236] Server: Stores meeting logs, chat history, shared files, sentiment data, and generated prompt ideas in a database.

[2237] Device: Users can review recordings after the meeting and download them if necessary.

[2238] Specific examples

[2239] 1. User signs up and logs in

[2240] On the device: The user opens the web app, clicks the "Sign Up" button, enters their username, email address, and password, and clicks "Register."

[2241] Server: Stores the entered information in a database and sends a confirmation email to the user. When the user clicks on the link in the email, the login page is displayed.

[2242] 2. A user books a web meeting

[2243] On the device: The user goes to the "Book a Meeting" section of their dashboard, selects the desired date and time from the calendar, clicks the "Book" button and receives a confirmation message.

[2244] Server: Saves reservation information in a database and sends reservation notifications to support staff.

[2245] 3. The user and the support staff join a web meeting

[2246] On your device: The user opens the web app at the designated time and clicks the "Join Meeting" button. The support staff also joins.

[2247] Server: Generates secure web meeting links and sends reminder notifications.

[2248] 4. User explains the business background

[2249] Device: The user shares the documents and screens they use in their work and explains the specific work background and challenges to the support staff.

[2250] Support staff's device: Understand the job content and listen to the user's explanation while taking notes.

[2251] 5. Operation of the Emotion Engine

[2252] Device: Captures the user's face with a camera, picks up their voice with a microphone, and collects emotional data in real time.

[2253] Server: Analyzes the collected data and determines the user's emotional state.

[2254] 6. Providing optimal prompts

[2255] Support staff device: Taking into account the user's work background and emotional state, the system creates prompt proposals suitable for the generative AI and presents them to the user.

[2256] User's device: The user accepts the suggestions and makes any necessary corrections or questions.

[2257] 7. Meeting Recording

[2258] Server: After the meeting ends, the logs, chat history, shared files, emotion data, and generated prompts are stored in a database.

[2259] On-device: Users can review saved records later and download them as needed.

[2260] In this way, the system helps users maximize the use of generative AI and emotion recognition to improve work efficiency.

[2261] The processing flow will be explained below.

[2262] Sign up and log in

[2263] Step 1: View the sign-up page

[2264] Device: User visits web app and clicks "Sign up" button.

[2265] Server: Sends the sign-up page to the browser and displays it.

[2266] Step 2: Enter your sign-up information

[2267] On the device: The user fills in a form with their username, email address, password, and confirmation password.

[2268] Step 3: Submit your sign-up information

[2269] Terminal: The user clicks the "Register" button and sends the information to the server.

[2270] Step 4: Verify and save your sign-up information

[2271] Server: Checks the entered information for consistency and duplication, and if there are no problems, saves it to the database.

[2272] Server: Sends a confirmation email to the user.

[2273] Step 5: Display the login page

[2274] Server: After successful sign-up, display the login page to the user.

[2275] Step 6: Enter your login details

[2276] Device: The user enters their email address and password and clicks the "Login" button.

[2277] Step 7: Verify your login details

[2278] Server: The entered email address and password are verified against the database.

[2279] Step 8: View the Dashboard

[2280] Server: If authentication is successful, display the user's dashboard page.

[2281] Web Meeting Reservation

[2282] Step 1: View the booking page

[2283] On your device: The user accesses the "Book a Meeting" section of their dashboard.

[2284] Server: Sends data to display the reservation page.

[2285] Step 2: Select a reservation slot

[2286] Terminal: The user selects the desired date and time from the calendar.

[2287] Step 3: Confirm and submit your booking

[2288] Terminal: The user clicks the "Confirm Reservation" button and sends the information to the server.

[2289] Step 4: Save your reservation

[2290] Server: Saves the entered reservation information in a database and sends a reservation notification to the support staff.

[2291] Step 5: View the reservation confirmation message

[2292] Server: Display a confirmation message to the user that the reservation was successful.

[2293] Joining a web meeting

[2294] Step 1: Meeting start notification

[2295] Server: Send a reminder to users and support staff 10 minutes before the meeting start time.

[2296] Step 2: Generate a meetings link

[2297] Server: Generates and sends secure web meeting links to users and support personnel.

[2298] Step 3: Join the meeting

[2299] Terminal: The user and support staff click on the link sent to join the web meeting.

[2300] Step 4: Start Screen Sharing

[2301] Terminal: The user uses the screen sharing function to show the current work status to the support staff.

[2302] User's business background

[2303] Step 1: Oral explanation of the business background

[2304] Terminal: The user verbally explains the specific background and issues of the work to the support staff.

[2305] Step 2: Share materials

[2306] Terminal: The user provides relevant materials and documents to the support staff via screen sharing or file transfer.

[2307] Emotion Engine Operation

[2308] Step 1: Collecting emotion data

[2309] Device: Emotional data is collected by taking a picture of the user's face with a camera and picking up their voice with a microphone.

[2310] Step 2: Analyze the emotion data

[2311] Server: Analyzes collected data in real time and determines the user's emotional state.

[2312] Providing optimal prompts

[2313] Step 1: Create a Prompt

[2314] Support staff device: Based on the user's work background and emotional data, the device creates a prompt suitable for the generating AI and presents it to the user.

[2315] Step 2: Propose a Prompt

[2316] Terminal: The support staff presents the proposed prompt to the user and explains it to them.

[2317] Step 3: Receive feedback and make corrections

[2318] On the device: The user provides feedback on the proposed prompt and requests revisions if necessary.

[2319] Support staff device: Receive feedback and revise the proposed prompt.

[2320] Meeting recording

[2321] Step 1: Save the meeting recording

[2322] Server: Stores meeting logs (conversation records, screen sharing content, emotion data, and generated prompts) in a database.

[2323] Step 2: Notification and Access to Records

[2324] Server: After the meeting ends, notify the user that the recording has been saved and provide an access link.

[2325] Step 3: Review and download the recording

[2326] On your device: Users can click the link provided to log in, view the meeting recording, and download it if necessary.

[2327] In this way, specific actions are performed at each processing step, and the system helps users maximize the use of generative AI and emotion recognition to improve work efficiency.

[2328] Example 2

[2329] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2330] Conventional business support systems have difficulty providing optimal instructions based on the user's work context, and do not take the user's emotions into consideration when providing support. As a result, there are problems with user satisfaction and work efficiency declining.

[2331] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2332] In this invention, the server includes a means for a user to sign up and log in, a means for a user to reserve an online meeting based on the user's work, a means for a support person to participate in the online meeting and provide optimal instructions based on the user's work background, a means for analyzing the user's facial expressions and voice and collecting emotional data, a means for processing the collected emotional data in real time and recognizing the user's emotions, and a means for saving a record of the meeting on the server. This makes it possible to provide appropriate instructions according to the user's work background and support that takes the user's emotions into consideration.

[2333] "Signing up" is the process by which a user creates a new account and registers with the system.

[2334] "Logging in" is the process of accessing a system using existing account information.

[2335] An "online meeting" is a conference held over the Internet, allowing participants to communicate in real time.

[2336] "Support Personnel" means personnel who provide technical and business support to Users.

[2337] "Instructions" are texts created using generative AI models that provide specific suggestions or guidelines to help users with their work.

[2338] "Facial Expression" refers to the muscle movements and other physical characteristics of a user's face that indicate an emotional state.

[2339] "Voice" refers to the voice produced by the user, and is data used to determine the user's emotional state using the tone, pitch, volume, etc.

[2340] "Emotional data" is information that expresses the user's emotional state as numerical or categorical data.

[2341] "Real-time processing" means processing data immediately after it is generated, providing results with little to no delay.

[2342] A "meeting recording" is a series of records containing digital data, such as conversations, chats, and shared files, that take place during an online meeting.

[2343] This invention is a support system that utilizes a generative AI model to help users improve their work efficiency, and also combines it with an emotion engine that recognizes and responds to user emotions.

[2344] The system includes the following means:

[2345] 1. Sign-up and login methods

[2346] Device: A user visits a web app using a JavaScript framework to sign up and log in. Specifically, the user enters a username, email address, and password, and submits the sign-up form.

[2347] Server: Build a backend API using Node.js and Express, store this information in a MySQL database, and use an SMTP server to send confirmation emails, for example, sendgrid.

[2348] 2. Online Meeting Reservation Methods

[2349] On the device: Users access a dashboard powered by React.js and use a calendar component such as FullCalendar.js to book online meetings.

[2350] Server: Reservation information is stored in a MySQL database, and the server sends reservation confirmation emails to users and support staff.

[2351] 3. How to join an online meeting

[2352] Device: Users and support staff log in to the web app at the specified date and time and join through the API of online meeting tools such as Google Meet and Zoom.

[2353] Server: Sends reminders near the scheduled time and generates a secure meeting link.

[2354] 4. Means of explaining the business background

[2355] Device: During online meetings, users can use Zoom's screen sharing function to explain the current work background and related materials to support staff.

[2356] Support staff's device: The support staff listens to the user's explanation and takes notes of important points using a note-taking app (e.g., OneNote) on their PC.

[2357] 5. Emotion Engine Operation Method

[2358] Device: The user's face is photographed with a webcam and their voice is collected with a microphone. A facial expression recognition model and voice analysis tool using TensorFlow.js are used.

[2359] Server: Uses a Python sentiment analysis library (e.g., emotion_recognition) to process the collected data in real time and recognize user emotions.

[2360] 6. Optimal Means of Providing Instructions

[2361] Support staff terminal: The support staff uses a generative AI model (e.g., Chat GPT) to generate instructions based on the user's work background and emotional data.

[2362] User's device: Provide the generated instructions to the user, correcting or asking questions as needed.

[2363] Examples of prompts:

[2364] "I want to use a generative AI model to generate optimal prompt suggestions. My user is preparing a presentation for a new marketing campaign and is particularly short on ideas for slide design. He's feeling stressed. What would be the best prompt?"

[2365] 7. Meeting Record Keeping Method

[2366] Server: After the online meeting ends, the logs, communication history, shared files, emotional data, and generated instructions are stored in a MySQL database.

[2367] On your device: Users can view saved recordings in the "Meeting History" section of the web app and download them if necessary.

[2368] In this way, the present invention provides specific assistance to users to make the most of generative AI models and emotion recognition to improve work efficiency.

[2369] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2370] Step 1:

[2371] Sign up and log in

[2372] On the device: The user opens the web app and clicks the "Sign up" button. The user enters their username, email address, and password and clicks "Register."

[2373] Enter your username, email address, and password.

[2374] Output: Signup request data.

[2375] Server: Receives the sign-up request data, stores the entered information in a MySQL database, and then sends a confirmation email using an SMTP server.

[2376] Input: Signup request data.

[2377] Output: Database saved results and confirmation email.

[2378] On the device: When the user clicks on the link in the confirmation email, they are authenticated and directed to a login page.

[2379] Enter: Click on the confirmation link.

[2380] Output: Login page displayed.

[2381] Step 2:

[2382] Schedule an online meeting

[2383] On the device: The user opens the dashboard, selects the desired date and time from the calendar, and clicks the "Book" button.

[2384] Input: Desired date and time.

[2385] Output: Booking request data.

[2386] Server: Receives the reservation request data, stores the reservation information in a MySQL database, and then sends a reservation confirmation email to the user and support staff.

[2387] Input: Booking request data.

[2388] Output: Database saved results and confirmation email.

[2389] Step 3:

[2390] Joining an online meeting

[2391] Device: The user and support staff open the web app at the designated time and click the "Join Meeting" button.

[2392] Input: Meeting time, Join button click.

[2393] Output: Meeting join request.

[2394] Server: Sends reminders before the specified time and generates a secure meeting link using Google Meet or Zoom APIs.

[2395] Input: Meeting join request.

[2396] Output: Reminder and meeting link.

[2397] Step 4:

[2398] Business background explanation

[2399] Terminal: The user uses the screen sharing function to show the support staff the current task and related materials, and provides verbal explanations.

[2400] Input: Start screen sharing, business documents.

[2401] Output: Screen sharing data.

[2402] Support staff's device: The support staff listens to the user's explanation and takes notes of important points using a note-taking app on their PC.

[2403] Input: Screen share data.

[2404] Output: Note data.

[2405] Step 5:

[2406] Emotion Engine Operation

[2407] Device: The user's face is photographed with a webcam and their voice is collected with a microphone. TensorFlow.js is used to capture facial expression and voice data in real time.

[2408] Input: Video and audio data.

[2409] Output: facial expression data, voice data.

[2410] Server: Sends the collected data to a Python sentiment analysis library to recognize the user's emotional state.

[2411] Input: facial expression data, voice data.

[2412] Output: Emotional state data.

[2413] Step 6:

[2414] Providing optimal instructions

[2415] Support staff device: Using the user's work context and emotional data, prompts are input into a generative AI model (e.g., Chat GPT) to generate optimal instructions.

[2416] Input: Work background data, emotional state data.

[2417] Output: The generated directive.

[2418] User's device: Provide the generated instructions to the user. If the user is not satisfied with the content, they can request corrections or ask questions.

[2419] Input: The generated instruction.

[2420] Output: User feedback.

[2421] Step 7:

[2422] Meeting Recording

[2423] Server: After the meeting ends, the logs, communication history, shared files, emotional data, and generated instructions are saved in a MySQL database.

[2424] Input: Meeting logs, communication history, shared files, emotion data, generated instructions.

[2425] Output: Database save results.

[2426] On your device: Users can view and optionally download recordings from the "Meeting History" section of the web app.

[2427] Input: User's record confirmation request.

[2428] Output: Meeting recording data.

[2429] (Application example 2)

[2430] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2431] Conventional business support systems lack the ability to utilize generative AI to provide prompts to users so that they can receive efficient support in their work. Furthermore, they lack the ability to provide flexible support based on the user's emotional state, which can increase stress and reduce work efficiency. Furthermore, they lack a system for monitoring the operating status of factory robots in real time and quickly detect and respond to abnormalities, which can lead to production line stagnation.

[2432] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[2433] In this invention, the server includes a means for users to sign up and log in, a means for users to reserve web meetings based on their work, a means for support staff to participate in the web meetings and provide optimal prompts based on the user's work background, a means for saving meeting records on the server, a means for collecting user emotion data using an emotion engine and adjusting prompt proposals based on that data, a means for monitoring the operating status of factory robots, and a means for sending notifications when an abnormality is detected. This allows users to use generative AI and emotion recognition to improve work efficiency, and enables them to monitor the operating status of factory robots in real time to quickly detect and respond to abnormalities.

[2434] "User" means an individual or corporation that uses the system to receive business support.

[2435] "Sign-up and login" refers to the process by which a user creates an account and accesses the system with an existing account.

[2436] A "web meeting" is a meeting conducted over the Internet between a user and a support staff member using screen sharing and audio / video calls.

[2437] A "prompt" is a specific instruction or suggestion that a generative AI model provides to a user.

[2438] A "support staff member" is a person who provides business support and advice to users.

[2439] "Server" means a computer system that stores and manages user data and meeting records.

[2440] An "emotion engine" is software that analyzes a user's facial expressions and voice to recognize their emotional state.

[2441] "Emotional Data" refers to information about a user's emotions collected and analyzed by the Emotion Engine.

[2442] A "factory robot" is a robot used to perform automated tasks in a factory.

[2443] "Operation status" refers to the status of a factory robot, indicating whether it is operating normally or whether an error or abnormality has occurred.

[2444] "Anomaly detection" is the process of detecting abnormalities when they occur in the operation of factory robots.

[2445] A "notification" is a message sent by the system to notify the user of an abnormality or event.

[2446] A "log" is a detailed record of web meetings and activity within the system.

[2447] "Chat history" means a record of text communications that take place during a web meeting.

[2448] "Shared files" refers to documents and materials shared between the user and the support staff during a web meeting.

[2449] This invention provides a support system for improving business efficiency by combining generative AI and an emotion recognition engine. In particular, it can monitor the operating status of factory robots in real time and quickly detect and respond to abnormalities. Below, we will explain in detail the embodiments of this system.

[2450] This support system consists of the following means:

[2451] 1. How users sign up and log in

[2452] Device: The user uses the smartphone app to create an account (sign up) or log in with an existing account.

[2453] Server: Store the user's input information in a database and use a service such as Firebase Authentication for authentication.

[2454] 2. A way for users to schedule business-based web meetings

[2455] On the device: Users can check available slots on the app dashboard and book a web meeting at the desired date and time.

[2456] Server: Saves user reservation information in a database and notifies support staff using the Google Calendar API, etc.

[2457] 3. A way for support staff to join a web meeting and provide the most appropriate prompts based on the user's work context

[2458] Device: When the web meeting starts, the user and support staff click the specified link to join. The screen sharing function is also used so that the user can explain the background of the work and the support staff can take notes.

[2459] Server: Generates secure web meeting links and sends reminder notifications.

[2460] 4. Using an emotion engine to gather user sentiment data and tailor prompts based on that data

[2461] Device: The camera captures the user's face, the microphone picks up their voice, and collects emotional data in real time.

[2462] Server: Analyzes emotional data using Microsoft Azure Emotion API and other tools to determine the user's emotional state.

[2463] 5. A means of monitoring the operation status of factory robots

[2464] Terminal: Factory workers can check the status of running robots on a dashboard and select the robot with the problem.

[2465] Server: Updates the robot's operating status data in real time and sends notifications if an abnormality is detected.

[2466] 6. How to send notifications when an anomaly is detected

[2467] Server: When an abnormality is detected, a notification is sent immediately and an alert is issued to the user, which encourages a prompt response.

[2468] 7. How to store meeting records on a server

[2469] Server: Use a database service such as Firebase Firestore to store meeting logs, chat history, shared files, and generated prompt ideas in a database.

[2470] On your device: Users can review the recording after the meeting and download it if needed.

[2471] This system allows users to make the most of generative AI and emotion recognition to improve work efficiency, while also enabling them to monitor the operating status of factory robots in real time and quickly detect and respond to abnormalities.

[2472] Specific examples

[2473] Users sign up and log in

[2474] Enter your username, email address, and password, then click "Register."

[2475] A confirmation email will be sent to you and a link will take you to the login page.

[2476] Prompt Sentence Examples

[2477] "User's work context: Monitoring the operation status of a robot in a factory. User's emotional state: Angry (due to frequent errors). Desired output from the generative AI model: Suggest specific steps to improve work efficiency and quickly resolve issues."

[2478] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2479] Specific flow of processing steps

[2480] Step 1:

[2481] User sign-up and login

[2482] Input: A user opens the smartphone app and enters their username, email address, and password.

[2483] Processing: The device sends the entered information to Firebase Authentication, and the server performs authentication. If this is a new registration, the server sends a confirmation email to the user and waits for them to click the confirmation link.

[2484] Output: The server generates an authentication token and displays the user's target page.

[2485] What happens: The user receives a confirmation email and clicks on the link to authenticate their account. After successfully logging in, the dashboard will be displayed.

[2486] Step 2:

[2487] Users book web meetings

[2488] Input: User sees available slots on the app dashboard and selects their preferred time slot.

[2489] Processing: The device sends the reservation information to the server via the Google Calendar API. The server saves the reservation information in the database and notifies the support staff.

[2490] Output: A notification of the meeting reservation will be sent to the support staff. The reservation information will be added to Google Calendar.

[2491] Specific operation: The user selects the desired time and clicks the "Book" button. The server displays a confirmation message and adds the appointment to the support staff's calendar.

[2492] Step 3:

[2493] The user and the support staff join a web meeting

[2494] Input: When the web meeting starts, the user and support staff will click the specified link on their devices.

[2495] Processing: The server generates a secure web meeting link and sends a reminder. The user and support staff use screen sharing to explain the work background and take notes.

[2496] Output: The web meeting starts and both parties are connected correctly.

[2497] Specific operation: The server generates a meeting link, and the user and support staff join. The user shares materials, and the support staff takes notes while receiving explanations.

[2498] Step 4:

[2499] Emotional data collection and analysis using an emotion engine

[2500] Input: User's facial video and audio data are collected via camera and microphone.

[2501] Processing: The device sends the collected data to the Microsoft Azure Emotion API, where the server analyzes it and determines the user's emotional state.

[2502] Output: The user's emotional data is updated in real time and displayed to the support staff.

[2503] Specific operation: Data is collected in real time using a camera and microphone, and an emotion engine analyzes the user's state of joy, anger, sadness, etc.

[2504] Step 5:

[2505] Providing optimal prompts

[2506] Input: The support staff's device acquires the user's work background and emotional data.

[2507] Processing: Based on this data, the support staff sends the optimal prompt proposal to the generative AI model (e.g., OpenAI API). The generative AI model generates an appropriate prompt proposal and returns it to the support staff's device.

[2508] Output: The generated prompt proposal is displayed on the support staff's terminal. A suggestion is made to the user.

[2509] Specific operation: The support staff requests the AI ​​to generate prompts that best suit the user's situation, and then explains the generated prompts to the user.

[2510] Step 6:

[2511] Meeting recording

[2512] Input: Meeting logs, chat history, shared files, and generated prompt ideas are sent to the server.

[2513] Processing: The server stores this data in a database such as Firebase Firestore.

[2514] Output: Records are stored in a database for users to review and download at a later time.

[2515] Specific behavior: After the meeting ends, the server automatically saves logs and other records, allowing users to view details of past meetings.

[2516] In this way, the system helps users maximize the use of generative AI and emotion recognition to improve work efficiency.

[2517] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2518] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2519] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2520] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2521] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2522] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2523] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2524] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2525] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on...

Claims

1. A means for users to sign up and log in; A way for users to book web meetings based on their tasks; A means for support staff to join a web meeting and provide the most appropriate prompts based on the user's work context; a means for storing meeting records on a server; A system including:

2. The system according to claim 1 , further comprising means for allowing a user to share a screen to explain a business background to a support person in real time.

3. The system of claim 1 , further comprising means for storing meeting logs, chat histories, shared files, and generated prompt ideas in a database.

Images