System

Abstract

A system is provided.SOLUTION: The system includes a means for acquiring schedule information of a user, a means for analyzing the acquired schedule information and determining the necessity of icebreaking, a means for proposing an appropriate icebreaking activity, and a means for adding the proposed icebreaking activity to the schedule of the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In today's corporate environment, long meetings and continuous work tasks tend to reduce employee concentration and motivation, resulting in a decline in work efficiency and productivity. It is also difficult to record and manage all the information and key points discussed during important meetings, further contributing to inefficiency. Effective tools are needed to resolve these issues, organize employees' minds, and improve work efficiency. [Means for solving the problem]

[0005] The present invention provides a system that acquires a user's schedule information and analyzes it to determine the need for an icebreaker. The system then includes means for suggesting appropriate icebreaker activities and adding the suggested icebreaker activities to the user's schedule. The system further includes means for collecting audio data of the conversation, converting the collected audio data into text data, and extracting key points from the converted text data. Based on the extracted key points, new tasks and reminders are added to the user's schedule, thereby ensuring that important information from the conversation is not overlooked and improving work efficiency. The system also includes means for displaying notifications to the user and allowing the user to review, approve, and adjust the suggested icebreaker activities and added tasks and reminders.

[0006] "User schedule information" is information recorded in a digital calendar or scheduling system used by a user to manage appointments and tasks.

[0007] A "means for obtaining" is a method or device for collecting a user's schedule information from a digital calendar or scheduling system.

[0008] The "means for analyzing" is software or algorithms for analyzing the acquired schedule information and determining the need for an icebreaker.

[0009] "Need for icebreaking" is a criterion for determining whether refreshing activities are necessary to alleviate users' loss of concentration and increased stress caused by long meetings or continuous tasks.

[0010] An "icebreaker" is a short activity or exercise that allows employees and participants to refresh and relax.

[0011] The "means for suggesting" is a method or system for selecting an appropriate ice-breaking activity based on the analysis results and suggesting it to the user.

[0012] A "means for collecting audio data" is a method or device for digitally recording audio during a conference or meeting using a microphone or other audio recording device.

[0013] "Means for converting into text data" refers to an algorithm or software that converts collected voice data into text data using voice recognition technology.

[0014] A "gist extraction means" is a method or system that uses natural language processing techniques to identify and identify important information or keywords from the converted text data.

[0015] The "means for adding tasks and reminders" is a method or system that automatically sets new appointments and reminders in a user's digital calendar or scheduling system based on the extracted key points.

[0016] A "means for displaying notifications" is a method or system for displaying and notifying a user of suggested icebreaker activities, added tasks, and reminder information on a user's computer or mobile device. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] System configuration

[0039] This invention is a business efficiency improvement system that integrates the acquisition and analysis of user schedule information, ice-breaker suggestions, automatic text conversion and key point extraction of conversation content, schedule creation and notification. The roles and specific operations of the server, terminal, and user are explained below.

[0040] Server Operation

[0041] 1. Acquisition and analysis of schedule information

[0042] The server retrieves the user's schedule information from an external scheduling system (e.g., Google Calendar, Microsoft Outlook). It analyzes the retrieved schedule information and determines whether an icebreaker is necessary. For example, if there are consecutive long meetings, it determines that an icebreaker is necessary.

[0043] 2. Propose and confirm icebreakers

[0044] Based on the analysis results, the server generates a list of ice-breaking activities (e.g., two minutes of stretching, five minutes of chatting) and suggests them to the user's device. The ice-breaking activity selected by the user is added to the schedule.

[0045] 3. Conversation text conversion and key points extraction

[0046] The server receives the voice data from the meeting and converts it into text data using speech recognition technology. Key points (e.g., matters that need to be reported at the next meeting) are extracted from the converted text data.

[0047] 4. Schedule Task Reminders

[0048] Based on the extracted key points, new tasks and reminders are added to the user's calendar, for example, an item such as "Prepare for the next meeting" is automatically added.

[0049] 5. Notification

[0050] The server sends new schedule details and reminders to the user's device, notifying the user. The user checks the notifications and adjusts the schedule as necessary.

[0051] Device behavior

[0052] 1. Receiving and displaying schedule information

[0053] The terminal displays the schedule information received from the server to the user, who can then check and select the icebreaker activities suggested through the terminal.

[0054] 2. Collection of audio data

[0055] The device collects audio data during the meeting in real time and sends it to a server. The audio data is also recorded in digital format and can be viewed later.

[0056] 3. Display of notifications

[0057] The device displays notifications of new schedules and reminders received from the server to the user, who can then review, approve, and adjust the schedule.

[0058] User operations

[0059] 1. Check and select your schedule

[0060] The user checks the schedule on their device and selects an icebreaker activity suggested by the server. The selected activity is automatically added to the schedule.

[0061] 2. Record and review conversations

[0062] When a user starts a meeting, they can turn on the device's audio recording function to record the conversation. After the meeting, they can check the converted text data and extracted key points.

[0063] 3. Review and adjust notifications

[0064] Users can view notifications on their device, acknowledge new tasks and reminders, and adjust their schedules as needed.

[0065] Specific examples

[0066] Example 1: Long meetings

[0067] A user enters a week's schedule, including a three-hour series of meetings.

[0068] The server analyzes this and suggests appropriate icebreakers before, during, or after the meeting.

[0069] The user selects "5 minutes of stretching" and the server schedules it and sets up a notification.

[0070] Example 2: Recording important information during a meeting

[0071] A user starts a conference and turns on the audio recording function of the terminal. The audio during the conversation is collected by the terminal and sent to the server.

[0072] The server converts the speech into text and extracts key points.

[0073] It is determined that "Submission of market analysis report" is required before the next meeting, and this information is added to the calendar and notified to the user as a reminder.

[0074] As described above, the present invention is a useful system that improves the efficiency of users' schedule management and the recording and analysis of conversation content, thereby improving work efficiency.

[0075] The processing flow will be explained below.

[0076] Step 1:

[0077] The user accesses the tool and starts the schedule management service.

[0078] Step 2:

[0079] The device retrieves the user's calendar data from an external scheduling service such as Google Calendar or Microsoft Outlook, and then sends this information to the server.

[0080] Step 3:

[0081] The server analyzes the received schedule data and identifies when an icebreaker is needed, such as during a long meeting or when consecutive tasks are required.

[0082] Step 4:

[0083] The server determines the need for an icebreaker based on the analysis results, and if necessary, selects an appropriate icebreaker activity from the database.

[0084] Step 5:

[0085] The server generates a proposal for the selected icebreaker activity and transmits it to the terminal.

[0086] Step 6:

[0087] The device displays icebreaker suggestions to the user, who can then select any icebreaker activity from the suggestions.

[0088] Step 7:

[0089] The device sends the icebreaker activity selected by the user to the server, which adds it to the user's schedule.

[0090] Step 8:

[0091] When a user starts a conference, the device turns on the audio recording function, and the device collects the audio data of the conversation in real time and transmits it to the server.

[0092] Step 9:

[0093] The server converts the voice data into text data using voice recognition technology.

[0094] Step 10:

[0095] The server analyzes the converted text data and extracts important points, for example, identifying tasks such as "items that need to be reported at the next meeting."

[0096] Step 11:

[0097] The server adds new tasks and reminders to the user's calendar based on the extracted key points.

[0098] Step 12:

[0099] The server sends new schedules and reminders to the device and creates notifications, which the device displays to the user.

[0100] Step 13:

[0101] The user checks the notification and adjusts the schedule as necessary.

[0102] Example 1

[0103] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0104] In today's business environment, users need to efficiently manage numerous meetings and tasks. However, long meetings and continuous tasks often reduce users' concentration and productivity. Furthermore, there are cases where important comments or decisions made during meetings are not recorded, hindering subsequent work. There is a need for a system that can solve these issues and improve users' work efficiency.

[0105] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0106] In this invention, the server includes means for acquiring user schedule information, means for analyzing the acquired schedule information and determining the need for social activities, means for suggesting appropriate social activities, means for adding the suggested social activities to the user's schedule, means for collecting meeting audio information, means for converting the collected audio information into text data, means for extracting key points from the converted text data, means for adding new tasks and notifications to the user's schedule based on the extracted key points, and means for sending the new schedule contents and notifications to the user's terminal. This makes it possible to efficiently manage the user's work schedule and automatically record and notify important information.

[0107] The "means for acquiring user's schedule information" is a function for collecting schedule information from the user's calendar data or scheduling system.

[0108] The "means for analyzing schedule information and determining the necessity of social activities" is a function for analyzing the user's schedule based on the acquired schedule information and evaluating whether social activities are necessary.

[0109] The "means for suggesting appropriate social activities" is a function for suggesting appropriate relaxation methods and social activities to the user based on the analysis results.

[0110] The "means for adding an interaction activity to a user's schedule" is a function for automatically incorporating an interaction activity selected by the user into a schedule and adding it as a schedule.

[0111] The "means for collecting audio information from a meeting" is a function for digitally recording audio during a meeting and storing it for later processing.

[0112] The "means for converting collected voice information into text data" is a function that uses voice recognition technology to convert voice data into text data.

[0113] "Means for extracting key points from text data" is a function that uses natural language processing technology to extract important information and decisions from meeting content converted into text data.

[0114] The "means for adding new tasks and notifications to the user's schedule" is a function for automatically adding related tasks and reminders to the user's schedule based on the extracted key points.

[0115] "Means for sending new schedule details and notifications to the user's device" refers to a function for sending new schedule and reminder information to the user's device and notifying them via push notifications, etc.

[0116] The "means for checking and selecting suggested social activities on the user's terminal" is a function that allows the user to view suggested social activities through the terminal and select a desired activity.

[0117] "Means for reviewing, approving, and adjusting notifications displayed on the user's device" refers to a function that allows the user to review notifications displayed on the device, approve their contents, and change the schedule as necessary.

[0118] System configuration

[0119] This invention is a business efficiency system that integrates the acquisition and analysis of user schedule information, suggestion of social activities, automatic text conversion and key point extraction of conversation content, and scheduling and notification of new tasks and notifications. The roles and specific operations of the server, terminal, and user are explained below.

[0120] Server Operation

[0121] The server retrieves the user's schedule information from external scheduling systems such as Google Calendar or Microsoft Outlook. It analyzes the retrieved schedule information, detects consecutive meetings or tasks, and determines whether social activities are necessary. For example, if there are consecutive long meetings, it determines that social activities such as stretching or chatting are necessary in between.

[0122] Next, the server generates a list of appropriate social activities based on the analysis results and suggests them to the user's device. The social activities selected by the user are added to the schedule, or new events are added. The server also receives audio information from the meeting in real time and converts it into text data using speech recognition technology, such as the Google Cloud Speech-to-Text API. The server then extracts key points from the converted text data and identifies important information, such as what needs to be reported at the next meeting.

[0123] The server then adds new tasks and reminders to the user's calendar based on the extracted key points. For example, an item such as "Prepare for the next meeting" is automatically added. Finally, the server sends the new schedule and notifications to the user's device, notifying them via push notifications or other means.

[0124] Device behavior

[0125] The terminal displays the schedule information received from the server to the user. The user can check and select suggested social activities through the terminal. The terminal also collects audio data during the meeting in real time and sends it to the server. This audio data is recorded in digital format and can be checked later. The terminal displays notifications of new appointments and reminders received from the server to the user. The user can check, approve, and adjust the schedule from these notifications.

[0126] User operations

[0127] The user checks the schedule through the device and selects a social activity suggested by the server. The selected activity is automatically added to the schedule. When starting a meeting, the user turns on the device's audio recording function to record the conversation. After the meeting, the user can check the converted text data and extracted key points. The user can also check notifications displayed on the device and approve new tasks and reminders. The schedule can also be adjusted as necessary.

[0128] Specific examples

[0129] Example 1: Long meetings

[0130] A user enters a week's worth of events into Google Calendar, for example, one day includes three consecutive hours of meetings.

[0131] The server analyzes this and suggests appropriate social activities before, during, or after the meeting.

[0132] The user selects "5 minutes of stretching" and the server schedules it and sets up a notification.

[0133] Example 2: Recording important information during a meeting

[0134] A user starts a conference and turns on the audio recording function of the terminal. The audio during the conversation is collected by the terminal and sent to the server.

[0135] The server converts the speech into text and extracts key points.

[0136] It is determined that "Submission of market analysis report" is required before the next meeting, and this information is added to the calendar and notified to the user as a reminder.

[0137] Prompt Sentence Examples

[0138] Example prompt 1:

[0139] "Please provide a meeting schedule. Please also provide key points about your next meeting."

[0140] Example prompt 2:

[0141] "Please suggest what social activities we can add to the schedule during long meetings."

[0142] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0143] Step 1:

[0144] The server retrieves the user's schedule information from an external scheduling system (e.g., Google Calendar, Microsoft Outlook).

[0145] Input: User's calendar information (via API)

[0146] Output: A dataset of retrieved schedule information

[0147] Specific operation: The server uses an API to read the user's schedule information from an external system. For example, it uses the Google Calendar API to get the schedule information for the next week.

[0148] Step 2:

[0149] The server analyzes the schedule information it obtains and detects consecutive meetings and tasks.

[0150] Input: Dataset of acquired schedule information

[0151] Output: Analysis results (identification of consecutive meetings and tasks)

[0152] Specific operation: The server sorts the schedule information in chronological order and executes logic to check whether consecutive meetings exceed a certain time (e.g., 3 hours).

[0153] Step 3:

[0154] The server determines the need for social activities and suggests appropriate social activities.

[0155] Input: Analysis results (identification of consecutive meetings and tasks)

[0156] Output: A list of suggested suitable social activities

[0157] Specific operation: When consecutive meetings are detected, the server generates a list of suggested social activities for the user, such as "two minutes of stretching" or "five minutes of chatting."

[0158] Step 4:

[0159] The terminal displays the social activity suggestions received from the server to the user.

[0160] Input: A list of suggested social activities sent by the server

[0161] Output: A proposed interface for the user to see.

[0162] Specific operation: The terminal displays the suggested interaction activities on the user interface, allowing the user to confirm the suggestions.

[0163] Step 5:

[0164] The user selects a suggested social activity on the device.

[0165] Input: A proposed interface for user display

[0166] Output: Selected interaction activities

[0167] Specific operation: The user uses the device interface to select the desired interaction activity from the suggested activities.

[0168] Step 6:

[0169] The server adds the user's selected social activity to the schedule.

[0170] Input: Selected interaction activity

[0171] Output: Updated schedule

[0172] Specific operation: The server schedules the interaction activity selected by the user and adds it as an appointment.

[0173] Step 7:

[0174] The terminal collects audio data during the meeting in real time and sends it to the server.

[0175] Input: Real-time audio in a meeting

[0176] Output: Collected audio data (digital format)

[0177] Specific operation: The device uses a voice input device to record audio during the meeting and save it in digital format. This data is then streamed to the server.

[0178] Step 8:

[0179] The server converts the received voice data into text data.

[0180] Input: Collected audio data

[0181] Output: Text data converted from audio

[0182] Specific operation: The server uses voice recognition technology (e.g., Google Cloud Speech-to-Text API) to convert the voice data into text data.

[0183] Step 9:

[0184] The server extracts the main points from the converted text data.

[0185] Input: Text data converted from speech

[0186] Output: Extracted key information

[0187] How it works: The server uses natural language processing techniques to extract important information from the text data, such as what needs to be reported at the next meeting.

[0188] Step 10:

[0189] The server adds new tasks and reminders to the user's calendar based on the extracted key points.

[0190] Input: Extracted gist information

[0191] Output: Updated calendar information (with new tasks and reminders added)

[0192] What happens: The server adds upcoming meeting arrangements and other necessary tasks to the user's calendar.

[0193] Step 11:

[0194] The server sends new schedule details and notifications to the user terminal and notifies them.

[0195] Input: Updated calendar information

[0196] Output: Notification to user terminal

[0197] Specific operation: The server notifies the user by sending the updated schedule to the user's device via push notification or other notification method.

[0198] Step 12:

[0199] The user checks the notification on the device and approves or adjusts it if necessary.

[0200] Input: Notification to user terminal

[0201] Output: User approved and adjusted schedule

[0202] What happens: The user sees the notifications on their device, acknowledges the new task or reminder, adjusts their schedule if necessary, and confirms the changes.

[0203] (Application example 1)

[0204] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0205] Robots and machines working in factories are often subjected to continuous, long-term operation, which can result in problems such as overheating and abnormal vibrations. It can also be difficult for workers to detect abnormalities in real time and take appropriate action. This can lead to reduced production efficiency and increased risk of machine breakdowns. Furthermore, workers often do not take appropriate breaks or undergo maintenance during long periods of work, which also reduces the operating efficiency and lifespan of machines. The present invention aims to solve these problems.

[0206] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0207] In this invention, the server includes means for acquiring user schedule information, means for analyzing the acquired schedule information and determining the need for an ice breaker, means for suggesting appropriate ice breaker activities, means for adding the suggested ice breaker activities to the user's schedule, means for managing the work schedule of the industrial machinery and suggesting breaks between continuous work, and means for collecting work data of the industrial machinery in real time and detecting abnormalities. This reduces the burden on machinery in a factory due to long hours of operation and makes it possible to quickly detect and respond to abnormalities.

[0208] A "user" is someone who uses this system to improve schedule management and work efficiency.

[0209] "Schedule information" is data relating to plans or plans that are input or obtained by the user.

[0210] An "icebreaker" is an activity such as a short break or light exercise that takes place during long periods of work or meetings.

[0211] "Industrial machinery" refers to machinery and equipment used in industrial sites such as factories.

[0212] A "work schedule" is a schedule or timetable for work performed by industrial machines and workers.

[0213] A "break" is a temporary pause between successive tasks.

[0214] "Audio data" refers to digital data of conversations or audio recordings.

[0215] "Text data" refers to digital data converted into character information.

[0216] A "gist" is a particularly important part of a conversation or piece of information.

[0217] "Abnormal" means any operation or condition of a machine that deviates from normal operating conditions.

[0218] "Real-time" refers to data collection and processing occurring immediately, without delay.

[0219] "Notification" means informing the user of new information or alerts.

[0220] A "smart device" is a mobile device such as a smartphone or tablet that can connect to the Internet and run applications.

[0221] System Configuration

[0222] This invention is a business efficiency improvement system that integrates the acquisition and analysis of user schedule information, ice-breaker suggestions, automatic text conversion and key point extraction of conversation content, schedule creation and notification. This system is mainly composed of three elements: a server, a terminal, and a user.

[0223] Server Operation

[0224] The server operates using the following methods:

[0225] 1. Acquisition and analysis of schedule information

[0226] The server obtains the user's schedule information from an external scheduling system (e.g., Google Calendar, Microsoft Outlook API). It analyzes the obtained schedule information and determines whether an icebreaker is necessary. For example, if a long meeting or continuous work on industrial machinery is scheduled, it determines that an icebreaker is necessary in between.

[0227] 2. Propose and confirm icebreakers

[0228] The server generates a list of ice-breaking activities (e.g., two minutes of stretching, five minutes of maintenance check) based on the analysis results and suggests them to the user's device. The ice-breaking activities selected by the user are added to the schedule.

[0229] 3. Conversation text conversion and key points extraction

[0230] The server receives the audio data from the meeting and converts it into text using speech recognition software such as the Google Speech-to-Text API. The server then extracts key points from the converted text.

[0231] 4. Schedule Task Reminders

[0232] Based on the extracted key points, new tasks and reminders are added to the user's calendar using data analysis techniques using Python and the pandas library.

[0233] 5. Industrial Machinery Data Collection and Anomaly Detection

[0234] The server collects operational data (e.g., temperature, vibration, and operating time) from industrial machinery in real time and detects abnormalities. This data collection and analysis is done using Python and the pandas library. If an abnormality is detected, an alert is sent to the worker's smart device.

[0235] Device behavior

[0236] The terminal operates using the following means:

[0237] 1. Receiving and displaying schedule information

[0238] The terminal displays the schedule information received from the server to the user, who can then check and select the icebreaker activities suggested through the terminal.

[0239] 2. Collection of audio data

[0240] The terminal collects audio data during the meeting in real time and sends it to a server. The audio data is also recorded in digital format and can be viewed later.

[0241] 3. Display of notifications

[0242] The device displays notifications of new schedules and reminders received from the server to the user, who can then review, approve, and adjust the schedule.

[0243] User operations

[0244] The user interacts with the device using the following means:

[0245] 1. Check and select your schedule

[0246] The user checks the schedule through their device and selects an icebreaker activity suggested by the server. The selected activity is automatically added to the schedule.

[0247] 2. Record and review conversations

[0248] When starting a meeting, users can turn on the device's audio recording function to record the conversation. After the meeting, they can check the converted text data and extracted key points.

[0249] 3. Review and adjust notifications

[0250] Users can view notifications on their device, acknowledge new tasks and reminders, and adjust their schedules as needed.

[0251] Specific examples

[0252] The following are specific examples for carrying out the present invention:

[0253] Example 1: Long meetings and continuous work

[0254] For example, if a user has a three-hour continuous meeting or continuous work schedule for an industrial machine, the server analyzes the schedule and suggests suitable icebreakers (e.g., five-minute stretching or maintenance check) before, during, or after the meeting or work. The icebreaker activities selected by the user are incorporated into the schedule and notifications are set.

[0255] Example 2: Recording important information during meetings and detecting anomalies

[0256] A user starts a meeting and turns on the audio recording function on their device. The audio during the conversation is collected by the device and sent to the server. The server converts the audio into text and extracts key points. It determines that a "market analysis report" needs to be submitted before the next meeting, and this information is added to the calendar and notified to the user as a reminder. Furthermore, if an industrial machine detects abnormal vibrations, the server will issue a warning to the worker's smart device.

[0257] Prompt Sentence Examples

[0258] As a concrete example, consider the following prompt input to a generative AI model:

[0259] Enter the following data: a list of work schedules, activities, icebreaker suggestions, and work data (temperature, vibration, etc.). For example, "2023-10-10 09:00-12:00: Work A, 2023-10-10 12:00-13:00: Break."

[0260] The system suggests an icebreaker: "5 minutes of maintenance after 20 minutes of work." It analyzes data and issues an alert if it detects an abnormality.

[0261] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0262] Step 1:

[0263] The server retrieves the user's schedule information using the API of an external scheduling system (e.g., Google Calendar or Microsoft Outlook), which requires an API key and a calendar ID as input. The output is the retrieved schedule information.

[0264] Step 2:

[0265] The server analyzes the schedule information it has acquired and detects long meetings and work. The input is the schedule information acquired in step 1, and the data is analyzed to extract events with long meetings or continuous work. This analysis identifies locations where ice breakers are needed. The output is the analysis results, i.e., a list of events where ice breakers are needed.

[0266] Step 3:

[0267] The server proposes ice-breaking activities (e.g., 2 minutes of stretching, 5 minutes of maintenance check) based on the analysis results. The input is the list of events requiring ice-breaking obtained in step 2. The server assigns ice-breaking activities to these events and generates a list of activities to propose to the user. The output is a list of proposed ice-breaking activities.

[0268] Step 4:

[0269] The terminal displays the list of icebreaker activities received from the server to the user. The input is the list of icebreaker activities generated in step 3, and the terminal displays it on the user's interface so that the user can select one. The output is the icebreaker activity selected by the user.

[0270] Step 5:

[0271] The server adds the icebreaker activity selected by the user to the schedule and notifies the user. The input is the icebreaker activity selected by the user in step 4, the server incorporates it into the schedule, and generates a new schedule. The output is the updated schedule information.

[0272] Step 6:

[0273] The terminal collects audio data during the meeting in real time and sends it to the server. The input is the audio data recorded during the meeting, which the terminal records in digital format and sends to the server. The output is the audio data sent to the server.

[0274] Step 7:

[0275] The server converts the audio data into text data and extracts the main points. The input is the audio data sent in step 6, and the server converts the audio data into text data using the Google Speech-to-Text API. It then performs natural language processing to extract the main points from the text. The output is a list of main points.

[0276] Step 8:

[0277] The server adds new tasks and reminders to the user's schedule based on the extracted key points. The input is the list of key points generated in step 7, and the server automatically adds new tasks and reminders to the schedule based on this. The output is the updated schedule information.

[0278] Step 9:

[0279] The server collects operational data from industrial machinery in real time and detects abnormalities. The input is the operational data of the industrial machinery (such as temperature, vibration, operating time, etc.), which the server monitors in real time and analyzes abnormalities. The output is an alert when an abnormality is detected.

[0280] Step 10:

[0281] When the server detects an abnormality, it issues a warning to the worker's smart device. The input is the abnormal data detected in step 9, and a warning message is generated based on that content and notified to the worker's smart device. The output is the notification sent to the worker.

[0282] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0283] System configuration

[0284] This invention is a business efficiency improvement system that integrates the acquisition and analysis of user schedule information, ice-breaker suggestions, automatic text conversion and key point extraction of conversation content, schedule creation and notifications, and also incorporates an emotion engine. The roles and specific operations of the server, terminal, and user are explained below.

[0285] Server Operation

[0286] 1. Acquisition and analysis of schedule information

[0287] The server retrieves the user's schedule information from an external scheduling system (e.g., Google Calendar, Microsoft Outlook). It analyzes the retrieved schedule information and determines whether an icebreaker is necessary. For example, if there are consecutive long meetings, it determines that an icebreaker is necessary.

[0288] 2. Propose and confirm icebreakers

[0289] Based on the analysis results, the server generates a list of ice-breaking activities (e.g., two minutes of stretching, five minutes of chatting) and suggests them to the user's device. The ice-breaking activities selected by the user are added to the schedule. Furthermore, the selection of ice-breaking activities is optimized based on the analysis results of the emotion engine.

[0290] 3. Conversation text conversion and key points extraction

[0291] The server receives the voice data from the meeting and converts it into text using speech recognition technology. Key points (e.g., matters that need to be reported at the next meeting) are extracted from the converted text data. The emotion engine also recognizes the user's emotions in real time during the conversation and adjusts the extraction of key points and the addition of tasks according to their emotional state.

[0292] 4. Schedule Task Reminders

[0293] Based on the extracted key points, new tasks and reminders are added to the user's calendar. For example, an item such as "Prepare for the next meeting" is automatically added. The emotion engine analyzes the user's stress and fatigue levels and dynamically adjusts the content and timing of reminders based on the results.

[0294] 5. Notification

[0295] The server sends new schedule information and reminders to the user's device, notifying the user. The user checks the notification and adjusts the schedule as necessary.

[0296] Device behavior

[0297] 1. Receiving and displaying schedule information

[0298] The terminal displays the schedule information received from the server to the user, who can then check and select the icebreaker activities suggested through the terminal.

[0299] 2. Collection of audio data

[0300] The device collects audio data during the meeting in real time and sends it to a server. The audio data is also recorded in digital format and can be viewed later.

[0301] 3. Use of Emotion Engine

[0302] The device analyzes the user's facial expressions and voice tone and uses an emotion engine to analyze the user's emotional state in real time. The analysis results of the emotion engine are sent to a server and used to suggest icebreakers, extract key points, and add tasks.

[0303] 4. Display of notifications

[0304] The device displays notifications of new schedules and reminders received from the server to the user, who can then review, approve, and adjust the schedule.

[0305] User operations

[0306] 1. Check and select your schedule

[0307] The user checks the schedule on their device and selects an icebreaker activity suggested by the server. The selected activity is automatically added to the schedule.

[0308] 2. Record and review conversations

[0309] When a user starts a meeting, they can turn on the device's audio recording function to record the conversation. After the meeting, they can check the converted text data and extracted key points.

[0310] 3. Review and adjust notifications

[0311] Users can view notifications on their device, acknowledge new tasks and reminders, and adjust their schedules as needed.

[0312] Specific examples

[0313] Example 1: Long meetings

[0314] A user enters a week's schedule, including a three-hour series of meetings.

[0315] The server analyzes this and suggests appropriate icebreakers before, during, or after the meeting. The emotion engine analyzes the user's fatigue level and recommends icebreakers to relieve stress.

[0316] The user selects "5 minutes of stretching" and the server schedules it and sets up a notification.

[0317] Example 2: Recording important information during a meeting

[0318] A user starts a conference and turns on the audio recording function of the terminal. The audio during the conversation is collected by the terminal and sent to the server.

[0319] The server converts the speech into text and extracts key points, while an emotion engine analyzes the user's emotions and adjusts the key points taking into account the importance and urgency of the topic.

[0320] It is determined that "Submission of market analysis report" is required before the next meeting, and this information is added to the calendar and notified to the user as a reminder.

[0321] As described above, the present invention improves the efficiency of user schedule management and the recording and analysis of conversation content, and by combining it with an emotion engine, it provides optimal business support based on the user's emotional state.

[0322] The processing flow will be explained below.

[0323] Step 1:

[0324] The user accesses the tool and starts the schedule management service.

[0325] Step 2:

[0326] The device retrieves the user's calendar data from an external scheduling service such as Google Calendar or Microsoft Outlook, and then sends this information to the server.

[0327] Step 3:

[0328] The server analyzes the received schedule data and identifies when an icebreaker is needed, such as during a long meeting or when consecutive tasks are required.

[0329] Step 4:

[0330] The server determines the need for an icebreaker based on the analysis results, and if necessary, selects an appropriate icebreaker activity from the database.

[0331] Step 5:

[0332] The server generates a proposal for the selected icebreaker activity and transmits it to the terminal.

[0333] Step 6:

[0334] The device displays icebreaker suggestions to the user, who can then select any icebreaker activity from the suggestions.

[0335] Step 7:

[0336] The device sends the icebreaker activity selected by the user to the server, which adds it to the user's schedule.

[0337] Step 8:

[0338] When a user starts a conference, the device turns on the audio recording function, and the device collects the audio data of the conversation in real time and transmits it to the server.

[0339] Step 9:

[0340] The server converts the voice data into text data using voice recognition technology.

[0341] Step 10:

[0342] The server analyzes the converted text data and extracts important points, for example, identifying tasks such as "items that need to be reported at the next meeting."

[0343] Step 11:

[0344] The server adds new tasks and reminders to the user's calendar based on the extracted key points.

[0345] Step 12:

[0346] The server sends new schedules and reminders to the device and creates notifications, which the device displays to the user.

[0347] Step 13:

[0348] The user checks the notification and adjusts the schedule as necessary.

[0349] Example 2

[0350] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0351] Conventional schedule management systems do not provide effective ways to reduce fatigue and stress caused by continuous meetings and tasks. Furthermore, the efficient recording of important comments and decisions during meetings and the manual task management based on these records result in reduced work efficiency. Furthermore, the lack of timely reminders and notifications that take into account the user's emotional state makes it difficult for users to maintain their productivity.

[0352] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for acquiring user schedule information, means for analyzing the acquired schedule information and determining the need for an icebreaker, means for proposing appropriate icebreaker activities, means for adding the proposed icebreaker activities to the user's schedule, means for analyzing the user's emotional state and optimizing the icebreaker activities, means for collecting voice data of the conversation, means for converting the collected voice data into text data, means for extracting key points from the converted text data, means for adding new tasks or reminders to the user's schedule based on the extracted key points, and means for sending and notifying the new schedule or reminder to the user terminal. This effectively reduces fatigue and stress caused by continuous meetings and tasks for the user, efficiently records important comments and decisions made during meetings, automates task management based on the records, and enables appropriate timing of reminders and notifications taking the user's emotional state into consideration.

[0353] "Schedule information" refers to data relating to the user's planned dates and activities.

[0354] "Acquisition means" refers to the methods or functions for collecting data or information from external or internal systems.

[0355] "Analysis means" refers to the methods and functions used to analyze acquired data and information and understand its content and meaning.

[0356] "Ice-breaking activities" are refreshing activities or short recreational activities to relieve fatigue and stress caused by long meetings or task completion.

[0357] "Suggestion means" refers to methods or functions for presenting appropriate information or activities to users.

[0358] "Additional means" refers to methods or functions for inserting new information or activities into a user's schedule.

[0359] "Emotional state" refers to the mental state or mood of the user.

[0360] "Audio data" refers to data that has been recorded in digital form as a conversation or voice signal.

[0361] "Text data" refers to data obtained by converting voice data into character information.

[0362] "Key point extraction means" refers to a method or function for extracting important information or key content from text data.

[0363] A "task" is a specific task or job that a user must undertake.

[0364] A "reminder" is a notification or alarm that reminds a user of a specific date, time, or event.

[0365] "Notification means" refers to the method or function for notifying the user of new information or reminders.

[0366] A "terminal" is a computing device that is directly operated by a user.

[0367] A "server" is a central computer system that processes and manages data on a network.

[0368] This is a business efficiency improvement system that integrates the acquisition and analysis of user schedule information, ice-breaker suggestions, automatic text conversion and key point extraction of conversation content, and task and reminder scheduling and notification. Furthermore, it incorporates an emotion engine to provide optimal support based on the user's emotional state.

[0369] System Configuration

[0370] Server Operation

[0371] The server first obtains the user's schedule information from an external scheduling system (e.g., Google Calendar or Microsoft Outlook) via API. The obtained schedule information is stored in an internal database, and an analysis engine uses this information to determine whether an icebreaker is necessary.

[0372] Next, based on the analysis results, the server lists multiple ice-breaking activities (e.g., two minutes of stretching, five minutes of chatting) and sends this list in JSON format to the user's device. At the same time, the emotion engine analyzes the user's stress level and fatigue level to select the most appropriate ice-breaking activity. The server adds the ice-breaking activity selected by the user to the schedule and notifies the user's device again.

[0373] During the meeting, the server receives real-time voice data from the devices and converts it into text data using the Google Speech-to-Text API. The converted text data is stored in a database, and a natural language processing (NLP) engine extracts key points. An emotion engine analyzes the user's emotions and adjusts the importance of the key points.

[0374] Furthermore, new tasks and reminders are added to the user's calendar based on the extracted key points. The emotion engine dynamically adjusts the content and timing of reminders, taking into account the user's stress level and fatigue level. The server then sends the new schedule and reminders to the user's device and notifies the user.

[0375] Device behavior

[0376] The device displays the schedule information received from the server to the user via a GUI. The user can then review and select suggested icebreaker activities. During the meeting, the device's microphone collects audio data in real time, digitizes it, and sends it to the server. This audio data is also stored locally and can be played back later.

[0377] The device uses its built-in camera and microphone to analyze the user's facial expressions and voice tone and input them into the emotion engine. The analysis results are sent to the server and reflected in icebreaker suggestions and key point extraction. In addition, the device receives new schedule and reminder notifications from the server and displays them as push notifications, allowing the user to confirm, approve, or adjust them.

[0378] User operations

[0379] The user checks the schedule on their device and selects an icebreaker activity suggested by the server. The selected activity is automatically added to the schedule. When starting a meeting, the user turns on the device's audio recording function to record the conversation. After the meeting, the converted text data and extracted key points can be viewed. The user can also check notifications displayed on the device, approve new tasks and reminders, and adjust the schedule as necessary.

[0380] Specific examples

[0381] Example 1: Long meetings

[0382] A user enters a week's worth of events into Google Calendar, and one day has three consecutive meetings scheduled.

[0383] The server analyzes this and suggests appropriate icebreakers before, during, or after the meeting. The emotion engine analyzes the user's fatigue level and recommends "5 minutes of stretching" to relieve stress.

[0384] The user selects "5 minutes of stretching" and the server schedules it and sets up a notification.

[0385] Example 2: Recording important information during a meeting

[0386] A user starts a conference and turns on the audio recording function of the device. The audio during the conversation is collected by the device and transmitted to the server in real time.

[0387] The server converts the speech into text and extracts key points. The emotion engine analyzes the user's emotions and adjusts the key points based on the importance and urgency of the topic.

[0388] It turns out that "Submission of Market Analysis Report" is required before the next meeting, and this information is added to the calendar and notified to the user as a reminder.

[0389] Prompt Sentence Examples

[0390] "Use your schedule and sentiment data to suggest the best icebreaker activities, extract key points from meetings, and add upcoming tasks to your schedule."

[0391] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0392] Step 1: Get schedule information

[0393] The server gets information from an external scheduling system

[0394] Input: User authentication information, API key

[0395] Output: User schedule information (JSON format)

[0396] How it works: The server uses the APIs of Google Calendar and Microsoft Outlook to retrieve user schedule information. This information is sent to the server in JSON format and stored in an internal database.

[0397] Step 2: Analyze schedule information

[0398] The server analyzes the schedule information and determines whether an icebreaker is necessary.

[0399] Input: Schedule information (JSON format)

[0400] Output: Analysis results (flag indicating need for icebreaker)

[0401] How it works: The server's analysis engine analyzes schedule information and determines whether an icebreaker is necessary if there are consecutive "meetings" or "long-term tasks." For example, if the interval between consecutive meetings exceeds a certain time, it determines that an icebreaker is necessary.

[0402] Step 3: Suggest an icebreaker activity

[0403] Your server will suggest appropriate icebreaker activities

[0404] Input: Analysis results, emotion engine data

[0405] Output: Icebreaker proposal list (JSON format)

[0406] How it works: The server suggests icebreaker activities suitable for the user based on the analysis results and emotion engine data. The list of suggestions is sent to the device in JSON format.

[0407] Step 4: Determine icebreaker activities

[0408] The server adds icebreaker activities to the schedule based on the user's selection.

[0409] Input: User's choice

[0410] Output: Updated schedule information

[0411] How it works: After a user selects an icebreaker activity on the device, the server adds the selection to the schedule and sends the updated schedule information back to the device.

[0412] Step 5: Collecting audio data from the conversation

[0413] The device collects audio data during the meeting and sends it to the server.

[0414] Input: Audio during the meeting

[0415] Output: Digital audio data

[0416] How it works: The device's microphone collects audio during the meeting in real time and sends it digitally to a server, where the audio data is stored locally.

[0417] Step 6: Convert audio data to text

[0418] The server converts the audio data into text data.

[0419] Input: Audio data

[0420] Output: Text data

[0421] How it works: The server uses the Google Speech-to-Text API to convert the received audio data into text data, which is then stored in a database.

[0422] Step 7: Extracting key points

[0423] The server extracts key points from the text data

[0424] Input: Text data

[0425] Output: Gist list

[0426] How it works: The server's natural language processing (NLP) engine analyzes the text data and extracts key conversation points and action items. The emotion engine also analyzes the user's emotions at this stage and adjusts the importance of key points.

[0427] Step 8: Schedule task reminders

[0428] The server adds new tasks and reminders to your schedule

[0429] Input: Key points list, emotion engine data

[0430] Output: Updated schedule

[0431] How it works: Based on the extracted key points, the server adds new tasks and reminders to the user's calendar. The emotion engine dynamically adjusts the timing of reminders based on the user's stress and fatigue levels.

[0432] Step 9: Notification

[0433] The server sends new schedules and reminders to the user's device and notifies them.

[0434] Input: Updated Schedule

[0435] Output: Notification message

[0436] How it works: The server sends updated schedule information and reminders to the device, which are displayed to the user as push notifications. The user can view the notifications and accept or adjust them as needed.

[0437] (Application example 2)

[0438] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0439] In modern factories and offices, workers often find it difficult to take appropriate breaks or refresh themselves when working long hours continuously or in the middle of important tasks. Furthermore, systems for accurately recording the work performed by workers and later reviewing and utilizing this information are often inadequate. Furthermore, to reduce workers' stress and fatigue, it is necessary to provide optimal intervals based on their emotional state. These challenges can reduce work efficiency and ultimately hinder improvements in productivity and work quality. A solution to these problems and achieve both work efficiency and worker health is needed.

[0440] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0441] In this invention, the server includes means for acquiring a user's schedule information, means for analyzing the acquired schedule information and determining the need for a break, means for suggesting appropriate break activities, means for adding the suggested break activities to the user's schedule, means for analyzing the user's emotional state using an emotion engine and optimizing the selection of break activities, means for collecting voice data during work and converting it into text data, means for extracting important work points from the converted text data, means for adding new tasks and reminders to the user's schedule based on the extracted points, and means for displaying notifications to the user and allowing the user to confirm, approve, and adjust the suggested break activities and added tasks and reminders. This enables workers to take breaks at appropriate times, maximize work efficiency, and accurately record and manage important work content and key points.

[0442] "User" refers to the workers and employees who use the system.

[0443] "Setup information" refers to information that indicates the work schedules, plans, and tasks of workers or employees.

[0444] "Emotion engine" refers to technology that analyzes a user's facial expressions and vocal tone to recognize and analyze their emotional state (e.g., stress or fatigue) in real time.

[0445] "Rest activities" refer to short breaks or refreshing activities (e.g., short stretching or chatting) that users take between tasks to reduce fatigue and stress.

[0446] "Voice data" refers to digital data of a user's voice collected while working or talking.

[0447] "Text data" refers to sentence data obtained by converting voice data into characters.

[0448] "Key points" are pieces of information or tasks that are considered particularly important in a work progress or conversation.

[0449] A "task" refers to a discrete task or activity that a user must perform.

[0450] A "reminder" is a notification or reminder that helps users remember specific tasks or appointments.

[0451] This system acquires setup information for workers and employees, proposes and schedules optimal break activities to improve work efficiency, and automatically records and manages important work content and key points of conversations. The roles and specific operations of the server, terminal, and user are explained below.

[0452] Server Operation

[0453] Acquisition and analysis of setup information

[0454] The server acquires setup information for workers and employees from the factory management system or business management system (MES or ERP system). It analyzes the acquired setup information and determines whether breaks are necessary. For example, if a long period of continuous work is scheduled, it determines that a break is necessary in between.

[0455] Proposing and confirming break activities

[0456] Based on the analysis results, the server generates a list of break activities (e.g., short breaks and stretching) and suggests them to the user's device. It uses an emotion engine to analyze the user's emotional state and selects the optimal break activity based on the results. The break activity selected by the user is added to the schedule.

[0457] Speech data text conversion and gist extraction

[0458] The server receives the working voice data and converts it into text using speech recognition technology (e.g., the speech_recognition library). It then uses a generative AI model (e.g., GPT-3.5) to extract key points from the converted text. The extraction of key points is optimized according to the analysis results of the emotion engine.

[0459] Schedule tasks and reminders

[0460] The server adds new tasks and reminders to the user's schedule based on the extracted key points. For example, information such as "Parts for the next process are missing" is added to the schedule and notified to the user as a reminder.

[0461] notification

[0462] The server sends new schedule information and reminders to the user's device, and notifies the user. The user can check the notification and adjust the schedule as necessary.

[0463] Device behavior

[0464] Receiving and displaying setup information

[0465] The terminal displays the schedule information received from the server to the user, who can then check and select the suggested break activities through the terminal.

[0466] Audio data collection

[0467] The device collects voice data in real time while the worker is working and sends it to a server, where it is recorded in digital format and can be viewed later.

[0468] Use of emotion engine

[0469] The device analyzes the user's facial expressions and voice tone and uses an emotion engine to analyze the user's emotional state in real time. The analysis results of the emotion engine are sent to a server and used to suggest break activities, extract key points, and add tasks.

[0470] Viewing notifications

[0471] The device displays notifications of new schedules and reminders received from the server to the user, who can then review, approve, and adjust the schedule.

[0472] User operations

[0473] Check and select setup

[0474] The user checks the schedule through the terminal and selects the break activity suggested by the server, which is then automatically added to the schedule.

[0475] Recording and checking work details

[0476] When a user starts a task, they can turn on the device's voice recording function to record the task's progress. After the task is completed, they can check the converted text data and extracted key points.

[0477] Review and adjust notifications

[0478] Users can view notifications on their device, acknowledge new tasks and reminders, and adjust their schedules as needed.

[0479] Specific examples

[0480] Example 1: Long working hours

[0481] The user enters their weekly schedule, and one day is scheduled for six hours of continuous robot operation.

[0482] The server analyzes this and suggests appropriate rest activities before, during, or after the operation. The emotion engine analyzes the user's fatigue level and recommends short breaks to relieve stress.

[0483] The user selects "5 minutes of stretching" and the server schedules it and sets up a notification.

[0484] Example 2: Recording important information during work

[0485] The user starts working and turns on the device's voice recording function. The voice recorded during the work is collected by the device and sent to the server.

[0486] The server converts the speech into text and extracts key points, while the emotion engine analyzes the user's emotions and optimizes the key points in real time.

[0487] If a "parts shortage" is discovered by the time the next process is completed, this information is added to the schedule and a reminder is sent to the user.

[0488] Example of a generative AI model prompt:

[0489] Prompt Sentence Examples

[0490] Text from audio recording: "We've encountered a parts supply issue. We are running low on parts needed for the next operation. We need to reconsider our parts supply plan."

[0491] Gist Extraction Generation AI Prompt: "Please extract the key points from the audio recording above."

[0492] Key point: "Parts supply problem. Parts for the next process are in short supply. Parts supply plan needs to be reconsidered."

[0493] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0494] Step 1:

[0495] The server obtains setup information for workers and employees from the factory management system or business management system (MES or ERP system). This setup information includes work schedules, plans, and tasks. It analyzes this information and evaluates the length and continuity of work time. It receives the worker's setup information (schedule, plan, tasks) as input and determines the need for breaks as a result of the analysis.

[0496] Step 2:

[0497] The server generates a list of rest activities based on the analysis results. It uses the emotion engine to analyze the user's emotional state and selects the optimal rest activity taking the results into consideration. For example, if the user has been working for a long time or is feeling emotional stress, it suggests stretching or a short break. The server uses the analysis results and emotion engine data as input and generates a list of rest activities as output.

[0498] Step 3:

[0499] The terminal receives the list of break activities sent from the server and displays it to the user. The user checks the proposed break activities through the terminal and selects one. The terminal receives the list of break activities as input and sends the selected break activity to the server as output.

[0500] Step 4:

[0501] The server adds the break activity selected by the user to the setup information and updates the schedule. This updated schedule information is sent to the user's terminal and notified. The server receives the break activity selected by the user as input, generates updated schedule information as output, and sends it to the terminal.

[0502] Step 5:

[0503] When a user starts working, the terminal turns on the voice recording function, collects voice data during work, and sends this data to the server. The terminal collects the user's voice data as input and sends the voice data to the server as output.

[0504] Step 6:

[0505] The server converts the received voice data into text data using speech recognition technology (e.g., the speech_recognition library), and extracts key points from the converted text data using a generative AI model (e.g., GPT-3.5). It receives voice data as input and generates text data and key points as output.

[0506] Step 7:

[0507] The server adds new tasks and reminders to the schedule information based on the extracted key points. This information is sent to the user's terminal and displayed as a notification. As input, the server generates new tasks and reminders based on the extracted key points, and as output, adds these to the schedule and sends them to the terminal.

[0508] Step 8:

[0509] The user can check the notifications displayed on the device, acknowledge new tasks and reminders, and adjust the schedule as needed. The input is the notification displayed by the device, and the output is the necessary adjustment.

[0510] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0511] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0512] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0513] [Second embodiment]

[0514] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0515] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0516] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0517] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0518] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0519] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0520] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0521] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0522] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0523] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0524] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0525] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0526] System configuration

[0527] This invention is a business efficiency improvement system that integrates the acquisition and analysis of user schedule information, ice-breaker suggestions, automatic text conversion and key point extraction of conversation content, schedule creation and notification. The roles and specific operations of the server, terminal, and user are explained below.

[0528] Server Operation

[0529] 1. Acquisition and analysis of schedule information

[0530] The server retrieves the user's schedule information from an external scheduling system (e.g., Google Calendar, Microsoft Outlook). It analyzes the retrieved schedule information and determines whether an icebreaker is necessary. For example, if there are consecutive long meetings, it determines that an icebreaker is necessary.

[0531] 2. Propose and confirm icebreakers

[0532] Based on the analysis results, the server generates a list of ice-breaking activities (e.g., two minutes of stretching, five minutes of chatting) and suggests them to the user's device. The ice-breaking activity selected by the user is added to the schedule.

[0533] 3. Conversation text conversion and key points extraction

[0534] The server receives the voice data from the meeting and converts it into text data using speech recognition technology. Key points (e.g., matters that need to be reported at the next meeting) are extracted from the converted text data.

[0535] 4. Schedule Task Reminders

[0536] Based on the extracted key points, new tasks and reminders are added to the user's calendar, for example, an item such as "Prepare for the next meeting" is automatically added.

[0537] 5. Notification

[0538] The server sends new schedule details and reminders to the user's device, notifying the user. The user checks the notifications and adjusts the schedule as necessary.

[0539] Device behavior

[0540] 1. Receiving and displaying schedule information

[0541] The terminal displays the schedule information received from the server to the user, who can then check and select the icebreaker activities suggested through the terminal.

[0542] 2. Collection of audio data

[0543] The device collects audio data during the meeting in real time and sends it to a server. The audio data is also recorded in digital format and can be viewed later.

[0544] 3. Display of notifications

[0545] The device displays notifications of new schedules and reminders received from the server to the user, who can then review, approve, and adjust the schedule.

[0546] User operations

[0547] 1. Check and select your schedule

[0548] The user checks the schedule on their device and selects an icebreaker activity suggested by the server. The selected activity is automatically added to the schedule.

[0549] 2. Record and review conversations

[0550] When a user starts a meeting, they can turn on the device's audio recording function to record the conversation. After the meeting, they can check the converted text data and extracted key points.

[0551] 3. Review and adjust notifications

[0552] Users can view notifications on their device, acknowledge new tasks and reminders, and adjust their schedules as needed.

[0553] Specific examples

[0554] Example 1: Long meetings

[0555] A user enters a week's schedule, including a three-hour series of meetings.

[0556] The server analyzes this and suggests appropriate icebreakers before, during, or after the meeting.

[0557] The user selects "5 minutes of stretching" and the server schedules it and sets up a notification.

[0558] Example 2: Recording important information during a meeting

[0559] A user starts a conference and turns on the audio recording function of the terminal. The audio during the conversation is collected by the terminal and sent to the server.

[0560] The server converts the speech into text and extracts key points.

[0561] It is determined that "Submission of market analysis report" is required before the next meeting, and this information is added to the calendar and notified to the user as a reminder.

[0562] As described above, the present invention is a useful system that improves the efficiency of users' schedule management and the recording and analysis of conversation content, thereby improving work efficiency.

[0563] The processing flow will be explained below.

[0564] Step 1:

[0565] The user accesses the tool and starts the schedule management service.

[0566] Step 2:

[0567] The device retrieves the user's calendar data from an external scheduling service such as Google Calendar or Microsoft Outlook, and then sends this information to the server.

[0568] Step 3:

[0569] The server analyzes the received schedule data and identifies when an icebreaker is needed, such as during a long meeting or when consecutive tasks are required.

[0570] Step 4:

[0571] The server determines the need for an icebreaker based on the analysis results, and if necessary, selects an appropriate icebreaker activity from the database.

[0572] Step 5:

[0573] The server generates a proposal for the selected icebreaker activity and transmits it to the terminal.

[0574] Step 6:

[0575] The device displays icebreaker suggestions to the user, who can then select any icebreaker activity from the suggestions.

[0576] Step 7:

[0577] The device sends the icebreaker activity selected by the user to the server, which adds it to the user's schedule.

[0578] Step 8:

[0579] When a user starts a conference, the device turns on the audio recording function, and the device collects the audio data of the conversation in real time and transmits it to the server.

[0580] Step 9:

[0581] The server converts the voice data into text data using voice recognition technology.

[0582] Step 10:

[0583] The server analyzes the converted text data and extracts important points, for example, identifying tasks such as "items that need to be reported at the next meeting."

[0584] Step 11:

[0585] The server adds new tasks and reminders to the user's calendar based on the extracted key points.

[0586] Step 12:

[0587] The server sends new schedules and reminders to the device and creates notifications, which the device displays to the user.

[0588] Step 13:

[0589] The user checks the notification and adjusts the schedule as necessary.

[0590] Example 1

[0591] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0592] In today's business environment, users need to efficiently manage numerous meetings and tasks. However, long meetings and continuous tasks often reduce users' concentration and productivity. Furthermore, there are cases where important comments or decisions made during meetings are not recorded, hindering subsequent work. There is a need for a system that can solve these issues and improve users' work efficiency.

[0593] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0594] In this invention, the server includes means for acquiring user schedule information, means for analyzing the acquired schedule information and determining the need for social activities, means for suggesting appropriate social activities, means for adding the suggested social activities to the user's schedule, means for collecting meeting audio information, means for converting the collected audio information into text data, means for extracting key points from the converted text data, means for adding new tasks and notifications to the user's schedule based on the extracted key points, and means for sending the new schedule contents and notifications to the user's terminal. This makes it possible to efficiently manage the user's work schedule and automatically record and notify important information.

[0595] The "means for acquiring user's schedule information" is a function for collecting schedule information from the user's calendar data or scheduling system.

[0596] The "means for analyzing schedule information and determining the necessity of social activities" is a function for analyzing the user's schedule based on the acquired schedule information and evaluating whether social activities are necessary.

[0597] The "means for suggesting appropriate social activities" is a function for suggesting appropriate relaxation methods and social activities to the user based on the analysis results.

[0598] The "means for adding an interaction activity to a user's schedule" is a function for automatically incorporating an interaction activity selected by the user into a schedule and adding it as a schedule.

[0599] The "means for collecting audio information from a meeting" is a function for digitally recording audio during a meeting and storing it for later processing.

[0600] The "means for converting collected voice information into text data" is a function that uses voice recognition technology to convert voice data into text data.

[0601] "Means for extracting key points from text data" is a function that uses natural language processing technology to extract important information and decisions from meeting content converted into text data.

[0602] The "means for adding new tasks and notifications to the user's schedule" is a function for automatically adding related tasks and reminders to the user's schedule based on the extracted key points.

[0603] "Means for sending new schedule details and notifications to the user's device" refers to a function for sending new schedule and reminder information to the user's device and notifying them via push notifications, etc.

[0604] The "means for checking and selecting suggested social activities on the user's terminal" is a function that allows the user to view suggested social activities through the terminal and select a desired activity.

[0605] "Means for reviewing, approving, and adjusting notifications displayed on the user's device" refers to a function that allows the user to review notifications displayed on the device, approve their contents, and change the schedule as necessary.

[0606] System configuration

[0607] This invention is a business efficiency system that integrates the acquisition and analysis of user schedule information, suggestion of social activities, automatic text conversion and key point extraction of conversation content, and scheduling and notification of new tasks and notifications. The roles and specific operations of the server, terminal, and user are explained below.

[0608] Server Operation

[0609] The server retrieves the user's schedule information from external scheduling systems such as Google Calendar or Microsoft Outlook. It analyzes the retrieved schedule information, detects consecutive meetings or tasks, and determines whether social activities are necessary. For example, if there are consecutive long meetings, it determines that social activities such as stretching or chatting are necessary in between.

[0610] Next, the server generates a list of appropriate social activities based on the analysis results and suggests them to the user's device. The social activities selected by the user are added to the schedule, or new events are added. The server also receives audio information from the meeting in real time and converts it into text data using speech recognition technology, such as the Google Cloud Speech-to-Text API. The server then extracts key points from the converted text data and identifies important information, such as what needs to be reported at the next meeting.

[0611] The server then adds new tasks and reminders to the user's calendar based on the extracted key points. For example, an item such as "Prepare for the next meeting" is automatically added. Finally, the server sends the new schedule and notifications to the user's device, notifying them via push notifications or other means.

[0612] Device behavior

[0613] The terminal displays the schedule information received from the server to the user. The user can check and select suggested social activities through the terminal. The terminal also collects audio data during the meeting in real time and sends it to the server. This audio data is recorded in digital format and can be checked later. The terminal displays notifications of new appointments and reminders received from the server to the user. The user can check, approve, and adjust the schedule from these notifications.

[0614] User operations

[0615] The user checks the schedule through the device and selects a social activity suggested by the server. The selected activity is automatically added to the schedule. When starting a meeting, the user turns on the device's audio recording function to record the conversation. After the meeting, the user can check the converted text data and extracted key points. The user can also check notifications displayed on the device and approve new tasks and reminders. The schedule can also be adjusted as necessary.

[0616] Specific examples

[0617] Example 1: Long meetings

[0618] A user enters a week's worth of events into Google Calendar, for example, one day includes three consecutive hours of meetings.

[0619] The server analyzes this and suggests appropriate social activities before, during, or after the meeting.

[0620] The user selects "5 minutes of stretching" and the server schedules it and sets up a notification.

[0621] Example 2: Recording important information during a meeting

[0622] A user starts a conference and turns on the audio recording function of the terminal. The audio during the conversation is collected by the terminal and sent to the server.

[0623] The server converts the speech into text and extracts key points.

[0624] It is determined that "Submission of market analysis report" is required before the next meeting, and this information is added to the calendar and notified to the user as a reminder.

[0625] Prompt Sentence Examples

[0626] Example prompt 1:

[0627] "Please provide a meeting schedule. Please also provide key points about your next meeting."

[0628] Example prompt 2:

[0629] "Please suggest what social activities we can add to the schedule during long meetings."

[0630] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0631] Step 1:

[0632] The server retrieves the user's schedule information from an external scheduling system (e.g., Google Calendar, Microsoft Outlook).

[0633] Input: User's calendar information (via API)

[0634] Output: A dataset of retrieved schedule information

[0635] Specific operation: The server uses an API to read the user's schedule information from an external system. For example, it uses the Google Calendar API to get the schedule information for the next week.

[0636] Step 2:

[0637] The server analyzes the schedule information it obtains and detects consecutive meetings and tasks.

[0638] Input: Dataset of acquired schedule information

[0639] Output: Analysis results (identification of consecutive meetings and tasks)

[0640] Specific operation: The server sorts the schedule information in chronological order and executes logic to check whether consecutive meetings exceed a certain time (e.g., 3 hours).

[0641] Step 3:

[0642] The server determines the need for social activities and suggests appropriate social activities.

[0643] Input: Analysis results (identification of consecutive meetings and tasks)

[0644] Output: A list of suggested suitable social activities

[0645] Specific operation: When consecutive meetings are detected, the server generates a list of suggested social activities for the user, such as "two minutes of stretching" or "five minutes of chatting."

[0646] Step 4:

[0647] The terminal displays the social activity suggestions received from the server to the user.

[0648] Input: A list of suggested social activities sent by the server

[0649] Output: A proposed interface for the user to see.

[0650] Specific operation: The terminal displays the suggested interaction activities on the user interface, allowing the user to confirm the suggestions.

[0651] Step 5:

[0652] The user selects a suggested social activity on the device.

[0653] Input: A proposed interface for user display

[0654] Output: Selected interaction activities

[0655] Specific operation: The user uses the device interface to select the desired interaction activity from the suggested activities.

[0656] Step 6:

[0657] The server adds the user's selected social activity to the schedule.

[0658] Input: Selected interaction activity

[0659] Output: Updated schedule

[0660] Specific operation: The server schedules the interaction activity selected by the user and adds it as an appointment.

[0661] Step 7:

[0662] The terminal collects audio data during the meeting in real time and sends it to the server.

[0663] Input: Real-time audio in a meeting

[0664] Output: Collected audio data (digital format)

[0665] Specific operation: The device uses a voice input device to record audio during the meeting and save it in digital format. This data is then streamed to the server.

[0666] Step 8:

[0667] The server converts the received voice data into text data.

[0668] Input: Collected audio data

[0669] Output: Text data converted from audio

[0670] Specific operation: The server uses voice recognition technology (e.g., Google Cloud Speech-to-Text API) to convert the voice data into text data.

[0671] Step 9:

[0672] The server extracts the main points from the converted text data.

[0673] Input: Text data converted from speech

[0674] Output: Extracted key information

[0675] How it works: The server uses natural language processing techniques to extract important information from the text data, such as what needs to be reported at the next meeting.

[0676] Step 10:

[0677] The server adds new tasks and reminders to the user's calendar based on the extracted key points.

[0678] Input: Extracted gist information

[0679] Output: Updated calendar information (with new tasks and reminders added)

[0680] What happens: The server adds upcoming meeting arrangements and other necessary tasks to the user's calendar.

[0681] Step 11:

[0682] The server sends new schedule details and notifications to the user terminal and notifies them.

[0683] Input: Updated calendar information

[0684] Output: Notification to user terminal

[0685] Specific operation: The server notifies the user by sending the updated schedule to the user's device via push notification or other notification method.

[0686] Step 12:

[0687] The user checks the notification on the device and approves or adjusts it if necessary.

[0688] Input: Notification to user terminal

[0689] Output: User approved and adjusted schedule

[0690] What happens: The user sees the notifications on their device, acknowledges the new task or reminder, adjusts their schedule if necessary, and confirms the changes.

[0691] (Application example 1)

[0692] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0693] Robots and machines working in factories are often subjected to continuous, long-term operation, which can result in problems such as overheating and abnormal vibrations. It can also be difficult for workers to detect abnormalities in real time and take appropriate action. This can lead to reduced production efficiency and increased risk of machine breakdowns. Furthermore, workers often do not take appropriate breaks or undergo maintenance during long periods of work, which also reduces the operating efficiency and lifespan of machines. The present invention aims to solve these problems.

[0694] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0695] In this invention, the server includes means for acquiring user schedule information, means for analyzing the acquired schedule information and determining the need for an ice breaker, means for suggesting appropriate ice breaker activities, means for adding the suggested ice breaker activities to the user's schedule, means for managing the work schedule of the industrial machinery and suggesting breaks between continuous work, and means for collecting work data of the industrial machinery in real time and detecting abnormalities. This reduces the burden on machinery in a factory due to long hours of operation and makes it possible to quickly detect and respond to abnormalities.

[0696] A "user" is someone who uses this system to improve schedule management and work efficiency.

[0697] "Schedule information" is data relating to plans or plans that are input or obtained by the user.

[0698] An "icebreaker" is an activity such as a short break or light exercise that takes place during long periods of work or meetings.

[0699] "Industrial machinery" refers to machinery and equipment used in industrial sites such as factories.

[0700] A "work schedule" is a schedule or timetable for work performed by industrial machines and workers.

[0701] A "break" is a temporary pause between successive tasks.

[0702] "Audio data" refers to digital data of conversations or audio recordings.

[0703] "Text data" refers to digital data converted into character information.

[0704] A "gist" is a particularly important part of a conversation or piece of information.

[0705] "Abnormal" means any operation or condition of a machine that deviates from normal operating conditions.

[0706] "Real-time" refers to data collection and processing occurring immediately, without delay.

[0707] "Notification" means informing the user of new information or alerts.

[0708] A "smart device" is a mobile device such as a smartphone or tablet that can connect to the Internet and run applications.

[0709] System Configuration

[0710] This invention is a business efficiency improvement system that integrates the acquisition and analysis of user schedule information, ice-breaker suggestions, automatic text conversion and key point extraction of conversation content, schedule creation and notification. This system is mainly composed of three elements: a server, a terminal, and a user.

[0711] Server Operation

[0712] The server operates using the following methods:

[0713] 1. Acquisition and analysis of schedule information

[0714] The server obtains the user's schedule information from an external scheduling system (e.g., Google Calendar, Microsoft Outlook API). It analyzes the obtained schedule information and determines whether an icebreaker is necessary. For example, if a long meeting or continuous work on industrial machinery is scheduled, it determines that an icebreaker is necessary in between.

[0715] 2. Propose and confirm icebreakers

[0716] The server generates a list of ice-breaking activities (e.g., two minutes of stretching, five minutes of maintenance check) based on the analysis results and suggests them to the user's device. The ice-breaking activities selected by the user are added to the schedule.

[0717] 3. Conversation text conversion and key points extraction

[0718] The server receives the audio data from the meeting and converts it into text using speech recognition software such as the Google Speech-to-Text API. The server then extracts key points from the converted text.

[0719] 4. Schedule Task Reminders

[0720] Based on the extracted key points, new tasks and reminders are added to the user's calendar using data analysis techniques using Python and the pandas library.

[0721] 5. Industrial Machinery Data Collection and Anomaly Detection

[0722] The server collects operational data (e.g., temperature, vibration, and operating time) from industrial machinery in real time and detects abnormalities. This data collection and analysis is done using Python and the pandas library. If an abnormality is detected, an alert is sent to the worker's smart device.

[0723] Device behavior

[0724] The terminal operates using the following means:

[0725] 1. Receiving and displaying schedule information

[0726] The terminal displays the schedule information received from the server to the user, who can then check and select the icebreaker activities suggested through the terminal.

[0727] 2. Collection of audio data

[0728] The terminal collects audio data during the meeting in real time and sends it to a server. The audio data is also recorded in digital format and can be viewed later.

[0729] 3. Display of notifications

[0730] The device displays notifications of new schedules and reminders received from the server to the user, who can then review, approve, and adjust the schedule.

[0731] User operations

[0732] The user interacts with the device using the following means:

[0733] 1. Check and select your schedule

[0734] The user checks the schedule through their device and selects an icebreaker activity suggested by the server. The selected activity is automatically added to the schedule.

[0735] 2. Record and review conversations

[0736] When starting a meeting, users can turn on the device's audio recording function to record the conversation. After the meeting, they can check the converted text data and extracted key points.

[0737] 3. Review and adjust notifications

[0738] Users can view notifications on their device, acknowledge new tasks and reminders, and adjust their schedules as needed.

[0739] Specific examples

[0740] The following are specific examples for carrying out the present invention:

[0741] Example 1: Long meetings and continuous work

[0742] For example, if a user has a three-hour continuous meeting or continuous work schedule for an industrial machine, the server analyzes the schedule and suggests suitable icebreakers (e.g., five-minute stretching or maintenance check) before, during, or after the meeting or work. The icebreaker activities selected by the user are incorporated into the schedule and notifications are set.

[0743] Example 2: Recording important information during meetings and detecting anomalies

[0744] A user starts a meeting and turns on the audio recording function on their device. The audio during the conversation is collected by the device and sent to the server. The server converts the audio into text and extracts key points. It determines that a "market analysis report" needs to be submitted before the next meeting, and this information is added to the calendar and notified to the user as a reminder. Furthermore, if an industrial machine detects abnormal vibrations, the server will issue a warning to the worker's smart device.

[0745] Prompt Sentence Examples

[0746] As a concrete example, consider the following prompt input to a generative AI model:

[0747] Enter the following data: a list of work schedules, activities, icebreaker suggestions, and work data (temperature, vibration, etc.). For example, "2023-10-10 09:00-12:00: Work A, 2023-10-10 12:00-13:00: Break."

[0748] The system suggests an icebreaker: "5 minutes of maintenance after 20 minutes of work." It analyzes data and issues an alert if it detects an abnormality.

[0749] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0750] Step 1:

[0751] The server retrieves the user's schedule information using the API of an external scheduling system (e.g., Google Calendar or Microsoft Outlook), which requires an API key and a calendar ID as input. The output is the retrieved schedule information.

[0752] Step 2:

[0753] The server analyzes the schedule information it has acquired and detects long meetings and work. The input is the schedule information acquired in step 1, and the data is analyzed to extract events with long meetings or continuous work. This analysis identifies locations where ice breakers are needed. The output is the analysis results, i.e., a list of events where ice breakers are needed.

[0754] Step 3:

[0755] The server proposes ice-breaking activities (e.g., 2 minutes of stretching, 5 minutes of maintenance check) based on the analysis results. The input is the list of events requiring ice-breaking obtained in step 2. The server assigns ice-breaking activities to these events and generates a list of activities to propose to the user. The output is a list of proposed ice-breaking activities.

[0756] Step 4:

[0757] The terminal displays the list of icebreaker activities received from the server to the user. The input is the list of icebreaker activities generated in step 3, and the terminal displays it on the user's interface so that the user can select one. The output is the icebreaker activity selected by the user.

[0758] Step 5:

[0759] The server adds the icebreaker activity selected by the user to the schedule and notifies the user. The input is the icebreaker activity selected by the user in step 4, the server incorporates it into the schedule, and generates a new schedule. The output is the updated schedule information.

[0760] Step 6:

[0761] The terminal collects audio data during the meeting in real time and sends it to the server. The input is the audio data recorded during the meeting, which the terminal records in digital format and sends to the server. The output is the audio data sent to the server.

[0762] Step 7:

[0763] The server converts the audio data into text data and extracts the main points. The input is the audio data sent in step 6, and the server converts the audio data into text data using the Google Speech-to-Text API. It then performs natural language processing to extract the main points from the text. The output is a list of main points.

[0764] Step 8:

[0765] The server adds new tasks and reminders to the user's schedule based on the extracted key points. The input is the list of key points generated in step 7, and the server automatically adds new tasks and reminders to the schedule based on this. The output is the updated schedule information.

[0766] Step 9:

[0767] The server collects operational data from industrial machinery in real time and detects abnormalities. The input is the operational data of the industrial machinery (such as temperature, vibration, operating time, etc.), which the server monitors in real time and analyzes abnormalities. The output is an alert when an abnormality is detected.

[0768] Step 10:

[0769] When the server detects an abnormality, it issues a warning to the worker's smart device. The input is the abnormal data detected in step 9, and a warning message is generated based on that content and notified to the worker's smart device. The output is the notification sent to the worker.

[0770] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0771] System configuration

[0772] This invention is a business efficiency improvement system that integrates the acquisition and analysis of user schedule information, ice-breaker suggestions, automatic text conversion and key point extraction of conversation content, schedule creation and notifications, and also incorporates an emotion engine. The roles and specific operations of the server, terminal, and user are explained below.

[0773] Server Operation

[0774] 1. Acquisition and analysis of schedule information

[0775] The server retrieves the user's schedule information from an external scheduling system (e.g., Google Calendar, Microsoft Outlook). It analyzes the retrieved schedule information and determines whether an icebreaker is necessary. For example, if there are consecutive long meetings, it determines that an icebreaker is necessary.

[0776] 2. Propose and confirm icebreakers

[0777] Based on the analysis results, the server generates a list of ice-breaking activities (e.g., two minutes of stretching, five minutes of chatting) and suggests them to the user's device. The ice-breaking activities selected by the user are added to the schedule. Furthermore, the selection of ice-breaking activities is optimized based on the analysis results of the emotion engine.

[0778] 3. Conversation text conversion and key points extraction

[0779] The server receives the voice data from the meeting and converts it into text using speech recognition technology. Key points (e.g., matters that need to be reported at the next meeting) are extracted from the converted text data. The emotion engine also recognizes the user's emotions in real time during the conversation and adjusts the extraction of key points and the addition of tasks according to their emotional state.

[0780] 4. Schedule Task Reminders

[0781] Based on the extracted key points, new tasks and reminders are added to the user's calendar. For example, an item such as "Prepare for the next meeting" is automatically added. The emotion engine analyzes the user's stress and fatigue levels and dynamically adjusts the content and timing of reminders based on the results.

[0782] 5. Notification

[0783] The server sends new schedule information and reminders to the user's device, notifying the user. The user checks the notification and adjusts the schedule as necessary.

[0784] Device behavior

[0785] 1. Receiving and displaying schedule information

[0786] The terminal displays the schedule information received from the server to the user, who can then check and select the icebreaker activities suggested through the terminal.

[0787] 2. Collection of audio data

[0788] The device collects audio data during the meeting in real time and sends it to a server. The audio data is also recorded in digital format and can be viewed later.

[0789] 3. Use of Emotion Engine

[0790] The device analyzes the user's facial expressions and voice tone and uses an emotion engine to analyze the user's emotional state in real time. The analysis results of the emotion engine are sent to a server and used to suggest icebreakers, extract key points, and add tasks.

[0791] 4. Display of notifications

[0792] The device displays notifications of new schedules and reminders received from the server to the user, who can then review, approve, and adjust the schedule.

[0793] User operations

[0794] 1. Check and select your schedule

[0795] The user checks the schedule on their device and selects an icebreaker activity suggested by the server. The selected activity is automatically added to the schedule.

[0796] 2. Record and review conversations

[0797] When a user starts a meeting, they can turn on the device's audio recording function to record the conversation. After the meeting, they can check the converted text data and extracted key points.

[0798] 3. Review and adjust notifications

[0799] Users can view notifications on their device, acknowledge new tasks and reminders, and adjust their schedules as needed.

[0800] Specific examples

[0801] Example 1: Long meetings

[0802] A user enters a week's schedule, including a three-hour series of meetings.

[0803] The server analyzes this and suggests appropriate icebreakers before, during, or after the meeting. The emotion engine analyzes the user's fatigue level and recommends icebreakers to relieve stress.

[0804] The user selects "5 minutes of stretching" and the server schedules it and sets up a notification.

[0805] Example 2: Recording important information during a meeting

[0806] A user starts a conference and turns on the audio recording function of the terminal. The audio during the conversation is collected by the terminal and sent to the server.

[0807] The server converts the speech into text and extracts key points, while an emotion engine analyzes the user's emotions and adjusts the key points taking into account the importance and urgency of the topic.

[0808] It is determined that "Submission of market analysis report" is required before the next meeting, and this information is added to the calendar and notified to the user as a reminder.

[0809] As described above, the present invention improves the efficiency of user schedule management and the recording and analysis of conversation content, and by combining it with an emotion engine, it provides optimal business support based on the user's emotional state.

[0810] The processing flow will be explained below.

[0811] Step 1:

[0812] The user accesses the tool and starts the schedule management service.

[0813] Step 2:

[0814] The device retrieves the user's calendar data from an external scheduling service such as Google Calendar or Microsoft Outlook, and then sends this information to the server.

[0815] Step 3:

[0816] The server analyzes the received schedule data and identifies when an icebreaker is needed, such as during a long meeting or when consecutive tasks are required.

[0817] Step 4:

[0818] The server determines the need for an icebreaker based on the analysis results, and if necessary, selects an appropriate icebreaker activity from the database.

[0819] Step 5:

[0820] The server generates a proposal for the selected icebreaker activity and transmits it to the terminal.

[0821] Step 6:

[0822] The device displays icebreaker suggestions to the user, who can then select any icebreaker activity from the suggestions.

[0823] Step 7:

[0824] The device sends the icebreaker activity selected by the user to the server, which adds it to the user's schedule.

[0825] Step 8:

[0826] When a user starts a conference, the device turns on the audio recording function, and the device collects the audio data of the conversation in real time and transmits it to the server.

[0827] Step 9:

[0828] The server converts the voice data into text data using voice recognition technology.

[0829] Step 10:

[0830] The server analyzes the converted text data and extracts important points, for example, identifying tasks such as "items that need to be reported at the next meeting."

[0831] Step 11:

[0832] The server adds new tasks and reminders to the user's calendar based on the extracted key points.

[0833] Step 12:

[0834] The server sends new schedules and reminders to the device and creates notifications, which the device displays to the user.

[0835] Step 13:

[0836] The user checks the notification and adjusts the schedule as necessary.

[0837] Example 2

[0838] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0839] Conventional schedule management systems do not provide effective ways to reduce fatigue and stress caused by continuous meetings and tasks. Furthermore, the efficient recording of important comments and decisions during meetings and the manual task management based on these records result in reduced work efficiency. Furthermore, the lack of timely reminders and notifications that take into account the user's emotional state makes it difficult for users to maintain their productivity.

[0840] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for acquiring user schedule information, means for analyzing the acquired schedule information and determining the need for an icebreaker, means for proposing appropriate icebreaker activities, means for adding the proposed icebreaker activities to the user's schedule, means for analyzing the user's emotional state and optimizing the icebreaker activities, means for collecting voice data of the conversation, means for converting the collected voice data into text data, means for extracting key points from the converted text data, means for adding new tasks or reminders to the user's schedule based on the extracted key points, and means for sending and notifying the new schedule or reminder to the user terminal. This effectively reduces fatigue and stress caused by continuous meetings and tasks for the user, efficiently records important comments and decisions made during meetings, automates task management based on the records, and enables appropriate timing of reminders and notifications taking the user's emotional state into consideration.

[0841] "Schedule information" refers to data relating to the user's planned dates and activities.

[0842] "Acquisition means" refers to the methods or functions for collecting data or information from external or internal systems.

[0843] "Analysis means" refers to the methods and functions used to analyze acquired data and information and understand its content and meaning.

[0844] "Ice-breaking activities" are refreshing activities or short recreational activities to relieve fatigue and stress caused by long meetings or task completion.

[0845] "Suggestion means" refers to methods or functions for presenting appropriate information or activities to users.

[0846] "Additional means" refers to methods or functions for inserting new information or activities into a user's schedule.

[0847] "Emotional state" refers to the mental state or mood of the user.

[0848] "Audio data" refers to data that has been recorded in digital form as a conversation or voice signal.

[0849] "Text data" refers to data obtained by converting voice data into character information.

[0850] "Key point extraction means" refers to a method or function for extracting important information or key content from text data.

[0851] A "task" is a specific task or job that a user must undertake.

[0852] A "reminder" is a notification or alarm that reminds a user of a specific date, time, or event.

[0853] "Notification means" refers to the method or function for notifying the user of new information or reminders.

[0854] A "terminal" is a computing device that is directly operated by a user.

[0855] A "server" is a central computer system that processes and manages data on a network.

[0856] This is a business efficiency improvement system that integrates the acquisition and analysis of user schedule information, ice-breaker suggestions, automatic text conversion and key point extraction of conversation content, and task and reminder scheduling and notification. Furthermore, it incorporates an emotion engine to provide optimal support based on the user's emotional state.

[0857] System Configuration

[0858] Server Operation

[0859] The server first obtains the user's schedule information from an external scheduling system (e.g., Google Calendar or Microsoft Outlook) via API. The obtained schedule information is stored in an internal database, and an analysis engine uses this information to determine whether an icebreaker is necessary.

[0860] Next, based on the analysis results, the server lists multiple ice-breaking activities (e.g., two minutes of stretching, five minutes of chatting) and sends this list in JSON format to the user's device. At the same time, the emotion engine analyzes the user's stress level and fatigue level to select the most appropriate ice-breaking activity. The server adds the ice-breaking activity selected by the user to the schedule and notifies the user's device again.

[0861] During the meeting, the server receives real-time voice data from the devices and converts it into text data using the Google Speech-to-Text API. The converted text data is stored in a database, and a natural language processing (NLP) engine extracts key points. An emotion engine analyzes the user's emotions and adjusts the importance of the key points.

[0862] Furthermore, new tasks and reminders are added to the user's calendar based on the extracted key points. The emotion engine dynamically adjusts the content and timing of reminders, taking into account the user's stress level and fatigue level. The server then sends the new schedule and reminders to the user's device and notifies the user.

[0863] Device behavior

[0864] The device displays the schedule information received from the server to the user via a GUI. The user can then review and select suggested icebreaker activities. During the meeting, the device's microphone collects audio data in real time, digitizes it, and sends it to the server. This audio data is also stored locally and can be played back later.

[0865] The device uses its built-in camera and microphone to analyze the user's facial expressions and voice tone and input them into the emotion engine. The analysis results are sent to the server and reflected in icebreaker suggestions and key point extraction. In addition, the device receives new schedule and reminder notifications from the server and displays them as push notifications, allowing the user to confirm, approve, or adjust them.

[0866] User operations

[0867] The user checks the schedule on their device and selects an icebreaker activity suggested by the server. The selected activity is automatically added to the schedule. When starting a meeting, the user turns on the device's audio recording function to record the conversation. After the meeting, the converted text data and extracted key points can be viewed. The user can also check notifications displayed on the device, approve new tasks and reminders, and adjust the schedule as necessary.

[0868] Specific examples

[0869] Example 1: Long meetings

[0870] A user enters a week's worth of events into Google Calendar, and one day has three consecutive meetings scheduled.

[0871] The server analyzes this and suggests appropriate icebreakers before, during, or after the meeting. The emotion engine analyzes the user's fatigue level and recommends "5 minutes of stretching" to relieve stress.

[0872] The user selects "5 minutes of stretching" and the server schedules it and sets up a notification.

[0873] Example 2: Recording important information during a meeting

[0874] A user starts a conference and turns on the audio recording function of the device. The audio during the conversation is collected by the device and transmitted to the server in real time.

[0875] The server converts the speech into text and extracts key points. The emotion engine analyzes the user's emotions and adjusts the key points based on the importance and urgency of the topic.

[0876] It turns out that "Submission of Market Analysis Report" is required before the next meeting, and this information is added to the calendar and notified to the user as a reminder.

[0877] Prompt Sentence Examples

[0878] "Use your schedule and sentiment data to suggest the best icebreaker activities, extract key points from meetings, and add upcoming tasks to your schedule."

[0879] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0880] Step 1: Get schedule information

[0881] The server gets information from an external scheduling system

[0882] Input: User authentication information, API key

[0883] Output: User schedule information (JSON format)

[0884] How it works: The server uses the APIs of Google Calendar and Microsoft Outlook to retrieve user schedule information. This information is sent to the server in JSON format and stored in an internal database.

[0885] Step 2: Analyze schedule information

[0886] The server analyzes the schedule information and determines whether an icebreaker is necessary.

[0887] Input: Schedule information (JSON format)

[0888] Output: Analysis results (flag indicating need for icebreaker)

[0889] How it works: The server's analysis engine analyzes schedule information and determines whether an icebreaker is necessary if there are consecutive "meetings" or "long-term tasks." For example, if the interval between consecutive meetings exceeds a certain time, it determines that an icebreaker is necessary.

[0890] Step 3: Suggest an icebreaker activity

[0891] Your server will suggest appropriate icebreaker activities

[0892] Input: Analysis results, emotion engine data

[0893] Output: Icebreaker proposal list (JSON format)

[0894] How it works: The server suggests icebreaker activities suitable for the user based on the analysis results and emotion engine data. The list of suggestions is sent to the device in JSON format.

[0895] Step 4: Determine icebreaker activities

[0896] The server adds icebreaker activities to the schedule based on the user's selection.

[0897] Input: User's choice

[0898] Output: Updated schedule information

[0899] How it works: After a user selects an icebreaker activity on the device, the server adds the selection to the schedule and sends the updated schedule information back to the device.

[0900] Step 5: Collecting audio data from the conversation

[0901] The device collects audio data during the meeting and sends it to the server.

[0902] Input: Audio during the meeting

[0903] Output: Digital audio data

[0904] How it works: The device's microphone collects audio during the meeting in real time and sends it digitally to a server, where the audio data is stored locally.

[0905] Step 6: Convert audio data to text

[0906] The server converts the audio data into text data.

[0907] Input: Audio data

[0908] Output: Text data

[0909] How it works: The server uses the Google Speech-to-Text API to convert the received audio data into text data, which is then stored in a database.

[0910] Step 7: Extracting key points

[0911] The server extracts key points from the text data

[0912] Input: Text data

[0913] Output: Gist list

[0914] How it works: The server's natural language processing (NLP) engine analyzes the text data and extracts key conversation points and action items. The emotion engine also analyzes the user's emotions at this stage and adjusts the importance of key points.

[0915] Step 8: Schedule task reminders

[0916] The server adds new tasks and reminders to your schedule

[0917] Input: Key points list, emotion engine data

[0918] Output: Updated schedule

[0919] How it works: Based on the extracted key points, the server adds new tasks and reminders to the user's calendar. The emotion engine dynamically adjusts the timing of reminders based on the user's stress and fatigue levels.

[0920] Step 9: Notification

[0921] The server sends new schedules and reminders to the user's device and notifies them.

[0922] Input: Updated Schedule

[0923] Output: Notification message

[0924] How it works: The server sends updated schedule information and reminders to the device, which are displayed to the user as push notifications. The user can view the notifications and accept or adjust them as needed.

[0925] (Application example 2)

[0926] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0927] In modern factories and offices, workers often find it difficult to take appropriate breaks or refresh themselves when working long hours continuously or in the middle of important tasks. Furthermore, systems for accurately recording the work performed by workers and later reviewing and utilizing this information are often inadequate. Furthermore, to reduce workers' stress and fatigue, it is necessary to provide optimal intervals based on their emotional state. These challenges can reduce work efficiency and ultimately hinder improvements in productivity and work quality. A solution to these problems and achieve both work efficiency and worker health is needed.

[0928] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0929] In this invention, the server includes means for acquiring a user's schedule information, means for analyzing the acquired schedule information and determining the need for a break, means for suggesting appropriate break activities, means for adding the suggested break activities to the user's schedule, means for analyzing the user's emotional state using an emotion engine and optimizing the selection of break activities, means for collecting voice data during work and converting it into text data, means for extracting important work points from the converted text data, means for adding new tasks and reminders to the user's schedule based on the extracted points, and means for displaying notifications to the user and allowing the user to confirm, approve, and adjust the suggested break activities and added tasks and reminders. This enables workers to take breaks at appropriate times, maximize work efficiency, and accurately record and manage important work content and key points.

[0930] "User" refers to the workers and employees who use the system.

[0931] "Setup information" refers to information that indicates the work schedules, plans, and tasks of workers or employees.

[0932] "Emotion engine" refers to technology that analyzes a user's facial expressions and vocal tone to recognize and analyze their emotional state (e.g., stress or fatigue) in real time.

[0933] "Rest activities" refer to short breaks or refreshing activities (e.g., short stretching or chatting) that users take between tasks to reduce fatigue and stress.

[0934] "Voice data" refers to digital data of a user's voice collected while working or talking.

[0935] "Text data" refers to sentence data obtained by converting voice data into characters.

[0936] "Key points" are pieces of information or tasks that are considered particularly important in a work progress or conversation.

[0937] A "task" refers to a discrete task or activity that a user must perform.

[0938] A "reminder" is a notification or reminder that helps users remember specific tasks or appointments.

[0939] This system acquires setup information for workers and employees, proposes and schedules optimal break activities to improve work efficiency, and automatically records and manages important work content and key points of conversations. The roles and specific operations of the server, terminal, and user are explained below.

[0940] Server Operation

[0941] Acquisition and analysis of setup information

[0942] The server acquires setup information for workers and employees from the factory management system or business management system (MES or ERP system). It analyzes the acquired setup information and determines whether breaks are necessary. For example, if a long period of continuous work is scheduled, it determines that a break is necessary in between.

[0943] Proposing and confirming break activities

[0944] Based on the analysis results, the server generates a list of break activities (e.g., short breaks and stretching) and suggests them to the user's device. It uses an emotion engine to analyze the user's emotional state and selects the optimal break activity based on the results. The break activity selected by the user is added to the schedule.

[0945] Speech data text conversion and gist extraction

[0946] The server receives the working voice data and converts it into text using speech recognition technology (e.g., the speech_recognition library). It then uses a generative AI model (e.g., GPT-3.5) to extract key points from the converted text. The extraction of key points is optimized according to the analysis results of the emotion engine.

[0947] Schedule tasks and reminders

[0948] The server adds new tasks and reminders to the user's schedule based on the extracted key points. For example, information such as "Parts for the next process are missing" is added to the schedule and notified to the user as a reminder.

[0949] notification

[0950] The server sends new schedule information and reminders to the user's device, and notifies the user. The user can check the notification and adjust the schedule as necessary.

[0951] Device behavior

[0952] Receiving and displaying setup information

[0953] The terminal displays the schedule information received from the server to the user, who can then check and select the suggested break activities through the terminal.

[0954] Audio data collection

[0955] The device collects voice data in real time while the worker is working and sends it to a server, where it is recorded in digital format and can be viewed later.

[0956] Use of emotion engine

[0957] The device analyzes the user's facial expressions and voice tone and uses an emotion engine to analyze the user's emotional state in real time. The analysis results of the emotion engine are sent to a server and used to suggest break activities, extract key points, and add tasks.

[0958] Viewing notifications

[0959] The device displays notifications of new schedules and reminders received from the server to the user, who can then review, approve, and adjust the schedule.

[0960] User operations

[0961] Check and select setup

[0962] The user checks the schedule through the terminal and selects the break activity suggested by the server, which is then automatically added to the schedule.

[0963] Recording and checking work details

[0964] When a user starts a task, they can turn on the device's voice recording function to record the task's progress. After the task is completed, they can check the converted text data and extracted key points.

[0965] Review and adjust notifications

[0966] Users can view notifications on their device, acknowledge new tasks and reminders, and adjust their schedules as needed.

[0967] Specific examples

[0968] Example 1: Long working hours

[0969] The user enters their weekly schedule, and one day is scheduled for six hours of continuous robot operation.

[0970] The server analyzes this and suggests appropriate rest activities before, during, or after the operation. The emotion engine analyzes the user's fatigue level and recommends short breaks to relieve stress.

[0971] The user selects "5 minutes of stretching" and the server schedules it and sets up a notification.

[0972] Example 2: Recording important information during work

[0973] The user starts working and turns on the device's voice recording function. The voice recorded during the work is collected by the device and sent to the server.

[0974] The server converts the speech into text and extracts key points, while the emotion engine analyzes the user's emotions and optimizes the key points in real time.

[0975] If a "parts shortage" is discovered by the time the next process is completed, this information is added to the schedule and a reminder is sent to the user.

[0976] Example of a generative AI model prompt:

[0977] Prompt Sentence Examples

[0978] Text from audio recording: "We've encountered a parts supply issue. We are running low on parts needed for the next operation. We need to reconsider our parts supply plan."

[0979] Gist Extraction Generation AI Prompt: "Please extract the key points from the audio recording above."

[0980] Key point: "Parts supply problem. Parts for the next process are in short supply. Parts supply plan needs to be reconsidered."

[0981] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0982] Step 1:

[0983] The server obtains setup information for workers and employees from the factory management system or business management system (MES or ERP system). This setup information includes work schedules, plans, and tasks. It analyzes this information and evaluates the length and continuity of work time. It receives the worker's setup information (schedule, plan, tasks) as input and determines the need for breaks as a result of the analysis.

[0984] Step 2:

[0985] The server generates a list of rest activities based on the analysis results. It uses the emotion engine to analyze the user's emotional state and selects the optimal rest activity taking the results into consideration. For example, if the user has been working for a long time or is feeling emotional stress, it suggests stretching or a short break. The server uses the analysis results and emotion engine data as input and generates a list of rest activities as output.

[0986] Step 3:

[0987] The terminal receives the list of break activities sent from the server and displays it to the user. The user checks the proposed break activities through the terminal and selects one. The terminal receives the list of break activities as input and sends the selected break activity to the server as output.

[0988] Step 4:

[0989] The server adds the break activity selected by the user to the setup information and updates the schedule. This updated schedule information is sent to the user's terminal and notified. The server receives the break activity selected by the user as input, generates updated schedule information as output, and sends it to the terminal.

[0990] Step 5:

[0991] When a user starts working, the terminal turns on the voice recording function, collects voice data during work, and sends this data to the server. The terminal collects the user's voice data as input and sends the voice data to the server as output.

[0992] Step 6:

[0993] The server converts the received voice data into text data using speech recognition technology (e.g., the speech_recognition library), and extracts key points from the converted text data using a generative AI model (e.g., GPT-3.5). It receives voice data as input and generates text data and key points as output.

[0994] Step 7:

[0995] The server adds new tasks and reminders to the schedule information based on the extracted key points. This information is sent to the user's terminal and displayed as a notification. As input, the server generates new tasks and reminders based on the extracted key points, and as output, adds these to the schedule and sends them to the terminal.

[0996] Step 8:

[0997] The user can check the notifications displayed on the device, acknowledge new tasks and reminders, and adjust the schedule as needed. The input is the notification displayed by the device, and the output is the necessary adjustment.

[0998] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0999] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1000] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1001] [Third embodiment]

[1002] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1003] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1004] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1005] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1006] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1007] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1008] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1009] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1010] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1011] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1012] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1013] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1014] System configuration

[1015] This invention is a business efficiency improvement system that integrates the acquisition and analysis of user schedule information, ice-breaker suggestions, automatic text conversion and key point extraction of conversation content, schedule creation and notification. The roles and specific operations of the server, terminal, and user are explained below.

[1016] Server Operation

[1017] 1. Acquisition and analysis of schedule information

[1018] The server retrieves the user's schedule information from an external scheduling system (e.g., Google Calendar, Microsoft Outlook). It analyzes the retrieved schedule information and determines whether an icebreaker is necessary. For example, if there are consecutive long meetings, it determines that an icebreaker is necessary.

[1019] 2. Propose and confirm icebreakers

[1020] Based on the analysis results, the server generates a list of ice-breaking activities (e.g., two minutes of stretching, five minutes of chatting) and suggests them to the user's device. The ice-breaking activity selected by the user is added to the schedule.

[1021] 3. Conversation text conversion and key points extraction

[1022] The server receives the voice data from the meeting and converts it into text data using speech recognition technology. Key points (e.g., matters that need to be reported at the next meeting) are extracted from the converted text data.

[1023] 4. Schedule Task Reminders

[1024] Based on the extracted key points, new tasks and reminders are added to the user's calendar, for example, an item such as "Prepare for the next meeting" is automatically added.

[1025] 5. Notification

[1026] The server sends new schedule details and reminders to the user's device, notifying the user. The user checks the notifications and adjusts the schedule as necessary.

[1027] Device behavior

[1028] 1. Receiving and displaying schedule information

[1029] The terminal displays the schedule information received from the server to the user, who can then check and select the icebreaker activities suggested through the terminal.

[1030] 2. Collection of audio data

[1031] The device collects audio data during the meeting in real time and sends it to a server. The audio data is also recorded in digital format and can be viewed later.

[1032] 3. Display of notifications

[1033] The device displays notifications of new schedules and reminders received from the server to the user, who can then review, approve, and adjust the schedule.

[1034] User operations

[1035] 1. Check and select your schedule

[1036] The user checks the schedule on their device and selects an icebreaker activity suggested by the server. The selected activity is automatically added to the schedule.

[1037] 2. Record and review conversations

[1038] When a user starts a meeting, they can turn on the device's audio recording function to record the conversation. After the meeting, they can check the converted text data and extracted key points.

[1039] 3. Review and adjust notifications

[1040] Users can view notifications on their device, acknowledge new tasks and reminders, and adjust their schedules as needed.

[1041] Specific examples

[1042] Example 1: Long meetings

[1043] A user enters a week's schedule, including a three-hour series of meetings.

[1044] The server analyzes this and suggests appropriate icebreakers before, during, or after the meeting.

[1045] The user selects "5 minutes of stretching" and the server schedules it and sets up a notification.

[1046] Example 2: Recording important information during a meeting

[1047] A user starts a conference and turns on the audio recording function of the terminal. The audio during the conversation is collected by the terminal and sent to the server.

[1048] The server converts the speech into text and extracts key points.

[1049] It is determined that "Submission of market analysis report" is required before the next meeting, and this information is added to the calendar and notified to the user as a reminder.

[1050] As described above, the present invention is a useful system that improves the efficiency of users' schedule management and the recording and analysis of conversation content, thereby improving work efficiency.

[1051] The processing flow will be explained below.

[1052] Step 1:

[1053] The user accesses the tool and starts the schedule management service.

[1054] Step 2:

[1055] The device retrieves the user's calendar data from an external scheduling service such as Google Calendar or Microsoft Outlook, and then sends this information to the server.

[1056] Step 3:

[1057] The server analyzes the received schedule data and identifies when an icebreaker is needed, such as during a long meeting or when consecutive tasks are required.

[1058] Step 4:

[1059] The server determines the need for an icebreaker based on the analysis results, and if necessary, selects an appropriate icebreaker activity from the database.

[1060] Step 5:

[1061] The server generates a proposal for the selected icebreaker activity and transmits it to the terminal.

[1062] Step 6:

[1063] The device displays icebreaker suggestions to the user, who can then select any icebreaker activity from the suggestions.

[1064] Step 7:

[1065] The device sends the icebreaker activity selected by the user to the server, which adds it to the user's schedule.

[1066] Step 8:

[1067] When a user starts a conference, the device turns on the audio recording function, and the device collects the audio data of the conversation in real time and transmits it to the server.

[1068] Step 9:

[1069] The server converts the voice data into text data using voice recognition technology.

[1070] Step 10:

[1071] The server analyzes the converted text data and extracts important points, for example, identifying tasks such as "items that need to be reported at the next meeting."

[1072] Step 11:

[1073] The server adds new tasks and reminders to the user's calendar based on the extracted key points.

[1074] Step 12:

[1075] The server sends new schedules and reminders to the device and creates notifications, which the device displays to the user.

[1076] Step 13:

[1077] The user checks the notification and adjusts the schedule as necessary.

[1078] Example 1

[1079] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1080] In today's business environment, users need to efficiently manage numerous meetings and tasks. However, long meetings and continuous tasks often reduce users' concentration and productivity. Furthermore, there are cases where important comments or decisions made during meetings are not recorded, hindering subsequent work. There is a need for a system that can solve these issues and improve users' work efficiency.

[1081] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1082] In this invention, the server includes means for acquiring user schedule information, means for analyzing the acquired schedule information and determining the need for social activities, means for suggesting appropriate social activities, means for adding the suggested social activities to the user's schedule, means for collecting meeting audio information, means for converting the collected audio information into text data, means for extracting key points from the converted text data, means for adding new tasks and notifications to the user's schedule based on the extracted key points, and means for sending the new schedule contents and notifications to the user's terminal. This makes it possible to efficiently manage the user's work schedule and automatically record and notify important information.

[1083] The "means for acquiring user's schedule information" is a function for collecting schedule information from the user's calendar data or scheduling system.

[1084] The "means for analyzing schedule information and determining the necessity of social activities" is a function for analyzing the user's schedule based on the acquired schedule information and evaluating whether social activities are necessary.

[1085] The "means for suggesting appropriate social activities" is a function for suggesting appropriate relaxation methods and social activities to the user based on the analysis results.

[1086] The "means for adding an interaction activity to a user's schedule" is a function for automatically incorporating an interaction activity selected by the user into a schedule and adding it as a schedule.

[1087] The "means for collecting audio information from a meeting" is a function for digitally recording audio during a meeting and storing it for later processing.

[1088] The "means for converting collected voice information into text data" is a function that uses voice recognition technology to convert voice data into text data.

[1089] "Means for extracting key points from text data" is a function that uses natural language processing technology to extract important information and decisions from meeting content converted into text data.

[1090] The "means for adding new tasks and notifications to the user's schedule" is a function for automatically adding related tasks and reminders to the user's schedule based on the extracted key points.

[1091] "Means for sending new schedule details and notifications to the user's device" refers to a function for sending new schedule and reminder information to the user's device and notifying them via push notifications, etc.

[1092] The "means for checking and selecting suggested social activities on the user's terminal" is a function that allows the user to view suggested social activities through the terminal and select a desired activity.

[1093] "Means for reviewing, approving, and adjusting notifications displayed on the user's device" refers to a function that allows the user to review notifications displayed on the device, approve their contents, and change the schedule as necessary.

[1094] System configuration

[1095] This invention is a business efficiency system that integrates the acquisition and analysis of user schedule information, suggestion of social activities, automatic text conversion and key point extraction of conversation content, and scheduling and notification of new tasks and notifications. The roles and specific operations of the server, terminal, and user are explained below.

[1096] Server Operation

[1097] The server retrieves the user's schedule information from external scheduling systems such as Google Calendar or Microsoft Outlook. It analyzes the retrieved schedule information, detects consecutive meetings or tasks, and determines whether social activities are necessary. For example, if there are consecutive long meetings, it determines that social activities such as stretching or chatting are necessary in between.

[1098] Next, the server generates a list of appropriate social activities based on the analysis results and suggests them to the user's device. The social activities selected by the user are added to the schedule, or new events are added. The server also receives audio information from the meeting in real time and converts it into text data using speech recognition technology, such as the Google Cloud Speech-to-Text API. The server then extracts key points from the converted text data and identifies important information, such as what needs to be reported at the next meeting.

[1099] The server then adds new tasks and reminders to the user's calendar based on the extracted key points. For example, an item such as "Prepare for the next meeting" is automatically added. Finally, the server sends the new schedule and notifications to the user's device, notifying them via push notifications or other means.

[1100] Device behavior

[1101] The terminal displays the schedule information received from the server to the user. The user can check and select suggested social activities through the terminal. The terminal also collects audio data during the meeting in real time and sends it to the server. This audio data is recorded in digital format and can be checked later. The terminal displays notifications of new appointments and reminders received from the server to the user. The user can check, approve, and adjust the schedule from these notifications.

[1102] User operations

[1103] The user checks the schedule through the device and selects a social activity suggested by the server. The selected activity is automatically added to the schedule. When starting a meeting, the user turns on the device's audio recording function to record the conversation. After the meeting, the user can check the converted text data and extracted key points. The user can also check notifications displayed on the device and approve new tasks and reminders. The schedule can also be adjusted as necessary.

[1104] Specific examples

[1105] Example 1: Long meetings

[1106] A user enters a week's worth of events into Google Calendar, for example, one day includes three consecutive hours of meetings.

[1107] The server analyzes this and suggests appropriate social activities before, during, or after the meeting.

[1108] The user selects "5 minutes of stretching" and the server schedules it and sets up a notification.

[1109] Example 2: Recording important information during a meeting

[1110] A user starts a conference and turns on the audio recording function of the terminal. The audio during the conversation is collected by the terminal and sent to the server.

[1111] The server converts the speech into text and extracts key points.

[1112] It is determined that "Submission of market analysis report" is required before the next meeting, and this information is added to the calendar and notified to the user as a reminder.

[1113] Prompt Sentence Examples

[1114] Example prompt 1:

[1115] "Please provide a meeting schedule. Please also provide key points about your next meeting."

[1116] Example prompt 2:

[1117] "Please suggest what social activities we can add to the schedule during long meetings."

[1118] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1119] Step 1:

[1120] The server retrieves the user's schedule information from an external scheduling system (e.g., Google Calendar, Microsoft Outlook).

[1121] Input: User's calendar information (via API)

[1122] Output: A dataset of retrieved schedule information

[1123] Specific operation: The server uses an API to read the user's schedule information from an external system. For example, it uses the Google Calendar API to get the schedule information for the next week.

[1124] Step 2:

[1125] The server analyzes the schedule information it obtains and detects consecutive meetings and tasks.

[1126] Input: Dataset of acquired schedule information

[1127] Output: Analysis results (identification of consecutive meetings and tasks)

[1128] Specific operation: The server sorts the schedule information in chronological order and executes logic to check whether consecutive meetings exceed a certain time (e.g., 3 hours).

[1129] Step 3:

[1130] The server determines the need for social activities and suggests appropriate social activities.

[1131] Input: Analysis results (identification of consecutive meetings and tasks)

[1132] Output: A list of suggested suitable social activities

[1133] Specific operation: When consecutive meetings are detected, the server generates a list of suggested social activities for the user, such as "two minutes of stretching" or "five minutes of chatting."

[1134] Step 4:

[1135] The terminal displays the social activity suggestions received from the server to the user.

[1136] Input: A list of suggested social activities sent by the server

[1137] Output: A proposed interface for the user to see.

[1138] Specific operation: The terminal displays the suggested interaction activities on the user interface, allowing the user to confirm the suggestions.

[1139] Step 5:

[1140] The user selects a suggested social activity on the device.

[1141] Input: A proposed interface for user display

[1142] Output: Selected interaction activities

[1143] Specific operation: The user uses the device interface to select the desired interaction activity from the suggested activities.

[1144] Step 6:

[1145] The server adds the user's selected social activity to the schedule.

[1146] Input: Selected interaction activity

[1147] Output: Updated schedule

[1148] Specific operation: The server schedules the interaction activity selected by the user and adds it as an appointment.

[1149] Step 7:

[1150] The terminal collects audio data during the meeting in real time and sends it to the server.

[1151] Input: Real-time audio in a meeting

[1152] Output: Collected audio data (digital format)

[1153] Specific operation: The device uses a voice input device to record audio during the meeting and save it in digital format. This data is then streamed to the server.

[1154] Step 8:

[1155] The server converts the received voice data into text data.

[1156] Input: Collected audio data

[1157] Output: Text data converted from audio

[1158] Specific operation: The server uses voice recognition technology (e.g., Google Cloud Speech-to-Text API) to convert the voice data into text data.

[1159] Step 9:

[1160] The server extracts the main points from the converted text data.

[1161] Input: Text data converted from speech

[1162] Output: Extracted key information

[1163] How it works: The server uses natural language processing techniques to extract important information from the text data, such as what needs to be reported at the next meeting.

[1164] Step 10:

[1165] The server adds new tasks and reminders to the user's calendar based on the extracted key points.

[1166] Input: Extracted gist information

[1167] Output: Updated calendar information (with new tasks and reminders added)

[1168] What happens: The server adds upcoming meeting arrangements and other necessary tasks to the user's calendar.

[1169] Step 11:

[1170] The server sends new schedule details and notifications to the user terminal and notifies them.

[1171] Input: Updated calendar information

[1172] Output: Notification to user terminal

[1173] Specific operation: The server notifies the user by sending the updated schedule to the user's device via push notification or other notification method.

[1174] Step 12:

[1175] The user checks the notification on the device and approves or adjusts it if necessary.

[1176] Input: Notification to user terminal

[1177] Output: User approved and adjusted schedule

[1178] What happens: The user sees the notifications on their device, acknowledges the new task or reminder, adjusts their schedule if necessary, and confirms the changes.

[1179] (Application example 1)

[1180] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1181] Robots and machines working in factories are often subjected to continuous, long-term operation, which can result in problems such as overheating and abnormal vibrations. It can also be difficult for workers to detect abnormalities in real time and take appropriate action. This can lead to reduced production efficiency and increased risk of machine breakdowns. Furthermore, workers often do not take appropriate breaks or undergo maintenance during long periods of work, which also reduces the operating efficiency and lifespan of machines. The present invention aims to solve these problems.

[1182] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1183] In this invention, the server includes means for acquiring user schedule information, means for analyzing the acquired schedule information and determining the need for an ice breaker, means for suggesting appropriate ice breaker activities, means for adding the suggested ice breaker activities to the user's schedule, means for managing the work schedule of the industrial machinery and suggesting breaks between continuous work, and means for collecting work data of the industrial machinery in real time and detecting abnormalities. This reduces the burden on machinery in a factory due to long hours of operation and makes it possible to quickly detect and respond to abnormalities.

[1184] A "user" is someone who uses this system to improve schedule management and work efficiency.

[1185] "Schedule information" is data relating to plans or plans that are input or obtained by the user.

[1186] An "icebreaker" is an activity such as a short break or light exercise that takes place during long periods of work or meetings.

[1187] "Industrial machinery" refers to machinery and equipment used in industrial sites such as factories.

[1188] A "work schedule" is a schedule or timetable for work performed by industrial machines and workers.

[1189] A "break" is a temporary pause between successive tasks.

[1190] "Audio data" refers to digital data of conversations or audio recordings.

[1191] "Text data" refers to digital data converted into character information.

[1192] A "gist" is a particularly important part of a conversation or piece of information.

[1193] "Abnormal" means any operation or condition of a machine that deviates from normal operating conditions.

[1194] "Real-time" refers to data collection and processing occurring immediately, without delay.

[1195] "Notification" means informing the user of new information or alerts.

[1196] A "smart device" is a mobile device such as a smartphone or tablet that can connect to the Internet and run applications.

[1197] System Configuration

[1198] This invention is a business efficiency improvement system that integrates the acquisition and analysis of user schedule information, ice-breaker suggestions, automatic text conversion and key point extraction of conversation content, schedule creation and notification. This system is mainly composed of three elements: a server, a terminal, and a user.

[1199] Server Operation

[1200] The server operates using the following methods:

[1201] 1. Acquisition and analysis of schedule information

[1202] The server obtains the user's schedule information from an external scheduling system (e.g., Google Calendar, Microsoft Outlook API). It analyzes the obtained schedule information and determines whether an icebreaker is necessary. For example, if a long meeting or continuous work on industrial machinery is scheduled, it determines that an icebreaker is necessary in between.

[1203] 2. Propose and confirm icebreakers

[1204] The server generates a list of ice-breaking activities (e.g., two minutes of stretching, five minutes of maintenance check) based on the analysis results and suggests them to the user's device. The ice-breaking activities selected by the user are added to the schedule.

[1205] 3. Conversation text conversion and key points extraction

[1206] The server receives the audio data from the meeting and converts it into text using speech recognition software such as the Google Speech-to-Text API. The server then extracts key points from the converted text.

[1207] 4. Schedule Task Reminders

[1208] Based on the extracted key points, new tasks and reminders are added to the user's calendar using data analysis techniques using Python and the pandas library.

[1209] 5. Industrial Machinery Data Collection and Anomaly Detection

[1210] The server collects operational data (e.g., temperature, vibration, and operating time) from industrial machinery in real time and detects abnormalities. This data collection and analysis is done using Python and the pandas library. If an abnormality is detected, an alert is sent to the worker's smart device.

[1211] Device behavior

[1212] The terminal operates using the following means:

[1213] 1. Receiving and displaying schedule information

[1214] The terminal displays the schedule information received from the server to the user, who can then check and select the icebreaker activities suggested through the terminal.

[1215] 2. Collection of audio data

[1216] The terminal collects audio data during the meeting in real time and sends it to a server. The audio data is also recorded in digital format and can be viewed later.

[1217] 3. Display of notifications

[1218] The device displays notifications of new schedules and reminders received from the server to the user, who can then review, approve, and adjust the schedule.

[1219] User operations

[1220] The user interacts with the device using the following means:

[1221] 1. Check and select your schedule

[1222] The user checks the schedule through their device and selects an icebreaker activity suggested by the server. The selected activity is automatically added to the schedule.

[1223] 2. Record and review conversations

[1224] When starting a meeting, users can turn on the device's audio recording function to record the conversation. After the meeting, they can check the converted text data and extracted key points.

[1225] 3. Review and adjust notifications

[1226] Users can view notifications on their device, acknowledge new tasks and reminders, and adjust their schedules as needed.

[1227] Specific examples

[1228] The following are specific examples for carrying out the present invention:

[1229] Example 1: Long meetings and continuous work

[1230] For example, if a user has a three-hour continuous meeting or continuous work schedule for an industrial machine, the server analyzes the schedule and suggests suitable icebreakers (e.g., five-minute stretching or maintenance check) before, during, or after the meeting or work. The icebreaker activities selected by the user are incorporated into the schedule and notifications are set.

[1231] Example 2: Recording important information during meetings and detecting anomalies

[1232] A user starts a meeting and turns on the audio recording function on their device. The audio during the conversation is collected by the device and sent to the server. The server converts the audio into text and extracts key points. It determines that a "market analysis report" needs to be submitted before the next meeting, and this information is added to the calendar and notified to the user as a reminder. Furthermore, if an industrial machine detects abnormal vibrations, the server will issue a warning to the worker's smart device.

[1233] Prompt Sentence Examples

[1234] As a concrete example, consider the following prompt input to a generative AI model:

[1235] Enter the following data: a list of work schedules, activities, icebreaker suggestions, and work data (temperature, vibration, etc.). For example, "2023-10-10 09:00-12:00: Work A, 2023-10-10 12:00-13:00: Break."

[1236] The system suggests an icebreaker: "5 minutes of maintenance after 20 minutes of work." It analyzes data and issues an alert if it detects an abnormality.

[1237] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1238] Step 1:

[1239] The server retrieves the user's schedule information using the API of an external scheduling system (e.g., Google Calendar or Microsoft Outlook), which requires an API key and a calendar ID as input. The output is the retrieved schedule information.

[1240] Step 2:

[1241] The server analyzes the schedule information it has acquired and detects long meetings and work. The input is the schedule information acquired in step 1, and the data is analyzed to extract events with long meetings or continuous work. This analysis identifies locations where ice breakers are needed. The output is the analysis results, i.e., a list of events where ice breakers are needed.

[1242] Step 3:

[1243] The server proposes ice-breaking activities (e.g., 2 minutes of stretching, 5 minutes of maintenance check) based on the analysis results. The input is the list of events requiring ice-breaking obtained in step 2. The server assigns ice-breaking activities to these events and generates a list of activities to propose to the user. The output is a list of proposed ice-breaking activities.

[1244] Step 4:

[1245] The terminal displays the list of icebreaker activities received from the server to the user. The input is the list of icebreaker activities generated in step 3, and the terminal displays it on the user's interface so that the user can select one. The output is the icebreaker activity selected by the user.

[1246] Step 5:

[1247] The server adds the icebreaker activity selected by the user to the schedule and notifies the user. The input is the icebreaker activity selected by the user in step 4, the server incorporates it into the schedule, and generates a new schedule. The output is the updated schedule information.

[1248] Step 6:

[1249] The terminal collects audio data during the meeting in real time and sends it to the server. The input is the audio data recorded during the meeting, which the terminal records in digital format and sends to the server. The output is the audio data sent to the server.

[1250] Step 7:

[1251] The server converts the audio data into text data and extracts the main points. The input is the audio data sent in step 6, and the server converts the audio data into text data using the Google Speech-to-Text API. It then performs natural language processing to extract the main points from the text. The output is a list of main points.

[1252] Step 8:

[1253] The server adds new tasks and reminders to the user's schedule based on the extracted key points. The input is the list of key points generated in step 7, and the server automatically adds new tasks and reminders to the schedule based on this. The output is the updated schedule information.

[1254] Step 9:

[1255] The server collects operational data from industrial machinery in real time and detects abnormalities. The input is the operational data of the industrial machinery (such as temperature, vibration, operating time, etc.), which the server monitors in real time and analyzes abnormalities. The output is an alert when an abnormality is detected.

[1256] Step 10:

[1257] When the server detects an abnormality, it issues a warning to the worker's smart device. The input is the abnormal data detected in step 9, and a warning message is generated based on that content and notified to the worker's smart device. The output is the notification sent to the worker.

[1258] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1259] System configuration

[1260] This invention is a business efficiency improvement system that integrates the acquisition and analysis of user schedule information, ice-breaker suggestions, automatic text conversion and key point extraction of conversation content, schedule creation and notifications, and also incorporates an emotion engine. The roles and specific operations of the server, terminal, and user are explained below.

[1261] Server Operation

[1262] 1. Acquisition and analysis of schedule information

[1263] The server retrieves the user's schedule information from an external scheduling system (e.g., Google Calendar, Microsoft Outlook). It analyzes the retrieved schedule information and determines whether an icebreaker is necessary. For example, if there are consecutive long meetings, it determines that an icebreaker is necessary.

[1264] 2. Propose and confirm icebreakers

[1265] Based on the analysis results, the server generates a list of ice-breaking activities (e.g., two minutes of stretching, five minutes of chatting) and suggests them to the user's device. The ice-breaking activities selected by the user are added to the schedule. Furthermore, the selection of ice-breaking activities is optimized based on the analysis results of the emotion engine.

[1266] 3. Conversation text conversion and key points extraction

[1267] The server receives the voice data from the meeting and converts it into text using speech recognition technology. Key points (e.g., matters that need to be reported at the next meeting) are extracted from the converted text data. The emotion engine also recognizes the user's emotions in real time during the conversation and adjusts the extraction of key points and the addition of tasks according to their emotional state.

[1268] 4. Schedule Task Reminders

[1269] Based on the extracted key points, new tasks and reminders are added to the user's calendar. For example, an item such as "Prepare for the next meeting" is automatically added. The emotion engine analyzes the user's stress and fatigue levels and dynamically adjusts the content and timing of reminders based on the results.

[1270] 5. Notification

[1271] The server sends new schedule information and reminders to the user's device, notifying the user. The user checks the notification and adjusts the schedule as necessary.

[1272] Device behavior

[1273] 1. Receiving and displaying schedule information

[1274] The terminal displays the schedule information received from the server to the user, who can then check and select the icebreaker activities suggested through the terminal.

[1275] 2. Collection of audio data

[1276] The device collects audio data during the meeting in real time and sends it to a server. The audio data is also recorded in digital format and can be viewed later.

[1277] 3. Use of Emotion Engine

[1278] The device analyzes the user's facial expressions and voice tone and uses an emotion engine to analyze the user's emotional state in real time. The analysis results of the emotion engine are sent to a server and used to suggest icebreakers, extract key points, and add tasks.

[1279] 4. Display of notifications

[1280] The device displays notifications of new schedules and reminders received from the server to the user, who can then review, approve, and adjust the schedule.

[1281] User operations

[1282] 1. Check and select your schedule

[1283] The user checks the schedule on their device and selects an icebreaker activity suggested by the server. The selected activity is automatically added to the schedule.

[1284] 2. Record and review conversations

[1285] When a user starts a meeting, they can turn on the device's audio recording function to record the conversation. After the meeting, they can check the converted text data and extracted key points.

[1286] 3. Review and adjust notifications

[1287] Users can view notifications on their device, acknowledge new tasks and reminders, and adjust their schedules as needed.

[1288] Specific examples

[1289] Example 1: Long meetings

[1290] A user enters a week's schedule, including a three-hour series of meetings.

[1291] The server analyzes this and suggests appropriate icebreakers before, during, or after the meeting. The emotion engine analyzes the user's fatigue level and recommends icebreakers to relieve stress.

[1292] The user selects "5 minutes of stretching" and the server schedules it and sets up a notification.

[1293] Example 2: Recording important information during a meeting

[1294] A user starts a conference and turns on the audio recording function of the terminal. The audio during the conversation is collected by the terminal and sent to the server.

[1295] The server converts the speech into text and extracts key points, while an emotion engine analyzes the user's emotions and adjusts the key points taking into account the importance and urgency of the topic.

[1296] It is determined that "Submission of market analysis report" is required before the next meeting, and this information is added to the calendar and notified to the user as a reminder.

[1297] As described above, the present invention improves the efficiency of user schedule management and the recording and analysis of conversation content, and by combining it with an emotion engine, it provides optimal business support based on the user's emotional state.

[1298] The processing flow will be explained below.

[1299] Step 1:

[1300] The user accesses the tool and starts the schedule management service.

[1301] Step 2:

[1302] The device retrieves the user's calendar data from an external scheduling service such as Google Calendar or Microsoft Outlook, and then sends this information to the server.

[1303] Step 3:

[1304] The server analyzes the received schedule data and identifies when an icebreaker is needed, such as during a long meeting or when consecutive tasks are required.

[1305] Step 4:

[1306] The server determines the need for an icebreaker based on the analysis results, and if necessary, selects an appropriate icebreaker activity from the database.

[1307] Step 5:

[1308] The server generates a proposal for the selected icebreaker activity and transmits it to the terminal.

[1309] Step 6:

[1310] The device displays icebreaker suggestions to the user, who can then select any icebreaker activity from the suggestions.

[1311] Step 7:

[1312] The device sends the icebreaker activity selected by the user to the server, which adds it to the user's schedule.

[1313] Step 8:

[1314] When a user starts a conference, the device turns on the audio recording function, and the device collects the audio data of the conversation in real time and transmits it to the server.

[1315] Step 9:

[1316] The server converts the voice data into text data using voice recognition technology.

[1317] Step 10:

[1318] The server analyzes the converted text data and extracts important points, for example, identifying tasks such as "items that need to be reported at the next meeting."

[1319] Step 11:

[1320] The server adds new tasks and reminders to the user's calendar based on the extracted key points.

[1321] Step 12:

[1322] The server sends new schedules and reminders to the device and creates notifications, which the device displays to the user.

[1323] Step 13:

[1324] The user checks the notification and adjusts the schedule as necessary.

[1325] Example 2

[1326] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1327] Conventional schedule management systems do not provide effective ways to reduce fatigue and stress caused by continuous meetings and tasks. Furthermore, the efficient recording of important comments and decisions during meetings and the manual task management based on these records result in reduced work efficiency. Furthermore, the lack of timely reminders and notifications that take into account the user's emotional state makes it difficult for users to maintain their productivity.

[1328] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for acquiring user schedule information, means for analyzing the acquired schedule information and determining the need for an icebreaker, means for proposing appropriate icebreaker activities, means for adding the proposed icebreaker activities to the user's schedule, means for analyzing the user's emotional state and optimizing the icebreaker activities, means for collecting voice data of the conversation, means for converting the collected voice data into text data, means for extracting key points from the converted text data, means for adding new tasks or reminders to the user's schedule based on the extracted key points, and means for sending and notifying the new schedule or reminder to the user terminal. This effectively reduces fatigue and stress caused by continuous meetings and tasks for the user, efficiently records important comments and decisions made during meetings, automates task management based on the records, and enables appropriate timing of reminders and notifications taking the user's emotional state into consideration.

[1329] "Schedule information" refers to data relating to the user's planned dates and activities.

[1330] "Acquisition means" refers to the methods or functions for collecting data or information from external or internal systems.

[1331] "Analysis means" refers to the methods and functions used to analyze acquired data and information and understand its content and meaning.

[1332] "Ice-breaking activities" are refreshing activities or short recreational activities to relieve fatigue and stress caused by long meetings or task completion.

[1333] "Suggestion means" refers to methods or functions for presenting appropriate information or activities to users.

[1334] "Additional means" refers to methods or functions for inserting new information or activities into a user's schedule.

[1335] "Emotional state" refers to the mental state or mood of the user.

[1336] "Audio data" refers to data that has been recorded in digital form as a conversation or voice signal.

[1337] "Text data" refers to data obtained by converting voice data into character information.

[1338] "Key point extraction means" refers to a method or function for extracting important information or key content from text data.

[1339] A "task" is a specific task or job that a user must undertake.

[1340] A "reminder" is a notification or alarm that reminds a user of a specific date, time, or event.

[1341] "Notification means" refers to the method or function for notifying the user of new information or reminders.

[1342] A "terminal" is a computing device that is directly operated by a user.

[1343] A "server" is a central computer system that processes and manages data on a network.

[1344] This is a business efficiency improvement system that integrates the acquisition and analysis of user schedule information, ice-breaker suggestions, automatic text conversion and key point extraction of conversation content, and task and reminder scheduling and notification. Furthermore, it incorporates an emotion engine to provide optimal support based on the user's emotional state.

[1345] System Configuration

[1346] Server Operation

[1347] The server first obtains the user's schedule information from an external scheduling system (e.g., Google Calendar or Microsoft Outlook) via API. The obtained schedule information is stored in an internal database, and an analysis engine uses this information to determine whether an icebreaker is necessary.

[1348] Next, based on the analysis results, the server lists multiple ice-breaking activities (e.g., two minutes of stretching, five minutes of chatting) and sends this list in JSON format to the user's device. At the same time, the emotion engine analyzes the user's stress level and fatigue level to select the most appropriate ice-breaking activity. The server adds the ice-breaking activity selected by the user to the schedule and notifies the user's device again.

[1349] During the meeting, the server receives real-time voice data from the devices and converts it into text data using the Google Speech-to-Text API. The converted text data is stored in a database, and a natural language processing (NLP) engine extracts key points. An emotion engine analyzes the user's emotions and adjusts the importance of the key points.

[1350] Furthermore, new tasks and reminders are added to the user's calendar based on the extracted key points. The emotion engine dynamically adjusts the content and timing of reminders, taking into account the user's stress level and fatigue level. The server then sends the new schedule and reminders to the user's device and notifies the user.

[1351] Device behavior

[1352] The device displays the schedule information received from the server to the user via a GUI. The user can then review and select suggested icebreaker activities. During the meeting, the device's microphone collects audio data in real time, digitizes it, and sends it to the server. This audio data is also stored locally and can be played back later.

[1353] The device uses its built-in camera and microphone to analyze the user's facial expressions and voice tone and input them into the emotion engine. The analysis results are sent to the server and reflected in icebreaker suggestions and key point extraction. In addition, the device receives new schedule and reminder notifications from the server and displays them as push notifications, allowing the user to confirm, approve, or adjust them.

[1354] User operations

[1355] The user checks the schedule on their device and selects an icebreaker activity suggested by the server. The selected activity is automatically added to the schedule. When starting a meeting, the user turns on the device's audio recording function to record the conversation. After the meeting, the converted text data and extracted key points can be viewed. The user can also check notifications displayed on the device, approve new tasks and reminders, and adjust the schedule as necessary.

[1356] Specific examples

[1357] Example 1: Long meetings

[1358] A user enters a week's worth of events into Google Calendar, and one day has three consecutive meetings scheduled.

[1359] The server analyzes this and suggests appropriate icebreakers before, during, or after the meeting. The emotion engine analyzes the user's fatigue level and recommends "5 minutes of stretching" to relieve stress.

[1360] The user selects "5 minutes of stretching" and the server schedules it and sets up a notification.

[1361] Example 2: Recording important information during a meeting

[1362] A user starts a conference and turns on the audio recording function of the device. The audio during the conversation is collected by the device and transmitted to the server in real time.

[1363] The server converts the speech into text and extracts key points. The emotion engine analyzes the user's emotions and adjusts the key points based on the importance and urgency of the topic.

[1364] It turns out that "Submission of Market Analysis Report" is required before the next meeting, and this information is added to the calendar and notified to the user as a reminder.

[1365] Prompt Sentence Examples

[1366] "Use your schedule and sentiment data to suggest the best icebreaker activities, extract key points from meetings, and add upcoming tasks to your schedule."

[1367] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1368] Step 1: Get schedule information

[1369] The server gets information from an external scheduling system

[1370] Input: User authentication information, API key

[1371] Output: User schedule information (JSON format)

[1372] How it works: The server uses the APIs of Google Calendar and Microsoft Outlook to retrieve user schedule information. This information is sent to the server in JSON format and stored in an internal database.

[1373] Step 2: Analyze schedule information

[1374] The server analyzes the schedule information and determines whether an icebreaker is necessary.

[1375] Input: Schedule information (JSON format)

[1376] Output: Analysis results (flag indicating need for icebreaker)

[1377] How it works: The server's analysis engine analyzes schedule information and determines whether an icebreaker is necessary if there are consecutive "meetings" or "long-term tasks." For example, if the interval between consecutive meetings exceeds a certain time, it determines that an icebreaker is necessary.

[1378] Step 3: Suggest an icebreaker activity

[1379] Your server will suggest appropriate icebreaker activities

[1380] Input: Analysis results, emotion engine data

[1381] Output: Icebreaker proposal list (JSON format)

[1382] How it works: The server suggests icebreaker activities suitable for the user based on the analysis results and emotion engine data. The list of suggestions is sent to the device in JSON format.

[1383] Step 4: Determine icebreaker activities

[1384] The server adds icebreaker activities to the schedule based on the user's selection.

[1385] Input: User's choice

[1386] Output: Updated schedule information

[1387] How it works: After a user selects an icebreaker activity on the device, the server adds the selection to the schedule and sends the updated schedule information back to the device.

[1388] Step 5: Collecting audio data from the conversation

[1389] The device collects audio data during the meeting and sends it to the server.

[1390] Input: Audio during the meeting

[1391] Output: Digital audio data

[1392] How it works: The device's microphone collects audio during the meeting in real time and sends it digitally to a server, where the audio data is stored locally.

[1393] Step 6: Convert audio data to text

[1394] The server converts the audio data into text data.

[1395] Input: Audio data

[1396] Output: Text data

[1397] How it works: The server uses the Google Speech-to-Text API to convert the received audio data into text data, which is then stored in a database.

[1398] Step 7: Extracting key points

[1399] The server extracts key points from the text data

[1400] Input: Text data

[1401] Output: Gist list

[1402] How it works: The server's natural language processing (NLP) engine analyzes the text data and extracts key conversation points and action items. The emotion engine also analyzes the user's emotions at this stage and adjusts the importance of key points.

[1403] Step 8: Schedule task reminders

[1404] The server adds new tasks and reminders to your schedule

[1405] Input: Key points list, emotion engine data

[1406] Output: Updated schedule

[1407] How it works: Based on the extracted key points, the server adds new tasks and reminders to the user's calendar. The emotion engine dynamically adjusts the timing of reminders based on the user's stress and fatigue levels.

[1408] Step 9: Notification

[1409] The server sends new schedules and reminders to the user's device and notifies them.

[1410] Input: Updated Schedule

[1411] Output: Notification message

[1412] How it works: The server sends updated schedule information and reminders to the device, which are displayed to the user as push notifications. The user can view the notifications and accept or adjust them as needed.

[1413] (Application example 2)

[1414] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1415] In modern factories and offices, workers often find it difficult to take appropriate breaks or refresh themselves when working long hours continuously or in the middle of important tasks. Furthermore, systems for accurately recording the work performed by workers and later reviewing and utilizing this information are often inadequate. Furthermore, to reduce workers' stress and fatigue, it is necessary to provide optimal intervals based on their emotional state. These challenges can reduce work efficiency and ultimately hinder improvements in productivity and work quality. A solution to these problems and achieve both work efficiency and worker health is needed.

[1416] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1417] In this invention, the server includes means for acquiring a user's schedule information, means for analyzing the acquired schedule information and determining the need for a break, means for suggesting appropriate break activities, means for adding the suggested break activities to the user's schedule, means for analyzing the user's emotional state using an emotion engine and optimizing the selection of break activities, means for collecting voice data during work and converting it into text data, means for extracting important work points from the converted text data, means for adding new tasks and reminders to the user's schedule based on the extracted points, and means for displaying notifications to the user and allowing the user to confirm, approve, and adjust the suggested break activities and added tasks and reminders. This enables workers to take breaks at appropriate times, maximize work efficiency, and accurately record and manage important work content and key points.

[1418] "User" refers to the workers and employees who use the system.

[1419] "Setup information" refers to information that indicates the work schedules, plans, and tasks of workers or employees.

[1420] "Emotion engine" refers to technology that analyzes a user's facial expressions and vocal tone to recognize and analyze their emotional state (e.g., stress or fatigue) in real time.

[1421] "Rest activities" refer to short breaks or refreshing activities (e.g., short stretching or chatting) that users take between tasks to reduce fatigue and stress.

[1422] "Voice data" refers to digital data of a user's voice collected while working or talking.

[1423] "Text data" refers to sentence data obtained by converting voice data into characters.

[1424] "Key points" are pieces of information or tasks that are considered particularly important in a work progress or conversation.

[1425] A "task" refers to a discrete task or activity that a user must perform.

[1426] A "reminder" is a notification or reminder that helps users remember specific tasks or appointments.

[1427] This system acquires setup information for workers and employees, proposes and schedules optimal break activities to improve work efficiency, and automatically records and manages important work content and key points of conversations. The roles and specific operations of the server, terminal, and user are explained below.

[1428] Server Operation

[1429] Acquisition and analysis of setup information

[1430] The server acquires setup information for workers and employees from the factory management system or business management system (MES or ERP system). It analyzes the acquired setup information and determines whether breaks are necessary. For example, if a long period of continuous work is scheduled, it determines that a break is necessary in between.

[1431] Proposing and confirming break activities

[1432] Based on the analysis results, the server generates a list of break activities (e.g., short breaks and stretching) and suggests them to the user's device. It uses an emotion engine to analyze the user's emotional state and selects the optimal break activity based on the results. The break activity selected by the user is added to the schedule.

[1433] Speech data text conversion and gist extraction

[1434] The server receives the working voice data and converts it into text using speech recognition technology (e.g., the speech_recognition library). It then uses a generative AI model (e.g., GPT-3.5) to extract key points from the converted text. The extraction of key points is optimized according to the analysis results of the emotion engine.

[1435] Schedule tasks and reminders

[1436] The server adds new tasks and reminders to the user's schedule based on the extracted key points. For example, information such as "Parts for the next process are missing" is added to the schedule and notified to the user as a reminder.

[1437] notification

[1438] The server sends new schedule information and reminders to the user's device, and notifies the user. The user can check the notification and adjust the schedule as necessary.

[1439] Device behavior

[1440] Receiving and displaying setup information

[1441] The terminal displays the schedule information received from the server to the user, who can then check and select the suggested break activities through the terminal.

[1442] Audio data collection

[1443] The device collects voice data in real time while the worker is working and sends it to a server, where it is recorded in digital format and can be viewed later.

[1444] Use of emotion engine

[1445] The device analyzes the user's facial expressions and voice tone and uses an emotion engine to analyze the user's emotional state in real time. The analysis results of the emotion engine are sent to a server and used to suggest break activities, extract key points, and add tasks.

[1446] Viewing notifications

[1447] The device displays notifications of new schedules and reminders received from the server to the user, who can then review, approve, and adjust the schedule.

[1448] User operations

[1449] Check and select setup

[1450] The user checks the schedule through the terminal and selects the break activity suggested by the server, which is then automatically added to the schedule.

[1451] Recording and checking work details

[1452] When a user starts a task, they can turn on the device's voice recording function to record the task's progress. After the task is completed, they can check the converted text data and extracted key points.

[1453] Review and adjust notifications

[1454] Users can view notifications on their device, acknowledge new tasks and reminders, and adjust their schedules as needed.

[1455] Specific examples

[1456] Example 1: Long working hours

[1457] The user enters their weekly schedule, and one day is scheduled for six hours of continuous robot operation.

[1458] The server analyzes this and suggests appropriate rest activities before, during, or after the operation. The emotion engine analyzes the user's fatigue level and recommends short breaks to relieve stress.

[1459] The user selects "5 minutes of stretching" and the server schedules it and sets up a notification.

[1460] Example 2: Recording important information during work

[1461] The user starts working and turns on the device's voice recording function. The voice recorded during the work is collected by the device and sent to the server.

[1462] The server converts the speech into text and extracts key points, while the emotion engine analyzes the user's emotions and optimizes the key points in real time.

[1463] If a "parts shortage" is discovered by the time the next process is completed, this information is added to the schedule and a reminder is sent to the user.

[1464] Example of a generative AI model prompt:

[1465] Prompt Sentence Examples

[1466] Text from audio recording: "We've encountered a parts supply issue. We are running low on parts needed for the next operation. We need to reconsider our parts supply plan."

[1467] Gist Extraction Generation AI Prompt: "Please extract the key points from the audio recording above."

[1468] Key point: "Parts supply problem. Parts for the next process are in short supply. Parts supply plan needs to be reconsidered."

[1469] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1470] Step 1:

[1471] The server obtains setup information for workers and employees from the factory management system or business management system (MES or ERP system). This setup information includes work schedules, plans, and tasks. It analyzes this information and evaluates the length and continuity of work time. It receives the worker's setup information (schedule, plan, tasks) as input and determines the need for breaks as a result of the analysis.

[1472] Step 2:

[1473] The server generates a list of rest activities based on the analysis results. It uses the emotion engine to analyze the user's emotional state and selects the optimal rest activity taking the results into consideration. For example, if the user has been working for a long time or is feeling emotional stress, it suggests stretching or a short break. The server uses the analysis results and emotion engine data as input and generates a list of rest activities as output.

[1474] Step 3:

[1475] The terminal receives the list of break activities sent from the server and displays it to the user. The user checks the proposed break activities through the terminal and selects one. The terminal receives the list of break activities as input and sends the selected break activity to the server as output.

[1476] Step 4:

[1477] The server adds the break activity selected by the user to the setup information and updates the schedule. This updated schedule information is sent to the user's terminal and notified. The server receives the break activity selected by the user as input, generates updated schedule information as output, and sends it to the terminal.

[1478] Step 5:

[1479] When a user starts working, the terminal turns on the voice recording function, collects voice data during work, and sends this data to the server. The terminal collects the user's voice data as input and sends the voice data to the server as output.

[1480] Step 6:

[1481] The server converts the received voice data into text data using speech recognition technology (e.g., the speech_recognition library), and extracts key points from the converted text data using a generative AI model (e.g., GPT-3.5). It receives voice data as input and generates text data and key points as output.

[1482] Step 7:

[1483] The server adds new tasks and reminders to the schedule information based on the extracted key points. This information is sent to the user's terminal and displayed as a notification. As input, the server generates new tasks and reminders based on the extracted key points, and as output, adds these to the schedule and sends them to the terminal.

[1484] Step 8:

[1485] The user can check the notifications displayed on the device, acknowledge new tasks and reminders, and adjust the schedule as needed. The input is the notification displayed by the device, and the output is the necessary adjustment.

[1486] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1487] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1488] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1489] [Fourth embodiment]

[1490] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1491] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1492] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1493] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1494] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1495] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1496] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1497] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1498] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1499] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1500] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1501] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1502] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1503] System configuration

[1504] This invention is a business efficiency improvement system that integrates the acquisition and analysis of user schedule information, ice-breaker suggestions, automatic text conversion and key point extraction of conversation content, schedule creation and notification. The roles and specific operations of the server, terminal, and user are explained below.

[1505] Server Operation

[1506] 1. Acquisition and analysis of schedule information

[1507] The server retrieves the user's schedule information from an external scheduling system (e.g., Google Calendar, Microsoft Outlook). It analyzes the retrieved schedule information and determines whether an icebreaker is necessary. For example, if there are consecutive long meetings, it determines that an icebreaker is necessary.

[1508] 2. Propose and confirm icebreakers

[1509] Based on the analysis results, the server generates a list of ice-breaking activities (e.g., two minutes of stretching, five minutes of chatting) and suggests them to the user's device. The ice-breaking activity selected by the user is added to the schedule.

[1510] 3. Conversation text conversion and key points extraction

[1511] The server receives the voice data from the meeting and converts it into text data using speech recognition technology. Key points (e.g., matters that need to be reported at the next meeting) are extracted from the converted text data.

[1512] 4. Schedule Task Reminders

[1513] Based on the extracted key points, new tasks and reminders are added to the user's calendar, for example, an item such as "Prepare for the next meeting" is automatically added.

[1514] 5. Notification

[1515] The server sends new schedule details and reminders to the user's device, notifying the user. The user checks the notifications and adjusts the schedule as necessary.

[1516] Device behavior

[1517] 1. Receiving and displaying schedule information

[1518] The terminal displays the schedule information received from the server to the user, who can then check and select the icebreaker activities suggested through the terminal.

[1519] 2. Collection of audio data

[1520] The device collects audio data during the meeting in real time and sends it to a server. The audio data is also recorded in digital format and can be viewed later.

[1521] 3. Display of notifications

[1522] The device displays notifications of new schedules and reminders received from the server to the user, who can then review, approve, and adjust the schedule.

[1523] User operations

[1524] 1. Check and select your schedule

[1525] The user checks the schedule on their device and selects an icebreaker activity suggested by the server. The selected activity is automatically added to the schedule.

[1526] 2. Record and review conversations

[1527] When a user starts a meeting, they can turn on the device's audio recording function to record the conversation. After the meeting, they can check the converted text data and extracted key points.

[1528] 3. Review and adjust notifications

[1529] Users can view notifications on their device, acknowledge new tasks and reminders, and adjust their schedules as needed.

[1530] Specific examples

[1531] Example 1: Long meetings

[1532] A user enters a week's schedule, including a three-hour series of meetings.

[1533] The server analyzes this and suggests appropriate icebreakers before, during, or after the meeting.

[1534] The user selects "5 minutes of stretching" and the server schedules it and sets up a notification.

[1535] Example 2: Recording important information during a meeting

[1536] A user starts a conference and turns on the audio recording function of the terminal. The audio during the conversation is collected by the terminal and sent to the server.

[1537] The server converts the speech into text and extracts key points.

[1538] It is determined that "Submission of market analysis report" is required before the next meeting, and this information is added to the calendar and notified to the user as a reminder.

[1539] As described above, the present invention is a useful system that improves the efficiency of users' schedule management and the recording and analysis of conversation content, thereby improving work efficiency.

[1540] The processing flow will be explained below.

[1541] Step 1:

[1542] The user accesses the tool and starts the schedule management service.

[1543] Step 2:

[1544] The device retrieves the user's calendar data from an external scheduling service such as Google Calendar or Microsoft Outlook, and then sends this information to the server.

[1545] Step 3:

[1546] The server analyzes the received schedule data and identifies when an icebreaker is needed, such as during a long meeting or when consecutive tasks are required.

[1547] Step 4:

[1548] The server determines the need for an icebreaker based on the analysis results, and if necessary, selects an appropriate icebreaker activity from the database.

[1549] Step 5:

[1550] The server generates a proposal for the selected icebreaker activity and transmits it to the terminal.

[1551] Step 6:

[1552] The device displays icebreaker suggestions to the user, who can then select any icebreaker activity from the suggestions.

[1553] Step 7:

[1554] The device sends the icebreaker activity selected by the user to the server, which adds it to the user's schedule.

[1555] Step 8:

[1556] When a user starts a conference, the device turns on the audio recording function, and the device collects the audio data of the conversation in real time and transmits it to the server.

[1557] Step 9:

[1558] The server converts the voice data into text data using voice recognition technology.

[1559] Step 10:

[1560] The server analyzes the converted text data and extracts important points, for example, identifying tasks such as "items that need to be reported at the next meeting."

[1561] Step 11:

[1562] The server adds new tasks and reminders to the user's calendar based on the extracted key points.

[1563] Step 12:

[1564] The server sends new schedules and reminders to the device and creates notifications, which the device displays to the user.

[1565] Step 13:

[1566] The user checks the notification and adjusts the schedule as necessary.

[1567] Example 1

[1568] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1569] In today's business environment, users need to efficiently manage numerous meetings and tasks. However, long meetings and continuous tasks often reduce users' concentration and productivity. Furthermore, there are cases where important comments or decisions made during meetings are not recorded, hindering subsequent work. There is a need for a system that can solve these issues and improve users' work efficiency.

[1570] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1571] In this invention, the server includes means for acquiring user schedule information, means for analyzing the acquired schedule information and determining the need for social activities, means for suggesting appropriate social activities, means for adding the suggested social activities to the user's schedule, means for collecting meeting audio information, means for converting the collected audio information into text data, means for extracting key points from the converted text data, means for adding new tasks and notifications to the user's schedule based on the extracted key points, and means for sending the new schedule contents and notifications to the user's terminal. This makes it possible to efficiently manage the user's work schedule and automatically record and notify important information.

[1572] The "means for acquiring user's schedule information" is a function for collecting schedule information from the user's calendar data or scheduling system.

[1573] The "means for analyzing schedule information and determining the necessity of social activities" is a function for analyzing the user's schedule based on the acquired schedule information and evaluating whether social activities are necessary.

[1574] The "means for suggesting appropriate social activities" is a function for suggesting appropriate relaxation methods and social activities to the user based on the analysis results.

[1575] The "means for adding an interaction activity to a user's schedule" is a function for automatically incorporating an interaction activity selected by the user into a schedule and adding it as a schedule.

[1576] The "means for collecting audio information from a meeting" is a function for digitally recording audio during a meeting and storing it for later processing.

[1577] The "means for converting collected voice information into text data" is a function that uses voice recognition technology to convert voice data into text data.

[1578] "Means for extracting key points from text data" is a function that uses natural language processing technology to extract important information and decisions from meeting content converted into text data.

[1579] The "means for adding new tasks and notifications to the user's schedule" is a function for automatically adding related tasks and reminders to the user's schedule based on the extracted key points.

[1580] "Means for sending new schedule details and notifications to the user's device" refers to a function for sending new schedule and reminder information to the user's device and notifying them via push notifications, etc.

[1581] The "means for checking and selecting suggested social activities on the user's terminal" is a function that allows the user to view suggested social activities through the terminal and select a desired activity.

[1582] "Means for reviewing, approving, and adjusting notifications displayed on the user's device" refers to a function that allows the user to review notifications displayed on the device, approve their contents, and change the schedule as necessary.

[1583] System configuration

[1584] This invention is a business efficiency system that integrates the acquisition and analysis of user schedule information, suggestion of social activities, automatic text conversion and key point extraction of conversation content, and scheduling and notification of new tasks and notifications. The roles and specific operations of the server, terminal, and user are explained below.

[1585] Server Operation

[1586] The server retrieves the user's schedule information from external scheduling systems such as Google Calendar or Microsoft Outlook. It analyzes the retrieved schedule information, detects consecutive meetings or tasks, and determines whether social activities are necessary. For example, if there are consecutive long meetings, it determines that social activities such as stretching or chatting are necessary in between.

[1587] Next, the server generates a list of appropriate social activities based on the analysis results and suggests them to the user's device. The social activities selected by the user are added to the schedule, or new events are added. The server also receives audio information from the meeting in real time and converts it into text data using speech recognition technology, such as the Google Cloud Speech-to-Text API. The server then extracts key points from the converted text data and identifies important information, such as what needs to be reported at the next meeting.

[1588] The server then adds new tasks and reminders to the user's calendar based on the extracted key points. For example, an item such as "Prepare for the next meeting" is automatically added. Finally, the server sends the new schedule and notifications to the user's device, notifying them via push notifications or other means.

[1589] Device behavior

[1590] The terminal displays the schedule information received from the server to the user. The user can check and select suggested social activities through the terminal. The terminal also collects audio data during the meeting in real time and sends it to the server. This audio data is recorded in digital format and can be checked later. The terminal displays notifications of new appointments and reminders received from the server to the user. The user can check, approve, and adjust the schedule from these notifications.

[1591] User operations

[1592] The user checks the schedule through the device and selects a social activity suggested by the server. The selected activity is automatically added to the schedule. When starting a meeting, the user turns on the device's audio recording function to record the conversation. After the meeting, the user can check the converted text data and extracted key points. The user can also check notifications displayed on the device and approve new tasks and reminders. The schedule can also be adjusted as necessary.

[1593] Specific examples

[1594] Example 1: Long meetings

[1595] A user enters a week's worth of events into Google Calendar, for example, one day includes three consecutive hours of meetings.

[1596] The server analyzes this and suggests appropriate social activities before, during, or after the meeting.

[1597] The user selects "5 minutes of stretching" and the server schedules it and sets up a notification.

[1598] Example 2: Recording important information during a meeting

[1599] A user starts a conference and turns on the audio recording function of the terminal. The audio during the conversation is collected by the terminal and sent to the server.

[1600] The server converts the speech into text and extracts key points.

[1601] It is determined that "Submission of market analysis report" is required before the next meeting, and this information is added to the calendar and notified to the user as a reminder.

[1602] Prompt Sentence Examples

[1603] Example prompt 1:

[1604] "Please provide a meeting schedule. Please also provide key points about your next meeting."

[1605] Example prompt 2:

[1606] "Please suggest what social activities we can add to the schedule during long meetings."

[1607] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1608] Step 1:

[1609] The server retrieves the user's schedule information from an external scheduling system (e.g., Google Calendar, Microsoft Outlook).

[1610] Input: User's calendar information (via API)

[1611] Output: A dataset of retrieved schedule information

[1612] Specific operation: The server uses an API to read the user's schedule information from an external system. For example, it uses the Google Calendar API to get the schedule information for the next week.

[1613] Step 2:

[1614] The server analyzes the schedule information it obtains and detects consecutive meetings and tasks.

[1615] Input: Dataset of acquired schedule information

[1616] Output: Analysis results (identification of consecutive meetings and tasks)

[1617] Specific operation: The server sorts the schedule information in chronological order and executes logic to check whether consecutive meetings exceed a certain time (e.g., 3 hours).

[1618] Step 3:

[1619] The server determines the need for social activities and suggests appropriate social activities.

[1620] Input: Analysis results (identification of consecutive meetings and tasks)

[1621] Output: A list of suggested suitable social activities

[1622] Specific operation: When consecutive meetings are detected, the server generates a list of suggested social activities for the user, such as "two minutes of stretching" or "five minutes of chatting."

[1623] Step 4:

[1624] The terminal displays the social activity suggestions received from the server to the user.

[1625] Input: A list of suggested social activities sent by the server

[1626] Output: A proposed interface for the user to see.

[1627] Specific operation: The terminal displays the suggested interaction activities on the user interface, allowing the user to confirm the suggestions.

[1628] Step 5:

[1629] The user selects a suggested social activity on the device.

[1630] Input: A proposed interface for user display

[1631] Output: Selected interaction activities

[1632] Specific operation: The user uses the device interface to select the desired interaction activity from the suggested activities.

[1633] Step 6:

[1634] The server adds the user's selected social activity to the schedule.

[1635] Input: Selected interaction activity

[1636] Output: Updated schedule

[1637] Specific operation: The server schedules the interaction activity selected by the user and adds it as an appointment.

[1638] Step 7:

[1639] The terminal collects audio data during the meeting in real time and sends it to the server.

[1640] Input: Real-time audio in a meeting

[1641] Output: Collected audio data (digital format)

[1642] Specific operation: The device uses a voice input device to record audio during the meeting and save it in digital format. This data is then streamed to the server.

[1643] Step 8:

[1644] The server converts the received voice data into text data.

[1645] Input: Collected audio data

[1646] Output: Text data converted from audio

[1647] Specific operation: The server uses voice recognition technology (e.g., Google Cloud Speech-to-Text API) to convert the voice data into text data.

[1648] Step 9:

[1649] The server extracts the main points from the converted text data.

[1650] Input: Text data converted from speech

[1651] Output: Extracted key information

[1652] How it works: The server uses natural language processing techniques to extract important information from the text data, such as what needs to be reported at the next meeting.

[1653] Step 10:

[1654] The server adds new tasks and reminders to the user's calendar based on the extracted key points.

[1655] Input: Extracted gist information

[1656] Output: Updated calendar information (with new tasks and reminders added)

[1657] What happens: The server adds upcoming meeting arrangements and other necessary tasks to the user's calendar.

[1658] Step 11:

[1659] The server sends new schedule details and notifications to the user terminal and notifies them.

[1660] Input: Updated calendar information

[1661] Output: Notification to user terminal

[1662] Specific operation: The server notifies the user by sending the updated schedule to the user's device via push notification or other notification method.

[1663] Step 12:

[1664] The user checks the notification on the device and approves or adjusts it if necessary.

[1665] Input: Notification to user terminal

[1666] Output: User approved and adjusted schedule

[1667] What happens: The user sees the notifications on their device, acknowledges the new task or reminder, adjusts their schedule if necessary, and confirms the changes.

[1668] (Application example 1)

[1669] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1670] Robots and machines working in factories are often subjected to continuous, long-term operation, which can result in problems such as overheating and abnormal vibrations. It can also be difficult for workers to detect abnormalities in real time and take appropriate action. This can lead to reduced production efficiency and increased risk of machine breakdowns. Furthermore, workers often do not take appropriate breaks or undergo maintenance during long periods of work, which also reduces the operating efficiency and lifespan of machines. The present invention aims to solve these problems.

[1671] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1672] In this invention, the server includes means for acquiring user schedule information, means for analyzing the acquired schedule information and determining the need for an ice breaker, means for suggesting appropriate ice breaker activities, means for adding the suggested ice breaker activities to the user's schedule, means for managing the work schedule of the industrial machinery and suggesting breaks between continuous work, and means for collecting work data of the industrial machinery in real time and detecting abnormalities. This reduces the burden on machinery in a factory due to long hours of operation and makes it possible to quickly detect and respond to abnormalities.

[1673] A "user" is someone who uses this system to improve schedule management and work efficiency.

[1674] "Schedule information" is data relating to plans or plans that are input or obtained by the user.

[1675] An "icebreaker" is an activity such as a short break or light exercise that takes place during long periods of work or meetings.

[1676] "Industrial machinery" refers to machinery and equipment used in industrial sites such as factories.

[1677] A "work schedule" is a schedule or timetable for work performed by industrial machines and workers.

[1678] A "break" is a temporary pause between successive tasks.

[1679] "Audio data" refers to digital data of conversations or audio recordings.

[1680] "Text data" refers to digital data converted into character information.

[1681] A "gist" is a particularly important part of a conversation or piece of information.

[1682] "Abnormal" means any operation or condition of a machine that deviates from normal operating conditions.

[1683] "Real-time" refers to data collection and processing occurring immediately, without delay.

[1684] "Notification" means informing the user of new information or alerts.

[1685] A "smart device" is a mobile device such as a smartphone or tablet that can connect to the Internet and run applications.

[1686] System Configuration

[1687] This invention is a business efficiency improvement system that integrates the acquisition and analysis of user schedule information, ice-breaker suggestions, automatic text conversion and key point extraction of conversation content, schedule creation and notification. This system is mainly composed of three elements: a server, a terminal, and a user.

[1688] Server Operation

[1689] The server operates using the following methods:

[1690] 1. Acquisition and analysis of schedule information

[1691] The server obtains the user's schedule information from an external scheduling system (e.g., Google Calendar, Microsoft Outlook API). It analyzes the obtained schedule information and determines whether an icebreaker is necessary. For example, if a long meeting or continuous work on industrial machinery is scheduled, it determines that an icebreaker is necessary in between.

[1692] 2. Propose and confirm icebreakers

[1693] The server generates a list of ice-breaking activities (e.g., two minutes of stretching, five minutes of maintenance check) based on the analysis results and suggests them to the user's device. The ice-breaking activities selected by the user are added to the schedule.

[1694] 3. Conversation text conversion and key points extraction

[1695] The server receives the audio data from the meeting and converts it into text using speech recognition software such as the Google Speech-to-Text API. The server then extracts key points from the converted text.

[1696] 4. Schedule Task Reminders

[1697] Based on the extracted key points, new tasks and reminders are added to the user's calendar using data analysis techniques using Python and the pandas library.

[1698] 5. Industrial Machinery Data Collection and Anomaly Detection

[1699] The server collects operational data (e.g., temperature, vibration, and operating time) from industrial machinery in real time and detects abnormalities. This data collection and analysis is done using Python and the pandas library. If an abnormality is detected, an alert is sent to the worker's smart device.

[1700] Device behavior

[1701] The terminal operates using the following means:

[1702] 1. Receiving and displaying schedule information

[1703] The terminal displays the schedule information received from the server to the user, who can then check and select the icebreaker activities suggested through the terminal.

[1704] 2. Collection of audio data

[1705] The terminal collects audio data during the meeting in real time and sends it to a server. The audio data is also recorded in digital format and can be viewed later.

[1706] 3. Display of notifications

[1707] The device displays notifications of new schedules and reminders received from the server to the user, who can then review, approve, and adjust the schedule.

[1708] User operations

[1709] The user interacts with the device using the following means:

[1710] 1. Check and select your schedule

[1711] The user checks the schedule through their device and selects an icebreaker activity suggested by the server. The selected activity is automatically added to the schedule.

[1712] 2. Record and review conversations

[1713] When starting a meeting, users can turn on the device's audio recording function to record the conversation. After the meeting, they can check the converted text data and extracted key points.

[1714] 3. Review and adjust notifications

[1715] Users can view notifications on their device, acknowledge new tasks and reminders, and adjust their schedules as needed.

[1716] Specific examples

[1717] The following are specific examples for carrying out the present invention:

[1718] Example 1: Long meetings and continuous work

[1719] For example, if a user has a three-hour continuous meeting or continuous work schedule for an industrial machine, the server analyzes the schedule and suggests suitable icebreakers (e.g., five-minute stretching or maintenance check) before, during, or after the meeting or work. The icebreaker activities selected by the user are incorporated into the schedule and notifications are set.

[1720] Example 2: Recording important information during meetings and detecting anomalies

[1721] A user starts a meeting and turns on the audio recording function on their device. The audio during the conversation is collected by the device and sent to the server. The server converts the audio into text and extracts key points. It determines that a "market analysis report" needs to be submitted before the next meeting, and this information is added to the calendar and notified to the user as a reminder. Furthermore, if an industrial machine detects abnormal vibrations, the server will issue a warning to the worker's smart device.

[1722] Prompt Sentence Examples

[1723] As a concrete example, consider the following prompt input to a generative AI model:

[1724] Enter the following data: a list of work schedules, activities, icebreaker suggestions, and work data (temperature, vibration, etc.). For example, "2023-10-10 09:00-12:00: Work A, 2023-10-10 12:00-13:00: Break."

[1725] The system suggests an icebreaker: "5 minutes of maintenance after 20 minutes of work." It analyzes data and issues an alert if it detects an abnormality.

[1726] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1727] Step 1:

[1728] The server retrieves the user's schedule information using the API of an external scheduling system (e.g., Google Calendar or Microsoft Outlook), which requires an API key and a calendar ID as input. The output is the retrieved schedule information.

[1729] Step 2:

[1730] The server analyzes the schedule information it has acquired and detects long meetings and work. The input is the schedule information acquired in step 1, and the data is analyzed to extract events with long meetings or continuous work. This analysis identifies locations where ice breakers are needed. The output is the analysis results, i.e., a list of events where ice breakers are needed.

[1731] Step 3:

[1732] The server proposes ice-breaking activities (e.g., 2 minutes of stretching, 5 minutes of maintenance check) based on the analysis results. The input is the list of events requiring ice-breaking obtained in step 2. The server assigns ice-breaking activities to these events and generates a list of activities to propose to the user. The output is a list of proposed ice-breaking activities.

[1733] Step 4:

[1734] The terminal displays the list of icebreaker activities received from the server to the user. The input is the list of icebreaker activities generated in step 3, and the terminal displays it on the user's interface so that the user can select one. The output is the icebreaker activity selected by the user.

[1735] Step 5:

[1736] The server adds the icebreaker activity selected by the user to the schedule and notifies the user. The input is the icebreaker activity selected by the user in step 4, the server incorporates it into the schedule, and generates a new schedule. The output is the updated schedule information.

[1737] Step 6:

[1738] The terminal collects audio data during the meeting in real time and sends it to the server. The input is the audio data recorded during the meeting, which the terminal records in digital format and sends to the server. The output is the audio data sent to the server.

[1739] Step 7:

[1740] The server converts the audio data into text data and extracts the main points. The input is the audio data sent in step 6, and the server converts the audio data into text data using the Google Speech-to-Text API. It then performs natural language processing to extract the main points from the text. The output is a list of main points.

[1741] Step 8:

[1742] The server adds new tasks and reminders to the user's schedule based on the extracted key points. The input is the list of key points generated in step 7, and the server automatically adds new tasks and reminders to the schedule based on this. The output is the updated schedule information.

[1743] Step 9:

[1744] The server collects operational data from industrial machinery in real time and detects abnormalities. The input is the operational data of the industrial machinery (such as temperature, vibration, operating time, etc.), which the server monitors in real time and analyzes abnormalities. The output is an alert when an abnormality is detected.

[1745] Step 10:

[1746] When the server detects an abnormality, it issues a warning to the worker's smart device. The input is the abnormal data detected in step 9, and a warning message is generated based on that content and notified to the worker's smart device. The output is the notification sent to the worker.

[1747] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1748] System configuration

[1749] This invention is a business efficiency improvement system that integrates the acquisition and analysis of user schedule information, ice-breaker suggestions, automatic text conversion and key point extraction of conversation content, schedule creation and notifications, and also incorporates an emotion engine. The roles and specific operations of the server, terminal, and user are explained below.

[1750] Server Operation

[1751] 1. Acquisition and analysis of schedule information

[1752] The server retrieves the user's schedule information from an external scheduling system (e.g., Google Calendar, Microsoft Outlook). It analyzes the retrieved schedule information and determines whether an icebreaker is necessary. For example, if there are consecutive long meetings, it determines that an icebreaker is necessary.

[1753] 2. Propose and confirm icebreakers

[1754] Based on the analysis results, the server generates a list of ice-breaking activities (e.g., two minutes of stretching, five minutes of chatting) and suggests them to the user's device. The ice-breaking activities selected by the user are added to the schedule. Furthermore, the selection of ice-breaking activities is optimized based on the analysis results of the emotion engine.

[1755] 3. Conversation text conversion and key points extraction

[1756] The server receives the voice data from the meeting and converts it into text using speech recognition technology. Key points (e.g., matters that need to be reported at the next meeting) are extracted from the converted text data. The emotion engine also recognizes the user's emotions in real time during the conversation and adjusts the extraction of key points and the addition of tasks according to their emotional state.

[1757] 4. Schedule Task Reminders

[1758] Based on the extracted key points, new tasks and reminders are added to the user's calendar. For example, an item such as "Prepare for the next meeting" is automatically added. The emotion engine analyzes the user's stress and fatigue levels and dynamically adjusts the content and timing of reminders based on the results.

[1759] 5. Notification

[1760] The server sends new schedule information and reminders to the user's device, notifying the user. The user checks the notification and adjusts the schedule as necessary.

[1761] Device behavior

[1762] 1. Receiving and displaying schedule information

[1763] The terminal displays the schedule information received from the server to the user, who can then check and select the icebreaker activities suggested through the terminal.

[1764] 2. Collection of audio data

[1765] The device collects audio data during the meeting in real time and sends it to a server. The audio data is also recorded in digital format and can be viewed later.

[1766] 3. Use of Emotion Engine

[1767] The device analyzes the user's facial expressions and voice tone and uses an emotion engine to analyze the user's emotional state in real time. The analysis results of the emotion engine are sent to a server and used to suggest icebreakers, extract key points, and add tasks.

[1768] 4. Display of notifications

[1769] The device displays notifications of new schedules and reminders received from the server to the user, who can then review, approve, and adjust the schedule.

[1770] User operations

[1771] 1. Check and select your schedule

[1772] The user checks the schedule on their device and selects an icebreaker activity suggested by the server. The selected activity is automatically added to the schedule.

[1773] 2. Record and review conversations

[1774] When a user starts a meeting, they can turn on the device's audio recording function to record the conversation. After the meeting, they can check the converted text data and extracted key points.

[1775] 3. Review and adjust notifications

[1776] Users can view notifications on their device, acknowledge new tasks and reminders, and adjust their schedules as needed.

[1777] Specific examples

[1778] Example 1: Long meetings

[1779] A user enters a week's schedule, including a three-hour series of meetings.

[1780] The server analyzes this and suggests appropriate icebreakers before, during, or after the meeting. The emotion engine analyzes the user's fatigue level and recommends icebreakers to relieve stress.

[1781] The user selects "5 minutes of stretching" and the server schedules it and sets up a notification.

[1782] Example 2: Recording important information during a meeting

[1783] A user starts a conference and turns on the audio recording function of the terminal. The audio during the conversation is collected by the terminal and sent to the server.

[1784] The server converts the speech into text and extracts key points, while an emotion engine analyzes the user's emotions and adjusts the key points taking into account the importance and urgency of the topic.

[1785] It is determined that "Submission of market analysis report" is required before the next meeting, and this information is added to the calendar and notified to the user as a reminder.

[1786] As described above, the present invention improves the efficiency of user schedule management and the recording and analysis of conversation content, and by combining it with an emotion engine, it provides optimal business support based on the user's emotional state.

[1787] The processing flow will be explained below.

[1788] Step 1:

[1789] The user accesses the tool and starts the schedule management service.

[1790] Step 2:

[1791] The device retrieves the user's calendar data from an external scheduling service such as Google Calendar or Microsoft Outlook, and then sends this information to the server.

[1792] Step 3:

[1793] The server analyzes the received schedule data and identifies when an icebreaker is needed, such as during a long meeting or when consecutive tasks are required.

[1794] Step 4:

[1795] The server determines the need for an icebreaker based on the analysis results, and if necessary, selects an appropriate icebreaker activity from the database.

[1796] Step 5:

[1797] The server generates a proposal for the selected icebreaker activity and transmits it to the terminal.

[1798] Step 6:

[1799] The device displays icebreaker suggestions to the user, who can then select any icebreaker activity from the suggestions.

[1800] Step 7:

[1801] The device sends the icebreaker activity selected by the user to the server, which adds it to the user's schedule.

[1802] Step 8:

[1803] When a user starts a conference, the device turns on the audio recording function, and the device collects the audio data of the conversation in real time and transmits it to the server.

[1804] Step 9:

[1805] The server converts the voice data into text data using voice recognition technology.

[1806] Step 10:

[1807] The server analyzes the converted text data and extracts important points, for example, identifying tasks such as "items that need to be reported at the next meeting."

[1808] Step 11:

[1809] The server adds new tasks and reminders to the user's calendar based on the extracted key points.

[1810] Step 12:

[1811] The server sends new schedules and reminders to the device and creates notifications, which the device displays to the user.

[1812] Step 13:

[1813] The user checks the notification and adjusts the schedule as necessary.

[1814] Example 2

[1815] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1816] Conventional schedule management systems do not provide effective ways to reduce fatigue and stress caused by continuous meetings and tasks. Furthermore, the efficient recording of important comments and decisions during meetings and the manual task management based on these records result in reduced work efficiency. Furthermore, the lack of timely reminders and notifications that take into account the user's emotional state makes it difficult for users to maintain their productivity.

[1817] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for acquiring user schedule information, means for analyzing the acquired schedule information and determining the need for an icebreaker, means for proposing appropriate icebreaker activities, means for adding the proposed icebreaker activities to the user's schedule, means for analyzing the user's emotional state and optimizing the icebreaker activities, means for collecting voice data of the conversation, means for converting the collected voice data into text data, means for extracting key points from the converted text data, means for adding new tasks or reminders to the user's schedule based on the extracted key points, and means for sending and notifying the new schedule or reminder to the user terminal. This effectively reduces fatigue and stress caused by continuous meetings and tasks for the user, efficiently records important comments and decisions made during meetings, automates task management based on the records, and enables appropriate timing of reminders and notifications taking the user's emotional state into consideration.

[1818] "Schedule information" refers to data relating to the user's planned dates and activities.

[1819] "Acquisition means" refers to the methods or functions for collecting data or information from external or internal systems.

[1820] "Analysis means" refers to the methods and functions used to analyze acquired data and information and understand its content and meaning.

[1821] "Ice-breaking activities" are refreshing activities or short recreational activities to relieve fatigue and stress caused by long meetings or task completion.

[1822] "Suggestion means" refers to methods or functions for presenting appropriate information or activities to users.

[1823] "Additional means" refers to methods or functions for inserting new information or activities into a user's schedule.

[1824] "Emotional state" refers to the mental state or mood of the user.

[1825] "Audio data" refers to data that has been recorded in digital form as a conversation or voice signal.

[1826] "Text data" refers to data obtained by converting voice data into character information.

[1827] "Key point extraction means" refers to a method or function for extracting important information or key content from text data.

[1828] A "task" is a specific task or job that a user must undertake.

[1829] A "reminder" is a notification or alarm that reminds a user of a specific date, time, or event.

[1830] "Notification means" refers to the method or function for notifying the user of new information or reminders.

[1831] A "terminal" is a computing device that is directly operated by a user.

[1832] A "server" is a central computer system that processes and manages data on a network.

[1833] This is a business efficiency improvement system that integrates the acquisition and analysis of user schedule information, ice-breaker suggestions, automatic text conversion and key point extraction of conversation content, and task and reminder scheduling and notification. Furthermore, it incorporates an emotion engine to provide optimal support based on the user's emotional state.

[1834] System Configuration

[1835] Server Operation

[1836] The server first obtains the user's schedule information from an external scheduling system (e.g., Google Calendar or Microsoft Outlook) via API. The obtained schedule information is stored in an internal database, and an analysis engine uses this information to determine whether an icebreaker is necessary.

[1837] Next, based on the analysis results, the server lists multiple ice-breaking activities (e.g., two minutes of stretching, five minutes of chatting) and sends this list in JSON format to the user's device. At the same time, the emotion engine analyzes the user's stress level and fatigue level to select the most appropriate ice-breaking activity. The server adds the ice-breaking activity selected by the user to the schedule and notifies the user's device again.

[1838] During the meeting, the server receives real-time voice data from the devices and converts it into text data using the Google Speech-to-Text API. The converted text data is stored in a database, and a natural language processing (NLP) engine extracts key points. An emotion engine analyzes the user's emotions and adjusts the importance of the key points.

[1839] Furthermore, new tasks and reminders are added to the user's calendar based on the extracted key points. The emotion engine dynamically adjusts the content and timing of reminders, taking into account the user's stress level and fatigue level. The server then sends the new schedule and reminders to the user's device and notifies the user.

[1840] Device behavior

[1841] The device displays the schedule information received from the server to the user via a GUI. The user can then review and select suggested icebreaker activities. During the meeting, the device's microphone collects audio data in real time, digitizes it, and sends it to the server. This audio data is also stored locally and can be played back later.

[1842] The device uses its built-in camera and microphone to analyze the user's facial expressions and voice tone and input them into the emotion engine. The analysis results are sent to the server and reflected in icebreaker suggestions and key point extraction. In addition, the device receives new schedule and reminder notifications from the server and displays them as push notifications, allowing the user to confirm, approve, or adjust them.

[1843] User operations

[1844] The user checks the schedule on their device and selects an icebreaker activity suggested by the server. The selected activity is automatically added to the schedule. When starting a meeting, the user turns on the device's audio recording function to record the conversation. After the meeting, the converted text data and extracted key points can be viewed. The user can also check notifications displayed on the device, approve new tasks and reminders, and adjust the schedule as necessary.

[1845] Specific examples

[1846] Example 1: Long meetings

[1847] A user enters a week's worth of events into Google Calendar, and one day has three consecutive meetings scheduled.

[1848] The server analyzes this and suggests appropriate icebreakers before, during, or after the meeting. The emotion engine analyzes the user's fatigue level and recommends "5 minutes of stretching" to relieve stress.

[1849] The user selects "5 minutes of stretching" and the server schedules it and sets up a notification.

[1850] Example 2: Recording important information during a meeting

[1851] A user starts a conference and turns on the audio recording function of the device. The audio during the conversation is collected by the device and transmitted to the server in real time.

[1852] The server converts the speech into text and extracts key points. The emotion engine analyzes the user's emotions and adjusts the key points based on the importance and urgency of the topic.

[1853] It turns out that "Submission of Market Analysis Report" is required before the next meeting, and this information is added to the calendar and notified to the user as a reminder.

[1854] Prompt Sentence Examples

[1855] "Use your schedule and sentiment data to suggest the best icebreaker activities, extract key points from meetings, and add upcoming tasks to your schedule."

[1856] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1857] Step 1: Get schedule information

[1858] The server gets information from an external scheduling system

[1859] Input: User authentication information, API key

[1860] Output: User schedule information (JSON format)

[1861] How it works: The server uses the APIs of Google Calendar and Microsoft Outlook to retrieve user schedule information. This information is sent to the server in JSON format and stored in an internal database.

[1862] Step 2: Analyze schedule information

[1863] The server analyzes the schedule information and determines whether an icebreaker is necessary.

[1864] Input: Schedule information (JSON format)

[1865] Output: Analysis results (flag indicating need for icebreaker)

[1866] How it works: The server's analysis engine analyzes schedule information and determines whether an icebreaker is necessary if there are consecutive "meetings" or "long-term tasks." For example, if the interval between consecutive meetings exceeds a certain time, it determines that an icebreaker is necessary.

[1867] Step 3: Suggest an icebreaker activity

[1868] Your server will suggest appropriate icebreaker activities

[1869] Input: Analysis results, emotion engine data

[1870] Output: Icebreaker proposal list (JSON format)

[1871] How it works: The server suggests icebreaker activities suitable for the user based on the analysis results and emotion engine data. The list of suggestions is sent to the device in JSON format.

[1872] Step 4: Determine icebreaker activities

[1873] The server adds icebreaker activities to the schedule based on the user's selection.

[1874] Input: User's choice

[1875] Output: Updated schedule information

[1876] How it works: After a user selects an icebreaker activity on the device, the server adds the selection to the schedule and sends the updated schedule information back to the device.

[1877] Step 5: Collecting audio data from the conversation

[1878] The device collects audio data during the meeting and sends it to the server.

[1879] Input: Audio during the meeting

[1880] Output: Digital audio data

[1881] How it works: The device's microphone collects audio during the meeting in real time and sends it digitally to a server, where the audio data is stored locally.

[1882] Step 6: Convert audio data to text

[1883] The server converts the audio data into text data.

[1884] Input: Audio data

[1885] Output: Text data

[1886] How it works: The server uses the Google Speech-to-Text API to convert the received audio data into text data, which is then stored in a database.

[1887] Step 7: Extracting key points

[1888] The server extracts key points from the text data

[1889] Input: Text data

[1890] Output: Gist list

[1891] How it works: The server's natural language processing (NLP) engine analyzes the text data and extracts key conversation points and action items. The emotion engine also analyzes the user's emotions at this stage and adjusts the importance of key points.

[1892] Step 8: Schedule task reminders

[1893] The server adds new tasks and reminders to your schedule

[1894] Input: Key points list, emotion engine data

[1895] Output: Updated schedule

[1896] How it works: Based on the extracted key points, the server adds new tasks and reminders to the user's calendar. The emotion engine dynamically adjusts the timing of reminders based on the user's stress and fatigue levels.

[1897] Step 9: Notification

[1898] The server sends new schedules and reminders to the user's device and notifies them.

[1899] Input: Updated Schedule

[1900] Output: Notification message

[1901] How it works: The server sends updated schedule information and reminders to the device, which are displayed to the user as push notifications. The user can view the notifications and accept or adjust them as needed.

[1902] (Application example 2)

[1903] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1904] In modern factories and offices, workers often find it difficult to take appropriate breaks or refresh themselves when working long hours continuously or in the middle of important tasks. Furthermore, systems for accurately recording the work performed by workers and later reviewing and utilizing this information are often inadequate. Furthermore, to reduce workers' stress and fatigue, it is necessary to provide optimal intervals based on their emotional state. These challenges can reduce work efficiency and ultimately hinder improvements in productivity and work quality. A solution to these problems and achieve both work efficiency and worker health is needed.

[1905] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1906] In this invention, the server includes means for acquiring a user's schedule information, means for analyzing the acquired schedule information and determining the need for a break, means for suggesting appropriate break activities, means for adding the suggested break activities to the user's schedule, means for analyzing the user's emotional state using an emotion engine and optimizing the selection of break activities, means for collecting voice data during work and converting it into text data, means for extracting important work points from the converted text data, means for adding new tasks and reminders to the user's schedule based on the extracted points, and means for displaying notifications to the user and allowing the user to confirm, approve, and adjust the suggested break activities and added tasks and reminders. This enables workers to take breaks at appropriate times, maximize work efficiency, and accurately record and manage important work content and key points.

[1907] "User" refers to the workers and employees who use the system.

[1908] "Setup information" refers to information that indicates the work schedules, plans, and tasks of workers or employees.

[1909] "Emotion engine" refers to technology that analyzes a user's facial expressions and vocal tone to recognize and analyze their emotional state (e.g., stress or fatigue) in real time.

[1910] "Rest activities" refer to short breaks or refreshing activities (e.g., short stretching or chatting) that users take between tasks to reduce fatigue and stress.

[1911] "Voice data" refers to digital data of a user's voice collected while working or talking.

[1912] "Text data" refers to sentence data obtained by converting voice data into characters.

[1913] "Key points" are pieces of information or tasks that are considered particularly important in a work progress or conversation.

[1914] A "task" refers to a discrete task or activity that a user must perform.

[1915] A "reminder" is a notification or reminder that helps users remember specific tasks or appointments.

[1916] This system acquires setup information for workers and employees, proposes and schedules optimal break activities to improve work efficiency, and automatically records and manages important work content and key points of conversations. The roles and specific operations of the server, terminal, and user are explained below.

[1917] Server Operation

[1918] Acquisition and analysis of setup information

[1919] The server acquires setup information for workers and employees from the factory management system or business management system (MES or ERP system). It analyzes the acquired setup information and determines whether breaks are necessary. For example, if a long period of continuous work is scheduled, it determines that a break is necessary in between.

[1920] Proposing and confirming break activities

[1921] Based on the analysis results, the server generates a list of break activities (e.g., short breaks and stretching) and suggests them to the user's device. It uses an emotion engine to analyze the user's emotional state and selects the optimal break activity based on the results. The break activity selected by the user is added to the schedule.

[1922] Speech data text conversion and gist extraction

[1923] The server receives the working voice data and converts it into text using speech recognition technology (e.g., the speech_recognition library). It then uses a generative AI model (e.g., GPT-3.5) to extract key points from the converted text. The extraction of key points is optimized according to the analysis results of the emotion engine.

[1924] Schedule tasks and reminders

[1925] The server adds new tasks and reminders to the user's schedule based on the extracted key points. For example, information such as "Parts for the next process are missing" is added to the schedule and notified to the user as a reminder.

[1926] notification

[1927] The server sends new schedule information and reminders to the user's device, and notifies the user. The user can check the notification and adjust the schedule as necessary.

[1928] Device behavior

[1929] Receiving and displaying setup information

[1930] The terminal displays the schedule information received from the server to the user, who can then check and select the suggested break activities through the terminal.

[1931] Audio data collection

[1932] The device collects voice data in real time while the worker is working and sends it to a server, where it is recorded in digital format and can be viewed later.

[1933] Use of emotion engine

[1934] The device analyzes the user's facial expressions and voice tone and uses an emotion engine to analyze the user's emotional state in real time. The analysis results of the emotion engine are sent to a server and used to suggest break activities, extract key points, and add tasks.

[1935] Viewing notifications

[1936] The device displays notifications of new schedules and reminders received from the server to the user, who can then review, approve, and adjust the schedule.

[1937] User operations

[1938] Check and select setup

[1939] The user checks the schedule through the terminal and selects the break activity suggested by the server, which is then automatically added to the schedule.

[1940] Recording and checking work details

[1941] When a user starts a task, they can turn on the device's voice recording function to record the task's progress. After the task is completed, they can check the converted text data and extracted key points.

[1942] Review and adjust notifications

[1943] Users can view notifications on their device, acknowledge new tasks and reminders, and adjust their schedules as needed.

[1944] Specific examples

[1945] Example 1: Long working hours

[1946] The user enters their weekly schedule, and one day is scheduled for six hours of continuous robot operation.

[1947] The server analyzes this and suggests appropriate rest activities before, during, or after the operation. The emotion engine analyzes the user's fatigue level and recommends short breaks to relieve stress.

[1948] The user selects "5 minutes of stretching" and the server schedules it and sets up a notification.

[1949] Example 2: Recording important information during work

[1950] The user starts working and turns on the device's voice recording function. The voice recorded during the work is collected by the device and sent to the server.

[1951] The server converts the speech into text and extracts key points, while the emotion engine analyzes the user's emotions and optimizes the key points in real time.

[1952] If a "parts shortage" is discovered by the time the next process is completed, this information is added to the schedule and a reminder is sent to the user.

[1953] Example of a generative AI model prompt:

[1954] Prompt Sentence Examples

[1955] Text from audio recording: "We've encountered a parts supply issue. We are running low on parts needed for the next operation. We need to reconsider our parts supply plan."

[1956] Gist Extraction Generation AI Prompt: "Please extract the key points from the audio recording above."

[1957] Key point: "Parts supply problem. Parts for the next process are in short supply. Parts supply plan needs to be reconsidered."

[1958] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1959] Step 1:

[1960] The server obtains setup information for workers and employees from the factory management system or business management system (MES or ERP system). This setup information includes work schedules, plans, and tasks. It analyzes this information and evaluates the length and continuity of work time. It receives the worker's setup information (schedule, plan, tasks) as input and determines the need for breaks as a result of the analysis.

[1961] Step 2:

[1962] The server generates a list of rest activities based on the analysis results. It uses the emotion engine to analyze the user's emotional state and selects the optimal rest activity taking the results into consideration. For example, if the user has been working for a long time or is feeling emotional stress, it suggests stretching or a short break. The server uses the analysis results and emotion engine data as input and generates a list of rest activities as output.

[1963] Step 3:

[1964] The terminal receives the list of break activities sent from the server and displays it to the user. The user checks the proposed break activities through the terminal and selects one. The terminal receives the list of break activities as input and sends the selected break activity to the server as output.

[1965] Step 4:

[1966] The server adds the break activity selected by the user to the setup information and updates the schedule. This updated schedule information is sent to the user's terminal and notified. The server receives the break activity selected by the user as input, generates updated schedule information as output, and sends it to the terminal.

[1967] Step 5:

[1968] When a user starts working, the terminal turns on the voice recording function, collects voice data during work, and sends this data to the server. The terminal collects the user's voice data as input and sends the voice data to the server as output.

[1969] Step 6:

[1970] The server converts the received voice data into text data using speech recognition technology (e.g., the speech_recognition library), and extracts key points from the converted text data using a generative AI model (e.g., GPT-3.5). It receives voice data as input and generates text data and key points as output.

[1971] Step 7:

[1972] The server adds new tasks and reminders to the schedule information based on the extracted key points. This information is sent to the user's terminal and displayed as a notification. As input, the server generates new tasks and reminders based on the extracted key points, and as output, adds these to the schedule and sends them to the terminal.

[1973] Step 8:

[1974] The user can check the notifications displayed on the device, acknowledge new tasks and reminders, and adjust the schedule as needed. The input is the notification displayed by the device, and the output is the necessary adjustment.

[1975] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1976] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1977] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1978] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1979] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1980] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1981] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1982] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1983] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1984] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1985] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1986] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1987] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosu...

Claims

1. A means for obtaining schedule information of a user; A means of analyzing the acquired schedule information and determining the need for icebreakers; How to suggest suitable icebreaker activities and The system includes a means for adding suggested icebreaker activities to a user's schedule.

2. a means for collecting audio data of the conversation; A means for converting the collected voice data into text data; means for extracting gist from the converted text data; The system of claim 1 further comprising means for adding new tasks or reminders to the user's schedule based on the extracted key points.

3. The system of claim 1 , further comprising means for displaying notifications to a user to review, approve, and adjust suggested icebreaker activities and added task reminders.

Images