System

Abstract

A system is provided.SOLUTION: A system for improving the efficiency of work on vacation, comprising: means for automatically responding to a telephone call on vacation; means for converting the content of a received telephone call into text and analyzing the text; and means for determining urgency from the analyzed content of the telephone call and notifying an emergency contact address as necessary.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] When employees take vacation, they may need to respond to important phone calls and emails during their vacation. Furthermore, after their vacation, it takes a lot of time to review accumulated emails and meeting details and return to their normal work. This puts a significant burden on employees, who must communicate with them during their vacation and check information after their vacation. There is a need for a system that can solve these problems, allowing employees to enjoy their vacation with peace of mind and return to their normal work efficiently after their vacation. [Means for solving the problem]

[0005] The present invention provides the following means as a system for improving the efficiency of work during holidays.

[0006] A means of automatically answering calls during vacation time;

[0007] A means of converting received phone calls into text and analyzing them;

[0008] A method to determine the urgency of the call based on the analyzed content and notify emergency contacts as necessary.

[0009] This system, including the above, will reduce the burden of answering phone calls during holidays.

[0010] Furthermore, a means for collecting new emails and meeting contents from the user's mailbox and meeting recording data;

[0011] means for generating a summary of the collected data;

[0012] A means of prioritizing from the summary;

[0013] The system, which includes this, will make the process of checking information after holidays more efficient.

[0014] Also, a means to generate a summary report at the end of the vacation period, including a summary of emails and meetings;

[0015] means for transmitting the generated summary report to a user;

[0016] The system also includes the following to help employees return to normal work efficiently after holidays.

[0017] This allows employees to feel safe during their leave and also helps to streamline work after they return.

[0018] An "automatic vacation call answering means" is a device or program that automatically plays a pre-defined answering message to calls received by an employee while on vacation.

[0019] "Means for converting received telephone calls into text and analyzing them" refers to a device or program that converts the voice data of received telephone calls into text data using voice recognition technology, and then analyzes the text data using natural language processing (NLP) technology.

[0020] "Means for determining the urgency of a call based on the analyzed content of the call and notifying an emergency contact if necessary" refers to a device or program that determines the urgency of a call based on the text-converted and analyzed content of the call and, if necessary, notifies a pre-set emergency contact.

[0021] "Means for collecting new emails and meeting contents from a user's mailbox and meeting recording data" refers to a device or program that periodically accesses a mail server and a meeting recording database to obtain newly received emails and meeting recording data.

[0022] The "means for generating a summary of collected data" refers to a device or program that automatically summarizes collected new emails and recorded meeting data using natural language processing technology to summarize the main points.

[0023] The "means for assigning priorities from summaries" is a device or program that assigns priorities based on the generated summary data in accordance with the importance and urgency of the data, and determines the priority of the tasks.

[0024] The "means for generating a summary report including email and meeting summaries to coincide with the end of a vacation" refers to a device or program that compiles summaries generated from collected new emails and recorded meeting data into a single report to coincide with the end of a vacation.

[0025] The "means for transmitting the generated summary report to the user" is a device or program that transmits the generated summary report to the user's terminal and notifies the user. [Brief explanation of the drawings]

[0026] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0027] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0028] First, the terms used in the following description will be explained.

[0029] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0030] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0031] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0032] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0033] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0034] [First embodiment]

[0035] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0036] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0037] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0038] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0039] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0040] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0041] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0042] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0043] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0044] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0045] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0046] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0047] The system of this invention consists of three main components: a server, a terminal, and a user. Each component operates as follows:

[0048] server

[0049] The server plays a central role in the system, hosting the AI ​​model and processing data in real time. The server has the following specific functions:

[0050] 1. Automated telephone answering

[0051] The server has the function of automatically detecting incoming calls when the user is on vacation and playing a pre-defined answering message, so that the user does not have to answer the phone directly while on vacation.

[0052] 2. Transcription and analysis of call content

[0053] The server converts the received call content into text data using voice recognition technology, and then analyzes the text data using natural language processing (NLP) technology to determine the urgency of the call content.

[0054] 3. Emergency notification

[0055] Based on the analysis results, the server will send notifications to pre-defined emergency contacts as necessary, ensuring that users only respond in truly urgent cases.

[0056] 4. Email and meeting summaries

[0057] The server collects new emails and meeting contents from the user's mailbox and meeting recording data, summarizes the collected data using a generation AI such as ChatGPT, and generates a summary report.

[0058] 5. Prioritize

[0059] The server evaluates the importance and urgency of the requests based on the generated summary data and assigns priorities to them, allowing users to return to work efficiently after their recovery.

[0060] 6. Generate and send a summary report

[0061] The server summarizes all collected data into a single summary report at the end of the vacation and sends it to the user's terminal.

[0062] Terminal

[0063] The terminal refers to the user's smartphone or computer, and acts as an interface with the server. The specific operations of the terminal are as follows:

[0064] 1. Display phone notifications

[0065] The device receives a notification from the server and displays a message to the user such as "AI has answered." If necessary, details of the call will also be displayed.

[0066] 2. View the summary report

[0067] The terminal receives the summary report sent from the server and notifies the user, which includes a summary and priority of emails and meetings received during the vacation.

[0068] User

[0069] The user is an employee who uses the system, and can be relieved from work during the vacation and return to work efficiently after the vacation. The specific operations and usage of the user are as follows:

[0070] 1. Vacation Settings

[0071] The user enters the start and end times of their vacation into the system, which prepares the server to activate the auto-answer mode.

[0072] 2. Checking during vacation

[0073] The user checks the notifications displayed on the device while on vacation. If there is an emergency, the user receives a detailed notification from the server and takes appropriate action.

[0074] 3. Resuming work after holidays

[0075] After the vacation, users can check a summary report on their device, which summarizes important emails and meeting details, allowing them to return to work smoothly based on their priorities.

[0076] Specific examples

[0077] For example, if a user sets a one-week vacation, the server will automatically answer all calls during that time, convert the calls into text, and analyze them. If the analysis determines that an emergency response is required, the server will immediately send a notification to the emergency contact.

[0078] At the same time, the server periodically checks the user's mailbox and conference recordings, generates summaries of new emails and conferences, and then prioritizes them into a summary report that is sent to the user's device at the end of the vacation.

[0079] When a user checks their device after a vacation, a summary report summarizing all the necessary information is displayed, allowing them to quickly start working on important tasks.

[0080] In this way, the system of the present invention allows users to maximize their vacation time while returning to work efficiently after their vacation.

[0081] The processing flow will be explained below.

[0082] Answering phone calls during holidays

[0083] Step 1:

[0084] The server receives the user's vacation setting information and enables the telephone auto-answer mode during the vacation period.

[0085] Step 2:

[0086] The server detects an incoming call while on vacation.

[0087] Step 3:

[0088] The server plays a pre-programmed answering message and converts the call into text using voice recognition technology.

[0089] Step 4:

[0090] The server analyzes the text of the call using natural language processing (NLP) technology to determine the urgency of the content.

[0091] Step 5:

[0092] The server will send notifications to pre-configured emergency contacts as needed.

[0093] Step 6:

[0094] The server generates a summary of the response content and sends it to the user's terminal.

[0095] Step 7:

[0096] The device will notify the user that "AI has responded" and, if necessary, display details of the response.

[0097] Vacation email and meeting summaries

[0098] Step 1:

[0099] The server periodically checks the user's mailbox and collects new emails.

[0100] Step 2:

[0101] The server checks the user's calendar and meeting recording data to collect the corresponding meeting details.

[0102] Step 3:

[0103] The server uses ChatGPT to analyze the contents of the collected new emails and generate summaries.

[0104] Step 4:

[0105] The server converts the meeting recording data into text using speech recognition technology and uses ChatGPT to generate a summary that extracts the key points.

[0106] Step 5:

[0107] The server determines the importance and urgency of emails and meetings based on the content of the generated summaries, and assigns priorities to them.

[0108] Step 6:

[0109] The server generates a priority list and sends it to the user's terminal.

[0110] Step 7:

[0111] The terminal receives the list and notifies the user.

[0112] Generate and view a summary report after the holidays

[0113] Step 1:

[0114] The server compiles all collected new emails and summaries generated from the meeting contents into one summary report at the end of the vacation.

[0115] Step 2:

[0116] The server transmits the generated summary report to the user's terminal.

[0117] Step 3:

[0118] The terminal notifies the user of receipt of the summary report.

[0119] Step 4:

[0120] The user checks the summary report on the terminal, which includes a summary and priority of all emails and meetings received during the vacation.

[0121] Step 5:

[0122] Based on the summary report, users can start responding to important emails and meeting details one by one.

[0123] Example 1

[0124] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0125] Employees are expected to be completely free from work during their vacation, while also being able to return to work quickly and efficiently after their vacation. However, traditional systems have limitations in ensuring that important communications or urgent responses are not overlooked during the vacation. Furthermore, dealing with the large amount of unprocessed emails and meeting details after the vacation takes time and effort, making it difficult to resume work smoothly.

[0126] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0127] In this invention, the server includes a means for automatically answering calls during vacation, a means for converting the contents of received calls into text data using voice recognition technology, and a means for analyzing the converted text data using natural language processing technology to determine the urgency of the call. This allows important communications and emergency responses to be made even during vacation, while allowing employees to return to work quickly and efficiently after their vacation.

[0128] "Means for automatically answering calls" refers to the ability of the system to answer calls on behalf of a vacation user and play a pre-defined voice message.

[0129] "Voice recognition technology" is a technology that analyzes voice data and converts it into text data.

[0130] "Text data" is character information converted from voice data using voice recognition technology.

[0131] "Natural language processing technology" is a technology that analyzes text data to understand its content, urgency, and intent.

[0132] The "means of determining urgency" is a function that uses natural language processing technology to analyze text data to identify important information and elements that require urgent action, and notify emergency contacts as necessary.

[0133] "Means for notifying emergency contacts" is a function that sends a notification to pre-set contacts when it is determined based on the analysis results that emergency contact is necessary.

[0134] A "mailbox" is a data storage device for receiving and storing a user's email.

[0135] "Meeting recording data" refers to data that records the audio information of a meeting.

[0136] A "generative AI model" is an artificial intelligence model that learns from large amounts of data to perform tasks such as text generation and summarization.

[0137] The "means of generating summaries" is a function that uses a generative AI model to extract key information from collected email and meeting recording data and provide it in a shortened format.

[0138] The "means for prioritizing" is a function that evaluates the importance and urgency of information based on the generated summary data and determines the order in which it should be processed.

[0139] A "Summary Report" is a report that lists key points and priorities generated from collected and summarized data.

[0140] A "terminal" is a device that allows a user to communicate with a server, and includes smartphones, computers, etc.

[0141] The "means for receiving notification" is a function by which the terminal receives information sent from the server and displays that information to the user.

[0142] This invention is based on a system that consists of three main components: a server, a terminal, and a user. Each component operates as follows:

[0143] server

[0144] The server plays a central role in this system, hosting the AI ​​model and processing data in real time. The server has the following specific functions:

[0145] 1. Automated telephone answering

[0146] The server automatically detects incoming calls from vacation users and plays pre-defined answering messages. Specifically, the server monitors the telephone line signal, and when a call comes in, it selects an appropriate voice file from templates stored in a database and plays it. For example, it could play a voice file that says, "I'm currently on vacation. I'll get back to you later."

[0147] 2. Transcription and analysis of call content

[0148] The server records the audio data of the call and converts it into text data using speech recognition technology, using technologies such as the Google Cloud Speech-to-Text API. The generated text data is then analyzed using natural language processing (NLP) technology to determine the urgency of the call content. The urgency is evaluated based on predefined rules (e.g., the presence of keywords such as "emergency" or "urgent"). For example, if the call content is something like "In the event of a fire," it is determined that an emergency response is required.

[0149] 3. Emergency notification

[0150] Based on the analysis, the server will send a notification to pre-defined emergency contacts if necessary. This notification can be sent in different ways, such as SMS, email, or push notification. For example, it may contain a message such as "A critical security issue has occurred. Your immediate attention is required."

[0151] 4. Email and meeting summaries

[0152] The server periodically checks the user's mailbox and meeting recording data. Specifically, it uses the Gmail API for the mailbox and speech recognition technology for the meeting recording. A generative AI model (e.g., ChatGPT) is used to generate a summary of the collected data. Prompts used include "Please summarize the following email" and "Please summarize the main points of this meeting." For example, a summary such as "The main topic of discussion was a progress report on a new project" can be generated.

[0153] 5. Prioritize

[0154] The server evaluates the importance and urgency of emails and meeting content based on the generated summary data, and assigns priorities to them. This prioritization is based on algorithms such as "urgency of time" and "job titles of the people involved." For example, "customer complaints" might be listed as the top priority.

[0155] 6. Generate and send a summary report

[0156] The server compiles all the summary data into a single summary report at the end of the vacation. This summary report is sent to the user's device via email or push notification. For example, it may contain a list of emails requiring replies under "Important matters during vacation," a summary of new emails, and key points from important meetings.

[0157] Terminal

[0158] The terminal refers to the user's smartphone or computer, and functions as an interface with the server. The specific operation of the terminal is as follows.

[0159] 1. Display phone notifications

[0160] The device receives notifications from the server in real time and displays them to the user. The notification content includes a message such as "AI has responded." For example, a notification may appear on the user's smartphone saying "An automatic response was made because the user is on vacation."

[0161] 2. View the summary report

[0162] The device receives the summary report sent from the server and notifies the user. The report contents can be checked from the notification center or a dedicated app, and important items are highlighted. For example, a notification may be displayed showing "two emails requiring attention" and "four important meetings."

[0163] User

[0164] The user is an employee who uses the system, and can be relieved from work during the vacation and return to work efficiently after the vacation. The specific operation and usage of the user are as follows.

[0165] 1. Vacation Settings

[0166] The user enters the system's settings screen and enters the start and end times of the vacation. The server receives this information and prepares the automatic response mode. For example, the user sets "Vacation start date: October 1, 2023" and "Vacation end date: October 7, 2023."

[0167] 2. Checking during vacation

[0168] While on vacation, users check the notifications displayed on their devices. Notifications with urgent nature will be displayed with special alerts, and users can respond accordingly. For example, if a message appears saying "You have an important notification," users can check the content and respond as necessary.

[0169] 3. Resuming work after holidays

[0170] After the vacation ends, the user checks the summary report on their device. The report summarizes important emails and meeting details, and the user can start addressing them based on priority. For example, the report shows "2 emails needing attention" and "4 important meetings," allowing the user to return to work immediately.

[0171] The system allows users to return to work quickly and efficiently without missing important communications while on vacation.

[0172] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0173] Server Processing Steps

[0174] Step 1: Auto-answer your phone

[0175] The server detects an incoming call. It receives the signal from the telephone line as input and plays a pre-defined answering message. For this, an audio file is selected from a database and played. For example, a voice message such as "I'm currently on vacation. I'll get back to you later" is played.

[0176] Step 2: Transcribe the call

[0177] The server records the audio data of the call. It receives the audio data as input and converts it into text data using speech recognition technology such as the Google Cloud Speech-to-Text API. The content of the call is saved in text format as output. For example, the content "There's a fire" is generated as text data.

[0178] Step 3: Analyzing the call

[0179] The server analyzes the generated text data using natural language processing (NLP) technology. It receives the text data as input and analyzes specific keywords and phrases to determine the urgency. The output is an assessment of the urgency level. For example, if the keyword "fire" is included, the urgency level is determined to be high.

[0180] Step 4: Send emergency notifications

[0181] Based on the analysis results, the server sends notifications to pre-configured emergency contacts as needed. It receives the urgency assessment result as input and sends notifications via SMS, email, etc. The output is a notification sent to the emergency contact. For example, a notification stating, "A critical security issue has occurred. Urgent action is required."

[0182] Step 5: Summarize emails and meetings

[0183] The server periodically checks the user's mailbox and meeting recording data. It receives email data and audio recording data as input and generates a summary using a generative AI model (e.g., ChatGPT). The output is a summary that extracts the key points. Prompts used include "Please summarize the following email" and "Please summarize the main points of this meeting." For example, a summary might be generated that reads, "The main topic was a progress report on a new project."

[0184] Step 6: Prioritize

[0185] The server evaluates the importance and urgency of the generated summary data and assigns priorities to them. It takes the summary data as input and runs a prioritization algorithm (e.g., time urgency, role of the involved parties). As output, a list of priorities is created. For example, "Customer complaints" may be listed as the highest priority.

[0186] Step 7: Generate and submit a summary report

[0187] The server compiles all the summary data into a single summary report when the vacation ends. It receives the generated summary data as input and formats it into a summary report. As output, it generates a summary report that is sent to the user's device at the end of the vacation. For example, it includes a list of emails that need replying as "important matters during vacation," a summary of new emails, and important meeting points.

[0188] Terminal processing steps

[0189] Step 1: View phone notifications

[0190] The device receives notifications from the server in real time and displays them to the user. Notifications from the server are received as input, and an application for displaying them on a smartphone or computer receives them. As output, a notification message is displayed to the user. For example, a notification saying "An automatic response has been sent because I am on vacation" is displayed.

[0191] Step 2: View the summary report

[0192] The device receives the summary report sent from the server and notifies the user. It receives the summary report data as input and displays it in a dedicated app or notification center. As output, the summary report is displayed to the user. For example, the device may be notified of "two emails that need attention" and "four important meetings."

[0193] User operation steps

[0194] Step 1: Vacation Settings

[0195] The user enters the system's settings screen and enters the start and end times of the vacation. As input, the start and end dates of the vacation are entered into the system, and the server receives this information. As output, the server-side settings are updated and the auto-response mode is ready. For example, the user sets "Vacation starts October 1, 2023" and "Vacation ends October 7, 2023."

[0196] Step 2: Vacation Check

[0197] The user checks the notifications displayed on the device while on vacation. As input, the user checks the notifications displayed on the device and takes emergency notices if necessary. As output, the user can take emergency measures. For example, if the message "You have an important notice" is displayed, the user checks the contents and takes action if necessary.

[0198] Step 3: Resuming work after the holidays

[0199] After the vacation ends, the user checks the summary report on their device. The summary report data is taken as input, and a plan is made to prioritize important matters. The output is the ability to quickly return to work based on work priorities. For example, "2 emails needing attention" and "4 important meetings" are displayed, allowing the user to quickly start dealing with the important tasks.

[0200] (Application example 1)

[0201] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0202] Modern business people are faced with the need to respond to emergencies and important tasks even while on vacation. However, answering phone calls, checking emails, and even responding to emergencies while on vacation can significantly reduce the quality of their vacation. This invention aims to solve this problem by providing a system that streamlines tasks that are treated as an extension of work even while users are on vacation, and enables rapid response to emergencies.

[0203] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0204] In this invention, the server includes means for automatically answering calls during vacation, means for converting received phone call content into text and analyzing it, means for determining the urgency of the call from the analyzed content and notifying an emergency contact as necessary, means for the user to monitor the surroundings using a visual device and detect abnormal behavior or an emergency, means for notifying the user of the detected abnormality or emergency by voice and providing instructions on its content, and means for analyzing the converted call content, assessing the urgency, and providing instructions on how to respond. This allows the user to effectively and efficiently handle work and emergencies even while on vacation.

[0205] A "system for improving work efficiency during vacation" is a system that automatically processes work on behalf of a user who is on vacation, improving efficiency after the user returns to work.

[0206] "Auto-answer" is a feature that allows the system to play a pre-defined answering message, eliminating the need for the user to answer the phone.

[0207] "Texting" is the process of converting audio data into text data, and is done using voice recognition technology.

[0208] "Analysis" is the process of using natural language processing techniques on text data to determine the intent and urgency of the content.

[0209] "Means for notifying emergency contacts" refers to a method for sending a notification to a designated emergency contact based on the results of the analyzed data.

[0210] A "visual device" is a device worn by a user that allows the user to obtain information visually, and is used as smart glasses or a head-mounted display.

[0211] The "means for detecting abnormal behavior and emergency situations" is a system that uses cameras and sensors in the visual device to monitor the user's surroundings and recognize abnormal activities and emergency situations.

[0212] "Voice notification means" is a function in which, when an emergency occurs, the system explains the situation to the user by voice and gives necessary instructions.

[0213] A "summary report" is a report that summarizes the collected data and comprehensively presents the information the user needs after completing their vacation.

[0214] The embodiment of the present invention is mainly composed of three components: a server, a terminal, and a user. This embodiment allows users to be freed from work during vacations while enabling a quick and appropriate response to emergencies.

[0215] server

[0216] The server plays a central role in this system, hosting the AI ​​model and processing data in real time. The server has the following specific functions:

[0217] 1. Auto-answer: Automatically detects incoming calls when the user is on vacation and plays a pre-defined answering message.

[0218] 2. Converting call content into text: Converting received phone calls into text data using voice recognition technology. This process is achieved using Google Speech API, etc.

[0219] 3. NLP analysis: Analyze text data using natural language processing technology (e.g., spaCy or NLTK) to determine the urgency of the call content.

[0220] 4. Emergency Notification: Based on the analysis results, notifications will be sent to pre-defined emergency contacts as necessary.

[0221] 5. Email and meeting summaries: Collect new emails and meeting contents from the user's mailbox and meeting recording data, summarize the collected data using a generative AI model (e.g., ChatGPT), and generate the results as a summary report.

[0222] 6. Prioritization: Based on the generated summary data, the importance and urgency are evaluated and prioritized.

[0223] 7. Generate and send summary report: At the end of the vacation, a summary of all collected data is compiled into a single summary report and sent to the user's device.

[0224] Terminal

[0225] The terminal refers to the user's smartphone or computer, and functions as an interface with the server. The specific operation of the terminal is as follows.

[0226] 1. Displaying phone call notification: Receives a notification from the server and displays a message to the user such as "AI has answered." If necessary, details of the call will also be displayed.

[0227] 2. Displaying summary reports: Receives summary reports sent from the server and notifies the user. The reports include summaries and priorities of emails and meetings received during vacation.

[0228] User

[0229] Users are staff who use the system, and there are specific operations and usage methods as follows.

[0230] 1. Vacation Settings: The user enters the start and end time of the vacation into the system, which prepares the server to activate the auto-answer mode.

[0231] 2. Checking while on vacation: The user checks the notifications displayed on the device while on vacation. If there is an emergency, the user receives a detailed notification from the server and takes appropriate action.

[0232] 3. Resuming work after vacation: After returning from vacation, users can check the summary report on their device. The report summarizes important emails and meeting details, allowing them to quickly start working on important tasks.

[0233] Specific examples

[0234] For example, if a user sets a one-week vacation, the server will automatically answer all incoming calls during the vacation period, converting the calls into text and analyzing them. If the analysis determines that an emergency response is required, the server will immediately send a notification to the emergency contact. At the same time, the server will regularly check the user's mailbox and meeting recordings, and generate summaries of new emails and meetings. The generated summaries are compiled into a summary report with prioritized information and sent to the user's device at the end of the vacation. When the user checks their device after the vacation, a summary report with all the necessary information will be displayed, allowing them to quickly start responding to important tasks.

[0235] Prompt Sentence Examples

[0236] "A suspicious person is detected while the user is on vacation. Real-time video and audio analysis is performed to determine the urgency. If the urgency is high, a notification is sent and response instructions are displayed."

[0237] In this way, the system of the present invention provides a concrete means for enabling users to enjoy their vacation while also returning to work efficiently after the vacation.

[0238] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0239] Step 1:

[0240] Vacation Settings

[0241] The user inputs the start and end times of the vacation into the system, which prepares the server to activate the automatic response mode. The start date and time and end date and time of the vacation are specified as input data, and the server sets the activation of the automatic response mode based on that information. Specifically, the user inputs the vacation information using the terminal interface, and the server saves the information in the database.

[0242] Step 2:

[0243] Automatic answering of incoming calls

[0244] If a call comes in while you're on vacation, the server automatically detects the call and plays a pre-defined answering message. It takes the received call's voice data as input and plays the answering message as output. Specifically, the server detects an incoming call and plays the answering message on an audio output device.

[0245] Step 3:

[0246] Transcription of call contents

[0247] The server converts the received voice data into text. In this step, it uses voice recognition technology such as Google Speech API to convert the input voice file into text data. The text data is output. Specifically, the server sends the voice data to the API and stores the converted text in a database.

[0248] Step 4:

[0249] NLP analysis

[0250] The server analyzes the text data using natural language processing (NLP) technology and evaluates its urgency. The input text is passed through an NLP module (e.g., spaCy or NLTK), and the data with an urgency rating is output. Specifically, if the analysis results indicate a high urgency, that information is reflected in the emergency notification settings.

[0251] Step 5:

[0252] emergency notification

[0253] Based on the analysis results, the server sends a notification to the pre-defined emergency contacts. The server uses the urgency assessment data as input data and sends a notification as output. Specifically, the server sends the notification to the emergency contacts via phone or message notification services.

[0254] Step 6:

[0255] Email and meeting summary generation

[0256] The server collects new emails and meeting contents from the user's mailbox and meeting recording data, and generates summaries using a generative AI model (e.g., ChatGPT). Emails and meeting records are used as input data, and summary text data is output. Specifically, the server periodically checks the mailbox and meeting recording data, and generates summaries by inputting the data into the generative AI model.

[0257] Step 7:

[0258] Prioritization

[0259] The server evaluates the importance and urgency of the generated summary data and assigns priorities to them. It uses the summary text as input data and outputs prioritized schedule data. Specifically, the server assigns a rank to each summary using an evaluation algorithm.

[0260] Step 8:

[0261] Generate and send summary reports

[0262] The server generates a summary report and sends it to the user's device. Prioritized summary data is used as input data, and a document constructed as a summary report is generated and sent as output. Specifically, the server sends the generated report to the user by email or other means.

[0263] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0264] The system of this invention consists of three main components: a server, a terminal, a user, and an emotion engine. Each component operates as follows:

[0265] server

[0266] The server plays a central role in the system, hosting the AI ​​model and emotion engine, and performing real-time data processing. The server has the following specific functions:

[0267] 1. Automated telephone answering

[0268] The server has the function of automatically detecting an incoming call when the user is on vacation and playing a pre-defined answering message.

[0269] 2. Transcription and analysis of call content

[0270] The server converts the received call content into text data using voice recognition technology, and then analyzes the text data using natural language processing (NLP) technology to determine the urgency of the call content.

[0271] 3. Collaboration with emotion engine

[0272] The server uses an emotion engine to analyze the user's emotions from their voice data. Based on the results of the emotion analysis, the server combines it with regular text analysis to more accurately determine the urgency of the message.

[0273] 4. Emergency notification

[0274] The server will send notifications to pre-configured emergency contacts as needed.

[0275] 5. Email and meeting summaries

[0276] The server collects new emails and meeting contents from the user's mailbox and meeting recording data, summarizes the collected data using a generation AI such as ChatGPT, and generates a summary report.

[0277] 6. Prioritize

[0278] The server takes into account the emotional data of emails and meeting contents analyzed using an emotion engine, and prioritizes them according to importance and urgency based on the summary data.

[0279] 7. Generate and send a summary report

[0280] The server compiles all new emails and summaries of meetings, including emotion data, into a single summary report at the end of the vacation, and sends it to the user's terminal.

[0281] Terminal

[0282] The terminal refers to the user's smartphone or computer, and acts as an interface with the server. The specific operations of the terminal are as follows:

[0283] 1. Display phone notifications

[0284] The device receives a notification from the server and displays a message to the user such as "AI has responded" along with the results of the emotion analysis. If necessary, details of the call content will also be displayed.

[0285] 2. View the summary report

[0286] The terminal receives a summary report sent from the server and notifies the user. The report includes a summary and priority of emails and meetings received during the vacation. It also displays the results of sentiment analysis.

[0287] User

[0288] The user is an employee who uses the system, and can be relieved from work during the vacation and return to work efficiently after the vacation. The specific operations and usage of the user are as follows:

[0289] 1. Vacation Settings

[0290] The user enters the start and end times of their vacation into the system, which prepares the server to activate the auto-answer mode.

[0291] 2. Checking during vacation

[0292] The user checks the notifications displayed on the device while on vacation. If there is an emergency, the user receives a detailed notification from the server and takes appropriate action.

[0293] 3. Resuming work after holidays

[0294] After the vacation, users can check a summary report on their device, which summarizes important emails and meeting details, allowing them to return to work smoothly based on their priorities.

[0295] Specific examples

[0296] For example, if a user sets a one-week vacation, the server will automatically respond to all calls during the vacation period, converting the calls into text and analyzing them. At the same time, the server uses an emotion engine to analyze the user's emotions from the voice data. For example, if the emotion analysis indicates a high level of urgency, the server will immediately send a notification to emergency contacts.

[0297] The server also periodically checks the user's mailbox and meeting recordings to generate summaries of new emails and meetings. Using an emotion engine, it analyzes the emotions of the collected data and prioritizes them based on importance and urgency. At the end of the vacation, a summary report containing the prioritized summaries is sent to the user's device.

[0298] When users check their device after their vacation, they will see a summary report that summarizes all the information they need, along with additional information based on sentiment analysis, allowing them to quickly start working on the most important tasks.

[0299] In this way, the system of the present invention allows users to enjoy their vacation while efficiently returning to work after their vacation. Furthermore, the emotion engine enables more accurate determination of urgency and prioritization of information.

[0300] The processing flow will be explained below.

[0301] Answering phone calls during holidays

[0302] Step 1:

[0303] The server receives the user's vacation setting information and enables the telephone auto-answer mode during the vacation period.

[0304] Step 2:

[0305] The server detects an incoming call while on vacation.

[0306] Step 3:

[0307] The server plays a pre-programmed answering message and converts the call into text using voice recognition technology.

[0308] Step 4:

[0309] The server analyzes the text of the call using natural language processing (NLP) technology to determine the urgency of the call.

[0310] Step 5:

[0311] The server uses an emotion engine to analyze emotions from the voice data, for example, detecting changes in tone and speed of the voice and using these as indicators of urgency.

[0312] Step 6:

[0313] The server combines the results of text analysis and sentiment analysis to determine the final urgency.

[0314] Step 7:

[0315] The server will send notifications to pre-configured emergency contacts as needed.

[0316] Step 8:

[0317] The server generates a summary of the response content and emotion analysis results and sends it to the user's terminal.

[0318] Step 9:

[0319] The device will notify the user that "AI has answered," along with the results of the emotion analysis. If necessary, it will also display details of the call.

[0320] Vacation email and meeting summaries

[0321] Step 1:

[0322] The server periodically checks the user's mailbox and collects new emails.

[0323] Step 2:

[0324] The server checks the user's calendar and meeting recording data to collect the corresponding meeting details.

[0325] Step 3:

[0326] The server analyzes the contents of newly arrived emails collected using ChatGPT and generates summaries.

[0327] Step 4:

[0328] The server converts the meeting recording data into text using speech recognition technology and uses ChatGPT to generate a summary that extracts the key points.

[0329] Step 5:

[0330] The server uses an emotion engine to analyze the user's emotions contained in emails and meeting content, for example, by assigning emotion labels to comments in emails and meetings.

[0331] Step 6:

[0332] The server determines the importance and urgency of emails and meetings based on the content of the generated summaries, and prioritizes them taking into account the results of sentiment analysis.

[0333] Step 7:

[0334] The server generates a priority list and sends it to the user's terminal.

[0335] Step 8:

[0336] The terminal receives the list and notifies the user.

[0337] Generate and view a summary report after the holidays

[0338] Step 1:

[0339] At the end of the vacation, the server compiles all collected new emails and summaries generated from the meeting content into a single summary report, incorporating sentiment analysis results.

[0340] Step 2:

[0341] The server transmits the generated summary report to the user's terminal.

[0342] Step 3:

[0343] The terminal notifies the user of receipt of the summary report.

[0344] Step 4:

[0345] The user can view a summary report on their device, which includes a summary and prioritization of all emails and meetings received during their vacation, as well as the results of sentiment analysis.

[0346] Step 5:

[0347] Based on the summary report, users can start responding to important emails and meeting details one by one.

[0348] Example 2

[0349] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0350] In today's business environment, employees are expected to respond efficiently to important phone calls, emails, and meeting details while on vacation. However, if work is completely neglected while on vacation, emergencies and important matters may be overlooked, resulting in a huge amount of work to be handled once the vacation returns. Furthermore, conventional systems do not perform sentiment analysis, making it difficult to determine urgency and insufficient to prioritize information. A new system is needed to solve these issues.

[0351] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0352] In this invention, the server includes: means for automatically answering calls during vacation; means for converting received call content into text using speech recognition technology; means for analyzing text data using natural language processing technology; means for analyzing emotions from the voice data; means for determining urgency based on the analyzed call content and emotion data and notifying emergency contacts as necessary; means for collecting new emails and meeting content from the user's mailbox and conference recording data; means for generating summaries of the collected data using a generative AI model; means for prioritizing from the summaries; means for generating a summary report including email and meeting summaries at the end of the vacation; and means for sending the generated summary report to the user. This allows for more efficient work during vacation and a smooth resumption of work after the vacation. Furthermore, using emotion analysis allows for more accurate determination of urgency and effective prioritization of information.

[0353] A "means for automatically answering vacation calls" is hardware or software that has the function of automatically detecting calls received while the user is on vacation and playing a pre-defined answering message.

[0354] "Means for converting received telephone calls into text using voice recognition technology" refers to hardware or software that uses voice recognition technology to convert the voice data of received telephone calls into text data.

[0355] "Means for analyzing text data using natural language processing technology" refers to hardware or software that uses natural language processing technology to analyze the content of text data obtained by speech recognition and extract useful information from it.

[0356] "Means for analyzing emotions from voice data" refers to hardware or software that utilizes emotion analysis technology to analyze the emotional state of a user or the other party from voice data or the text data converted from the voice data.

[0357] "Means for determining the urgency of a call based on the analyzed content and emotional data, and notifying emergency contacts as necessary" refers to hardware or software that evaluates the urgency of the content of a call based on the results of natural language processing and emotional analysis, and sends a notification to emergency contacts in the event of an emergency.

[0358] "Means for collecting new emails and conference contents from the user's mailbox and conference recording data" refers to hardware or software that periodically checks the user's mailbox and conference audio data to collect newly received emails and conference contents.

[0359] "Means for generating summaries of collected data using a generative AI model" refers to hardware or software capable of inputting collected email and meeting data into an AI model and generating summaries of the data.

[0360] The "means for prioritizing from summaries" refers to hardware or software capable of prioritizing the generated summary data based on importance and urgency.

[0361] "Means for generating a summary report including email and meeting summaries at the end of vacation" refers to hardware or software that compiles the data collected and summarized during the vacation into a single report at the end of the vacation.

[0362] The "means for transmitting the generated summary report to the user" is hardware or software for transmitting the generated summary report to the user's terminal after the vacation ends.

[0363] The "means for setting the start and end times of vacation" refers to an interface and system for a user to input the start and end times of vacation and record the schedule.

[0364] "Means for preparing the server to enable the auto-reply mode at the vacation start time" refers to programs and processes on the server for enabling the auto-reply mode when the vacation begins.

[0365] The "means for displaying vacation notices on a terminal" refers to hardware or software that receives notices from the server and displays them on the user's terminal.

[0366] The "means for reassessing urgency based on text data and emotion data" refers to hardware or software that reassess data obtained through natural language processing and emotion analysis to improve the accuracy of urgency.

[0367] "Means of sending notifications to emergency contacts via SMS or email when necessary" refers to a system that sends notifications to appropriate emergency contacts via SMS, email, etc. when a high level of urgency is deemed to exist.

[0368] "Means for inputting the collected data into a generative AI model to generate a summary and then integrating it with the sentiment analysis results" is hardware or software for inputting the collected data into an AI model and integrating the results with the sentiment analysis results.

[0369] The "means for using an algorithm to prioritize the generated summary data" refers to a system that applies a specific algorithm to set priorities for the summary data.

[0370] The system of this invention consists of three main components: a server, a terminal, and a user, as well as an emotion engine. Each component operates as follows:

[0371] server

[0372] The server plays a central role in this system, hosting various data processing and emotion engines. The specific hardware and software configuration is as follows:

[0373] 1. Automated telephone answering

[0374] The server uses an automated answering system such as Asterisk to automatically answer calls received while the user is on vacation. For example, when a user sets the start time of their vacation, the server launches Asterisk and plays a pre-recorded message.

[0375] 2. Transcription of call contents

[0376] The server uses the Google Cloud Speech-to-Text API to convert the call into text data. For example, when the call ends, the server saves the recording and calls the Google Cloud API to convert the voice data into text.

[0377] 3. Text Data Analysis

[0378] The server uses Python NLP libraries (such as NLTK and spaCy) to analyze the text data and determine the urgency of the call content. Specifically, it extracts important keywords and evaluates the urgency.

[0379] 4. Analysis by Emotion Engine

[0380] The server uses Azure Text Analytics for Sentiment and IBM Watson Tone Analyzer to analyze emotions from voice data, and based on the obtained emotional data, it can assess the urgency of the message with even greater accuracy.

[0381] 5. Sending emergency notifications

[0382] The server uses Twilio or SendGrid to send notifications to emergency contacts when necessary, for example, using Twilio to send SMS notifications for calls deemed to be urgent.

[0383] 6. Collect email and meeting data

[0384] The server periodically collects user mailbox and meeting recording data using the Gmail API and Microsoft Graph API. For example, it periodically calls the API every day to retrieve new emails and meeting data.

[0385] 7. Summarization using generative AI models

[0386] The server inputs the collected data into a generative AI model such as ChatGPT to generate a summary. For example, email data obtained from an API is passed to the generative AI model to obtain the summary result.

[0387] 8. Data Prioritization

[0388] The server prioritizes the summary data based on the importance and urgency of the data, based on the analysis results of the emotion engine. For example, data that is deemed "urgent" by emotion analysis has a high priority.

[0389] 9. Generate and send summary reports

[0390] The server compiles all the summary data into one summary report at the end of the vacation and sends it to the user's terminal using the SMTP protocol. For example, the server collects all the data on the day before the vacation ends and compiles it to generate a report.

[0391] Terminal

[0392] The terminal refers to the user's smartphone or computer, and functions as an interface with the server. The specific operation is as follows.

[0393] 1. Display phone notifications

[0394] The device receives a notification from the server and displays the emotion analysis results along with a message such as "AI has responded," prompting the user to confirm via the notification system.

[0395] 2. View the summary report

[0396] The device receives the summary report sent from the server and notifies the user, for example, by displaying the report through an email app or a dedicated app.

[0397] User

[0398] Users are employees who use the system, and the specific operations and usage are as follows.

[0399] 1. Vacation Settings

[0400] The user enters the start and end time of the vacation through the app, which causes the server to activate the auto-reply mode.

[0401] 2. Checking during vacation

[0402] The user checks the notifications displayed on the device while on vacation. If there is an emergency, the user receives a detailed notification from the server and takes appropriate action.

[0403] 3. Resuming work after holidays

[0404] After the vacation ends, the user checks the summary report on the terminal and starts working on important tasks based on the priority.

[0405] Specific examples

[0406] For example, if a user schedules a one-week vacation, the server will automatically answer all calls during that vacation period. When a call comes in, Asterisk detects it and plays a pre-recorded message. The call is recorded and converted to text data using Google Cloud Speech-to-Text. Next, the text data is analyzed using a Python NLP library to extract important keywords and evaluate the urgency. At the same time, Azure Text Analytics for Sentiment performs sentiment analysis of the call. If the call is deemed urgent, a notification is sent to emergency contacts using Twilio.

[0407] The server also periodically collects user mailbox and meeting recording data and generates summaries using generative AI models such as ChatGPT. The summary data is then prioritized based on the results of sentiment analysis. At the end of the vacation, a summary report containing all summaries is generated and sent to the user's device via SMTP. Users can then review the summary report after their vacation and quickly begin work on high-priority tasks.

[0408] Prompt Sentence Examples

[0409] "Please analyze my calls during my vacation and determine their urgency. Please also analyze the sentiment of the calls and send me emergency notifications if necessary. Also, please summarize emails and meetings and send me a summary report at the end."

[0410] In this way, the system of the present invention allows users to enjoy their vacation while efficiently returning to work after their vacation. The emotion engine also enables more accurate determination of urgency and prioritization of information.

[0411] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0412] Step 1: Set up your vacation time

[0413] Input: The user opens the dedicated application and enters the start and end dates and times of the vacation.

[0414] How it works: The user enters start and end dates and times into a vacation configuration form, the application validates them, and once validation is complete, the configuration information is sent to the server.

[0415] Output: The server receives the user's vacation settings and prepares to enable auto-answer mode at the vacation start time.

[0416] Step 2: Auto-answer your phone calls

[0417] Input: When the vacation begins, a call comes in to the user's phone.

[0418] How it works: The server uses Asterisk to detect incoming calls and play a pre-recorded greeting, such as "I'm on vacation right now."

[0419] Output: When a call comes in, an automatic answering message is played.

[0420] Step 3: Record and transcribe the call

[0421] Input: Audio data from the received call.

[0422] How it works: When the call ends, the call recording is saved to the server, which then calls the Google Cloud Speech-to-Text API to convert the audio data into text.

[0423] Output: The converted text data.

[0424] Step 4: Analyzing the text data

[0425] Input: The text data obtained in step 3.

[0426] How it works: The server uses Python NLP libraries (NLTK and spaCy) to analyze the text data and determine the urgency of the call. Specifically, it extracts important keywords (e.g., accident, emergency) and evaluates the urgency.

[0427] Output: Urgency assessment data of the call content.

[0428] Step 5: Analysis by Emotion Engine

[0429] Input: Text data from step 3.

[0430] How it works: The server uses Azure Text Analytics for Sentiment and IBM Watson Tone Analyzer to analyze emotions from the voice data. Based on the obtained emotional data, it can more accurately assess the urgency of the call.

[0431] Output: Sentiment analysis data of the call content.

[0432] Step 6: Send emergency notifications

[0433] Input: Urgency assessment data and sentiment analysis data from Step 4 and Step 5.

[0434] What it does: If the server determines there is an emergency, it will use Twilio or SendGrid to send a notification to emergency contacts, for example, calling the Twilio API to send an SMS notification.

[0435] Output: Notification message to emergency contacts.

[0436] Step 7: Collect email and meeting data

[0437] Input: User mailbox and meeting recording data.

[0438] How it works: The server periodically collects user mailbox and meeting recording data using the Gmail API or Microsoft Graph API. For example, it periodically calls the API every day to retrieve new emails and meeting data.

[0439] Output: Collected email and meeting data.

[0440] Step 8: Summarizing with a generative AI model

[0441] Input: Email and meeting data collected in step 7.

[0442] How it works: The server inputs the collected data into a generative AI model such as ChatGPT to generate a summary. For example, email data obtained from an API is passed to the generative AI model to obtain the summary result.

[0443] Output: The generated summary data.

[0444] Step 9: Prioritize your data

[0445] Input: Summary data from step 8 and sentiment analysis data from step 5.

[0446] How it works: The server prioritizes summary data based on importance and urgency based on the analysis results of the emotion engine. Specifically, data that is deemed "urgent" by emotion analysis is given a high priority.

[0447] Output: Prioritized summary data.

[0448] Step 10: Generate and send a summary report

[0449] Input: Prioritized summary data from step 9.

[0450] How it works: At the end of the vacation, the server compiles all the summary data into one summary report and sends it to the user's terminal using the SMTP protocol. For example, on the day before the vacation ends, the server collects all the data and generates the report.

[0451] Output: Summary report sent to the user.

[0452] Step 11: Check your vacation notices

[0453] Input: Notification data sent by the server.

[0454] How it works: The device receives a notification from the server and displays the emotion analysis results along with a message such as "AI has responded."

[0455] Output: The notification message displayed to the user.

[0456] Step 12: Resuming work after the holidays

[0457] Input: Summary report submitted in step 10.

[0458] Action: The user reviews the summary report and starts working on the most important tasks based on priority.

[0459] Output: Resume work starting with high priority tasks.

[0460] (Application example 2)

[0461] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0462] Errors and abnormal situations occurring during robotic operations in factories have a significant impact on work efficiency and productivity. Conventional robot monitoring systems often delay the detection and response of errors, preventing serious problems from being resolved quickly. Furthermore, analysis of error content and assessment of urgency are insufficient, preventing managers from being properly notified. This creates the risk of factory operations coming to a halt. Furthermore, because management reports for abnormal situations are not generated, it is difficult for managers to grasp the situation on a day-to-day basis. Another issue is that prioritization based on the urgency of errors is not performed, preventing appropriate responses.

[0463] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0464] In this invention, the server includes means for monitoring the working status of the robot, means for automatically generating a response message when the robot detects an error, means for converting the detected error into text and analyzing it, means for determining the urgency of the analyzed error, means for notifying a manager as necessary, means for recording the details and urgency of the error and generating a management report by summarizing daily abnormalities and errors, means for transmitting the generated management report, means for performing emotion analysis of the error content and prioritizing the content based on the urgency, and means for notifying the manager of events with a higher urgency first based on the priority. This enables a quick and appropriate response to errors and abnormalities, thereby improving the efficiency and safety of factory operations.

[0465] "Means for monitoring the working status of robots" refers to devices or systems that monitor the current operations and sensor information of robots operating in factories in real time and acquire their status.

[0466] "Means for automatically generating a response message" refers to a device or software that, when the robot detects an error, automatically generates a pre-set message based on the content of the error and instructs the robot to take an appropriate action.

[0467] "Means for converting detected error content into text and analyzing it" refers to a device or algorithm that converts the audio or log data of an error detected by a robot into text using natural language processing technology and analyzes that text.

[0468] The "means for determining the urgency of the analyzed error content" refers to a device or system that evaluates the severity and scope of the error based on the textualized and analyzed error information, and determines the urgency of the error.

[0469] The "means for notifying the administrator as necessary" refers to a device or service that promptly notifies the administrator of the details of an emergency error or abnormal situation when it occurs.

[0470] "Means for recording details and urgency of errors, summarizing daily abnormalities and errors, and generating management reports" refers to devices or software that store records of robot errors and abnormalities, compile and summarize daily data, and generate management reports that can be easily understood by managers.

[0471] The "means for transmitting the generated management report" refers to a device or system that transmits the management report to the administrator using email or a notification service.

[0472] "Means for performing emotional analysis of the error content and prioritizing the content based on its urgency" refers to a device or algorithm that evaluates the error content using emotional analysis technology, determines the urgency of the error based on the results, and sets priorities for response.

[0473] "Means for communicating urgent events to the administrator first based on priority" refers to devices or services that notify the administrator of urgent errors or abnormal situations on a priority basis, encouraging a rapid response.

[0474] To implement this invention, a system based on the following configuration and procedures is required: The system is mainly composed of three main components: a server, a robot, and an administrator terminal.

[0475] server

[0476] The server is the core of the system, and performs real-time data processing and analysis. Its specific functions are as follows:

[0477] 1. Monitoring the robot's working status

[0478] The server collects data from sensors and cameras installed on robots in the factory and monitors it in real time. This requires sensors and cameras to be installed on each robot. IoT technology is also used as the communication protocol for data collection.

[0479] 2. Error detection and automatic response

[0480] If the robot detects an error, the server immediately generates and sends a response message based on a pre-defined message. To achieve this, it leverages an AI model (e.g., ChatGPT).

[0481] 3. Error text and analysis

[0482] The detected error content is converted into text using speech recognition technology (e.g., Google Cloud Speech-to-Text) and then analyzed using NLP technology (e.g., Hugging Face Transformers).

[0483] 4. Urgency assessment and notification

[0484] The severity of the analyzed error is assessed using a sentiment analysis engine (e.g., IBM Watson Tone Analyzer), and if the severity is high, the administrator is notified using a notification system (e.g., Firebase Cloud Messaging or Twilio).

[0485] 5. Generate and send management reports

[0486] The server records the details and severity of the errors, and generates a management report summarizing the daily abnormalities and errors. The report is then periodically sent to the administrator.

[0487] robot

[0488] The robots operating in the factory are equipped with sensors and cameras and are responsible for sharing data with the server. If a robot detects an error, it immediately sends the data to the server.

[0489] Administrator terminal

[0490] The administrator's terminal receives and displays notifications and management reports sent from the server, allowing the administrator to quickly understand the situation and take necessary measures.

[0491] Specific examples

[0492] For example, if a factory robot detects an error on a production line, the sensor installed on the robot immediately sends the data to a server. The server uses NLP technology and a sentiment analysis engine to analyze the error and determine its level of urgency. If the level is high, it notifies the administrator using Firebase Cloud Messaging or Twilio with "Error code: 1234, Urgency: High." The server also aggregates daily error data, generates a management report, and sends it to the administrator's terminal.

[0493] An example prompt is:

[0494] "Please convert the error data collected from the robots in the factory into text and perform sentiment analysis to determine the urgency. If the urgency is high, please notify the administrator with 'Error Code: XXXX, Urgency: High' and generate a summary report."

[0495] This enables quick and appropriate responses to errors and abnormal situations, improving the efficiency and safety of factory operations.

[0496] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0497] Step 1:

[0498] The robot detects the error.

[0499] If the robot's onboard sensors or cameras detect an error or abnormality during operation, the information is immediately sent to the server, where the robot generates an error code, detailed sensor data, and a specific error message.

[0500] Input: Error data from sensors and cameras

[0501] Output: Error information sent to the server

[0502] Step 2:

[0503] The error information received by the server is converted to text.

[0504] The server converts the received error information into text data using speech recognition technology (e.g., Google Cloud Speech-to-Text). Here, sensor and camera data and error messages are recorded as text.

[0505] Input: Error information received from the robot

[0506] Output: Textual error data

[0507] Step 3:

[0508] The server parses the text data.

[0509] The server analyzes the textual error data using natural language processing (NLP) technology (e.g., Hugging Face Transformers) to identify the type and cause of the error. This analysis provides a detailed understanding of the error content.

[0510] Input: Textual error data

[0511] Output: Parsed error data

[0512] Step 4:

[0513] The server determines the urgency of the error.

[0514] Based on the analyzed error data, the server uses a sentiment analysis engine (e.g., IBM Watson Tone Analyzer) to evaluate the urgency of the error, which determines the severity of the error's impact on factory operations.

[0515] Input: Parsed error data

[0516] Output: Error data with urgency rating

[0517] Step 5:

[0518] The server notifies the administrator of any errors that are of high urgency.

[0519] If the error is deemed urgent, the server will use a notification system (e.g., Firebase Cloud Messaging or Twilio) to notify the administrator of the error details and urgency, including the error code, error message, and urgency.

[0520] Input: Error data with urgency rating

[0521] Output: Notification sent to administrator

[0522] Step 6:

[0523] The server records error data and summarizes daily anomalies and errors.

[0524] All error data and abnormal events are logged on a server, and the data is aggregated and summarized daily to generate management reports, including the frequency and urgency of errors.

[0525] Input: Error data with urgency rating

[0526] Output: Management Report

[0527] Step 7:

[0528] The server generates a management report and sends it to the administrator's terminal.

[0529] The generated management report is sent to the administrator's terminal on a regular basis, and the administrator can check it to understand any abnormalities that may have occurred.

[0530] Input: Management Report

[0531] Output: Management report sent to administrator

[0532] This enables quick and appropriate responses to errors and abnormal situations, improving the efficiency and safety of factory operations.

[0533] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0534] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0535] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0536] [Second embodiment]

[0537] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0538] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0539] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0540] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0541] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0542] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0543] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0544] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0545] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0546] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0547] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0548] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0549] The system of this invention consists of three main components: a server, a terminal, and a user. Each component operates as follows:

[0550] server

[0551] The server plays a central role in the system, hosting the AI ​​model and processing data in real time. The server has the following specific functions:

[0552] 1. Automated telephone answering

[0553] The server has the function of automatically detecting incoming calls when the user is on vacation and playing a pre-defined answering message, so that the user does not have to answer the phone directly while on vacation.

[0554] 2. Transcription and analysis of call content

[0555] The server converts the received call content into text data using voice recognition technology, and then analyzes the text data using natural language processing (NLP) technology to determine the urgency of the call content.

[0556] 3. Emergency notification

[0557] Based on the analysis results, the server will send notifications to pre-defined emergency contacts as necessary, ensuring that users only respond in truly urgent cases.

[0558] 4. Email and meeting summaries

[0559] The server collects new emails and meeting contents from the user's mailbox and meeting recording data, summarizes the collected data using a generation AI such as ChatGPT, and generates a summary report.

[0560] 5. Prioritize

[0561] The server evaluates the importance and urgency of the requests based on the generated summary data and assigns priorities to them, allowing users to return to work efficiently after their recovery.

[0562] 6. Generate and send a summary report

[0563] The server summarizes all collected data into a single summary report at the end of the vacation and sends it to the user's terminal.

[0564] Terminal

[0565] The terminal refers to the user's smartphone or computer, and acts as an interface with the server. The specific operations of the terminal are as follows:

[0566] 1. Display phone notifications

[0567] The device receives a notification from the server and displays a message to the user such as "AI has answered." If necessary, details of the call will also be displayed.

[0568] 2. View the summary report

[0569] The terminal receives the summary report sent from the server and notifies the user, which includes a summary and priority of emails and meetings received during the vacation.

[0570] User

[0571] The user is an employee who uses the system, and can be relieved from work during the vacation and return to work efficiently after the vacation. The specific operations and usage of the user are as follows:

[0572] 1. Vacation Settings

[0573] The user enters the start and end times of their vacation into the system, which prepares the server to activate the auto-answer mode.

[0574] 2. Checking during vacation

[0575] The user checks the notifications displayed on the device while on vacation. If there is an emergency, the user receives a detailed notification from the server and takes appropriate action.

[0576] 3. Resuming work after holidays

[0577] After the vacation, users can check a summary report on their device, which summarizes important emails and meeting details, allowing them to return to work smoothly based on their priorities.

[0578] Specific examples

[0579] For example, if a user sets a one-week vacation, the server will automatically answer all calls during that time, convert the calls into text, and analyze them. If the analysis determines that an emergency response is required, the server will immediately send a notification to the emergency contact.

[0580] At the same time, the server periodically checks the user's mailbox and conference recordings, generates summaries of new emails and conferences, and then prioritizes them into a summary report that is sent to the user's device at the end of the vacation.

[0581] When a user checks their device after a vacation, a summary report summarizing all the necessary information is displayed, allowing them to quickly start working on important tasks.

[0582] In this way, the system of the present invention allows users to maximize their vacation time while returning to work efficiently after their vacation.

[0583] The processing flow will be explained below.

[0584] Answering phone calls during holidays

[0585] Step 1:

[0586] The server receives the user's vacation setting information and enables the telephone auto-answer mode during the vacation period.

[0587] Step 2:

[0588] The server detects an incoming call while on vacation.

[0589] Step 3:

[0590] The server plays a pre-programmed answering message and converts the call into text using voice recognition technology.

[0591] Step 4:

[0592] The server analyzes the text of the call using natural language processing (NLP) technology to determine the urgency of the content.

[0593] Step 5:

[0594] The server will send notifications to pre-configured emergency contacts as needed.

[0595] Step 6:

[0596] The server generates a summary of the response content and sends it to the user's terminal.

[0597] Step 7:

[0598] The device will notify the user that "AI has responded" and, if necessary, display details of the response.

[0599] Vacation email and meeting summaries

[0600] Step 1:

[0601] The server periodically checks the user's mailbox and collects new emails.

[0602] Step 2:

[0603] The server checks the user's calendar and meeting recording data to collect the corresponding meeting details.

[0604] Step 3:

[0605] The server uses ChatGPT to analyze the contents of the collected new emails and generate summaries.

[0606] Step 4:

[0607] The server converts the meeting recording data into text using speech recognition technology and uses ChatGPT to generate a summary that extracts the key points.

[0608] Step 5:

[0609] The server determines the importance and urgency of emails and meetings based on the content of the generated summaries, and assigns priorities to them.

[0610] Step 6:

[0611] The server generates a priority list and sends it to the user's terminal.

[0612] Step 7:

[0613] The terminal receives the list and notifies the user.

[0614] Generate and view a summary report after the holidays

[0615] Step 1:

[0616] The server compiles all collected new emails and summaries generated from the meeting contents into one summary report at the end of the vacation.

[0617] Step 2:

[0618] The server transmits the generated summary report to the user's terminal.

[0619] Step 3:

[0620] The terminal notifies the user of receipt of the summary report.

[0621] Step 4:

[0622] The user checks the summary report on the terminal, which includes a summary and priority of all emails and meetings received during the vacation.

[0623] Step 5:

[0624] Based on the summary report, users can start responding to important emails and meeting details one by one.

[0625] Example 1

[0626] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0627] Employees are expected to be completely free from work during their vacation, while also being able to return to work quickly and efficiently after their vacation. However, traditional systems have limitations in ensuring that important communications or urgent responses are not overlooked during the vacation. Furthermore, dealing with the large amount of unprocessed emails and meeting details after the vacation takes time and effort, making it difficult to resume work smoothly.

[0628] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0629] In this invention, the server includes a means for automatically answering calls during vacation, a means for converting the contents of received calls into text data using voice recognition technology, and a means for analyzing the converted text data using natural language processing technology to determine the urgency of the call. This allows important communications and emergency responses to be made even during vacation, while allowing employees to return to work quickly and efficiently after their vacation.

[0630] "Means for automatically answering calls" refers to the ability of the system to answer calls on behalf of a vacation user and play a pre-defined voice message.

[0631] "Voice recognition technology" is a technology that analyzes voice data and converts it into text data.

[0632] "Text data" is character information converted from voice data using voice recognition technology.

[0633] "Natural language processing technology" is a technology that analyzes text data to understand its content, urgency, and intent.

[0634] The "means of determining urgency" is a function that uses natural language processing technology to analyze text data to identify important information and elements that require urgent action, and notify emergency contacts as necessary.

[0635] "Means for notifying emergency contacts" is a function that sends a notification to pre-set contacts when it is determined based on the analysis results that emergency contact is necessary.

[0636] A "mailbox" is a data storage device for receiving and storing a user's email.

[0637] "Meeting recording data" refers to data that records the audio information of a meeting.

[0638] A "generative AI model" is an artificial intelligence model that learns from large amounts of data to perform tasks such as text generation and summarization.

[0639] The "means of generating summaries" is a function that uses a generative AI model to extract key information from collected email and meeting recording data and provide it in a shortened format.

[0640] The "means for prioritizing" is a function that evaluates the importance and urgency of information based on the generated summary data and determines the order in which it should be processed.

[0641] A "Summary Report" is a report that lists key points and priorities generated from collected and summarized data.

[0642] A "terminal" is a device that allows a user to communicate with a server, and includes smartphones, computers, etc.

[0643] The "means for receiving notification" is a function by which the terminal receives information sent from the server and displays that information to the user.

[0644] This invention is based on a system that consists of three main components: a server, a terminal, and a user. Each component operates as follows:

[0645] server

[0646] The server plays a central role in this system, hosting the AI ​​model and processing data in real time. The server has the following specific functions:

[0647] 1. Automated telephone answering

[0648] The server automatically detects incoming calls from vacation users and plays pre-defined answering messages. Specifically, the server monitors the telephone line signal, and when a call comes in, it selects an appropriate voice file from templates stored in a database and plays it. For example, it could play a voice file that says, "I'm currently on vacation. I'll get back to you later."

[0649] 2. Transcription and analysis of call content

[0650] The server records the audio data of the call and converts it into text data using speech recognition technology, using technologies such as the Google Cloud Speech-to-Text API. The generated text data is then analyzed using natural language processing (NLP) technology to determine the urgency of the call content. The urgency is evaluated based on predefined rules (e.g., the presence of keywords such as "emergency" or "urgent"). For example, if the call content is something like "In the event of a fire," it is determined that an emergency response is required.

[0651] 3. Emergency notification

[0652] Based on the analysis, the server will send a notification to pre-defined emergency contacts if necessary. This notification can be sent in different ways, such as SMS, email, or push notification. For example, it may contain a message such as "A critical security issue has occurred. Your immediate attention is required."

[0653] 4. Email and meeting summaries

[0654] The server periodically checks the user's mailbox and meeting recording data. Specifically, it uses the Gmail API for the mailbox and speech recognition technology for the meeting recording. A generative AI model (e.g., ChatGPT) is used to generate a summary of the collected data. Prompts used include "Please summarize the following email" and "Please summarize the main points of this meeting." For example, a summary such as "The main topic of discussion was a progress report on a new project" can be generated.

[0655] 5. Prioritize

[0656] The server evaluates the importance and urgency of emails and meeting content based on the generated summary data, and assigns priorities to them. This prioritization is based on algorithms such as "urgency of time" and "job titles of the people involved." For example, "customer complaints" might be listed as the top priority.

[0657] 6. Generate and send a summary report

[0658] The server compiles all the summary data into a single summary report at the end of the vacation. This summary report is sent to the user's device via email or push notification. For example, it may contain a list of emails requiring replies under "Important matters during vacation," a summary of new emails, and key points from important meetings.

[0659] Terminal

[0660] The terminal refers to the user's smartphone or computer, and functions as an interface with the server. The specific operation of the terminal is as follows.

[0661] 1. Display phone notifications

[0662] The device receives notifications from the server in real time and displays them to the user. The notification content includes a message such as "AI has responded." For example, a notification may appear on the user's smartphone saying "An automatic response was made because the user is on vacation."

[0663] 2. View the summary report

[0664] The device receives the summary report sent from the server and notifies the user. The report contents can be checked from the notification center or a dedicated app, and important items are highlighted. For example, a notification may be displayed showing "two emails requiring attention" and "four important meetings."

[0665] User

[0666] The user is an employee who uses the system, and can be relieved from work during the vacation and return to work efficiently after the vacation. The specific operation and usage of the user are as follows.

[0667] 1. Vacation Settings

[0668] The user enters the system's settings screen and enters the start and end times of the vacation. The server receives this information and prepares the automatic response mode. For example, the user sets "Vacation start date: October 1, 2023" and "Vacation end date: October 7, 2023."

[0669] 2. Checking during vacation

[0670] While on vacation, users check the notifications displayed on their devices. Notifications with urgent nature will be displayed with special alerts, and users can respond accordingly. For example, if a message appears saying "You have an important notification," users can check the content and respond as necessary.

[0671] 3. Resuming work after holidays

[0672] After the vacation ends, the user checks the summary report on their device. The report summarizes important emails and meeting details, and the user can start addressing them based on priority. For example, the report shows "2 emails needing attention" and "4 important meetings," allowing the user to return to work immediately.

[0673] The system allows users to return to work quickly and efficiently without missing important communications while on vacation.

[0674] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0675] Server Processing Steps

[0676] Step 1: Auto-answer your phone

[0677] The server detects an incoming call. It receives the signal from the telephone line as input and plays a pre-defined answering message. For this, an audio file is selected from a database and played. For example, a voice message such as "I'm currently on vacation. I'll get back to you later" is played.

[0678] Step 2: Transcribe the call

[0679] The server records the audio data of the call. It receives the audio data as input and converts it into text data using speech recognition technology such as the Google Cloud Speech-to-Text API. The content of the call is saved in text format as output. For example, the content "There's a fire" is generated as text data.

[0680] Step 3: Analyzing the call

[0681] The server analyzes the generated text data using natural language processing (NLP) technology. It receives the text data as input and analyzes specific keywords and phrases to determine the urgency. The output is an assessment of the urgency level. For example, if the keyword "fire" is included, the urgency level is determined to be high.

[0682] Step 4: Send emergency notifications

[0683] Based on the analysis results, the server sends notifications to pre-configured emergency contacts as needed. It receives the urgency assessment result as input and sends notifications via SMS, email, etc. The output is a notification sent to the emergency contact. For example, a notification stating, "A critical security issue has occurred. Urgent action is required."

[0684] Step 5: Summarize emails and meetings

[0685] The server periodically checks the user's mailbox and meeting recording data. It receives email data and audio recording data as input and generates a summary using a generative AI model (e.g., ChatGPT). The output is a summary that extracts the key points. Prompts used include "Please summarize the following email" and "Please summarize the main points of this meeting." For example, a summary might be generated that reads, "The main topic was a progress report on a new project."

[0686] Step 6: Prioritize

[0687] The server evaluates the importance and urgency of the generated summary data and assigns priorities to them. It takes the summary data as input and runs a prioritization algorithm (e.g., time urgency, role of the involved parties). As output, a list of priorities is created. For example, "Customer complaints" may be listed as the highest priority.

[0688] Step 7: Generate and submit a summary report

[0689] The server compiles all the summary data into a single summary report when the vacation ends. It receives the generated summary data as input and formats it into a summary report. As output, it generates a summary report that is sent to the user's device at the end of the vacation. For example, it includes a list of emails that need replying as "important matters during vacation," a summary of new emails, and important meeting points.

[0690] Terminal processing steps

[0691] Step 1: View phone notifications

[0692] The device receives notifications from the server in real time and displays them to the user. Notifications from the server are received as input, and an application for displaying them on a smartphone or computer receives them. As output, a notification message is displayed to the user. For example, a notification saying "An automatic response has been sent because I am on vacation" is displayed.

[0693] Step 2: View the summary report

[0694] The device receives the summary report sent from the server and notifies the user. It receives the summary report data as input and displays it in a dedicated app or notification center. As output, the summary report is displayed to the user. For example, the device may be notified of "two emails that need attention" and "four important meetings."

[0695] User operation steps

[0696] Step 1: Vacation Settings

[0697] The user enters the system's settings screen and enters the start and end times of the vacation. As input, the start and end dates of the vacation are entered into the system, and the server receives this information. As output, the server-side settings are updated and the auto-response mode is ready. For example, the user sets "Vacation starts October 1, 2023" and "Vacation ends October 7, 2023."

[0698] Step 2: Vacation Check

[0699] The user checks the notifications displayed on the device while on vacation. As input, the user checks the notifications displayed on the device and takes emergency notices if necessary. As output, the user can take emergency measures. For example, if the message "You have an important notice" is displayed, the user checks the contents and takes action if necessary.

[0700] Step 3: Resuming work after the holidays

[0701] After the vacation ends, the user checks the summary report on their device. The summary report data is taken as input, and a plan is made to prioritize important matters. The output is the ability to quickly return to work based on work priorities. For example, "2 emails needing attention" and "4 important meetings" are displayed, allowing the user to quickly start dealing with the important tasks.

[0702] (Application example 1)

[0703] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0704] Modern business people are faced with the need to respond to emergencies and important tasks even while on vacation. However, answering phone calls, checking emails, and even responding to emergencies while on vacation can significantly reduce the quality of their vacation. This invention aims to solve this problem by providing a system that streamlines tasks that are treated as an extension of work even while users are on vacation, and enables rapid response to emergencies.

[0705] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0706] In this invention, the server includes means for automatically answering calls during vacation, means for converting received phone call content into text and analyzing it, means for determining the urgency of the call from the analyzed content and notifying an emergency contact as necessary, means for the user to monitor the surroundings using a visual device and detect abnormal behavior or an emergency, means for notifying the user of the detected abnormality or emergency by voice and providing instructions on its content, and means for analyzing the converted call content, assessing the urgency, and providing instructions on how to respond. This allows the user to effectively and efficiently handle work and emergencies even while on vacation.

[0707] A "system for improving work efficiency during vacation" is a system that automatically processes work on behalf of a user who is on vacation, improving efficiency after the user returns to work.

[0708] "Auto-answer" is a feature that allows the system to play a pre-defined answering message, eliminating the need for the user to answer the phone.

[0709] "Texting" is the process of converting audio data into text data, and is done using voice recognition technology.

[0710] "Analysis" is the process of using natural language processing techniques on text data to determine the intent and urgency of the content.

[0711] "Means for notifying emergency contacts" refers to a method for sending a notification to a designated emergency contact based on the results of the analyzed data.

[0712] A "visual device" is a device worn by a user that allows the user to obtain information visually, and is used as smart glasses or a head-mounted display.

[0713] The "means for detecting abnormal behavior and emergency situations" is a system that uses cameras and sensors in the visual device to monitor the user's surroundings and recognize abnormal activities and emergency situations.

[0714] "Voice notification means" is a function in which, when an emergency occurs, the system explains the situation to the user by voice and gives necessary instructions.

[0715] A "summary report" is a report that summarizes the collected data and comprehensively presents the information the user needs after completing their vacation.

[0716] The embodiment of the present invention is mainly composed of three components: a server, a terminal, and a user. This embodiment allows users to be freed from work during vacations while enabling a quick and appropriate response to emergencies.

[0717] server

[0718] The server plays a central role in this system, hosting the AI ​​model and processing data in real time. The server has the following specific functions:

[0719] 1. Auto-answer: Automatically detects incoming calls when the user is on vacation and plays a pre-defined answering message.

[0720] 2. Converting call content into text: Converting received phone calls into text data using voice recognition technology. This process is achieved using Google Speech API, etc.

[0721] 3. NLP analysis: Analyze text data using natural language processing technology (e.g., spaCy or NLTK) to determine the urgency of the call content.

[0722] 4. Emergency Notification: Based on the analysis results, notifications will be sent to pre-defined emergency contacts as necessary.

[0723] 5. Email and meeting summaries: Collect new emails and meeting contents from the user's mailbox and meeting recording data, summarize the collected data using a generative AI model (e.g., ChatGPT), and generate the results as a summary report.

[0724] 6. Prioritization: Based on the generated summary data, the importance and urgency are evaluated and prioritized.

[0725] 7. Generate and send summary report: At the end of the vacation, a summary of all collected data is compiled into a single summary report and sent to the user's device.

[0726] Terminal

[0727] The terminal refers to the user's smartphone or computer, and functions as an interface with the server. The specific operation of the terminal is as follows.

[0728] 1. Displaying phone call notification: Receives a notification from the server and displays a message to the user such as "AI has answered." If necessary, details of the call will also be displayed.

[0729] 2. Displaying summary reports: Receives summary reports sent from the server and notifies the user. The reports include summaries and priorities of emails and meetings received during vacation.

[0730] User

[0731] Users are staff who use the system, and there are specific operations and usage methods as follows.

[0732] 1. Vacation Settings: The user enters the start and end time of the vacation into the system, which prepares the server to activate the auto-answer mode.

[0733] 2. Checking while on vacation: The user checks the notifications displayed on the device while on vacation. If there is an emergency, the user receives a detailed notification from the server and takes appropriate action.

[0734] 3. Resuming work after vacation: After returning from vacation, users can check the summary report on their device. The report summarizes important emails and meeting details, allowing them to quickly start working on important tasks.

[0735] Specific examples

[0736] For example, if a user sets a one-week vacation, the server will automatically answer all incoming calls during the vacation period, converting the calls into text and analyzing them. If the analysis determines that an emergency response is required, the server will immediately send a notification to the emergency contact. At the same time, the server will regularly check the user's mailbox and meeting recordings, and generate summaries of new emails and meetings. The generated summaries are compiled into a summary report with prioritized information and sent to the user's device at the end of the vacation. When the user checks their device after the vacation, a summary report with all the necessary information will be displayed, allowing them to quickly start responding to important tasks.

[0737] Prompt Sentence Examples

[0738] "A suspicious person is detected while the user is on vacation. Real-time video and audio analysis is performed to determine the urgency. If the urgency is high, a notification is sent and response instructions are displayed."

[0739] In this way, the system of the present invention provides a concrete means for enabling users to enjoy their vacation while also returning to work efficiently after the vacation.

[0740] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0741] Step 1:

[0742] Vacation Settings

[0743] The user inputs the start and end times of the vacation into the system, which prepares the server to activate the automatic response mode. The start date and time and end date and time of the vacation are specified as input data, and the server sets the activation of the automatic response mode based on that information. Specifically, the user inputs the vacation information using the terminal interface, and the server saves the information in the database.

[0744] Step 2:

[0745] Automatic answering of incoming calls

[0746] If a call comes in while you're on vacation, the server automatically detects the call and plays a pre-defined answering message. It takes the received call's voice data as input and plays the answering message as output. Specifically, the server detects an incoming call and plays the answering message on an audio output device.

[0747] Step 3:

[0748] Transcription of call contents

[0749] The server converts the received voice data into text. In this step, it uses voice recognition technology such as Google Speech API to convert the input voice file into text data. The text data is output. Specifically, the server sends the voice data to the API and stores the converted text in a database.

[0750] Step 4:

[0751] NLP analysis

[0752] The server analyzes the text data using natural language processing (NLP) technology and evaluates its urgency. The input text is passed through an NLP module (e.g., spaCy or NLTK), and the data with an urgency rating is output. Specifically, if the analysis results indicate a high urgency, that information is reflected in the emergency notification settings.

[0753] Step 5:

[0754] emergency notification

[0755] Based on the analysis results, the server sends a notification to the pre-defined emergency contacts. The server uses the urgency assessment data as input data and sends a notification as output. Specifically, the server sends the notification to the emergency contacts via phone or message notification services.

[0756] Step 6:

[0757] Email and meeting summary generation

[0758] The server collects new emails and meeting contents from the user's mailbox and meeting recording data, and generates summaries using a generative AI model (e.g., ChatGPT). Emails and meeting records are used as input data, and summary text data is output. Specifically, the server periodically checks the mailbox and meeting recording data, and generates summaries by inputting the data into the generative AI model.

[0759] Step 7:

[0760] Prioritization

[0761] The server evaluates the importance and urgency of the generated summary data and assigns priorities to them. It uses the summary text as input data and outputs prioritized schedule data. Specifically, the server assigns a rank to each summary using an evaluation algorithm.

[0762] Step 8:

[0763] Generate and send summary reports

[0764] The server generates a summary report and sends it to the user's device. Prioritized summary data is used as input data, and a document constructed as a summary report is generated and sent as output. Specifically, the server sends the generated report to the user by email or other means.

[0765] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0766] The system of this invention consists of three main components: a server, a terminal, a user, and an emotion engine. Each component operates as follows:

[0767] server

[0768] The server plays a central role in the system, hosting the AI ​​model and emotion engine, and performing real-time data processing. The server has the following specific functions:

[0769] 1. Automated telephone answering

[0770] The server has the function of automatically detecting an incoming call when the user is on vacation and playing a pre-defined answering message.

[0771] 2. Transcription and analysis of call content

[0772] The server converts the received call content into text data using voice recognition technology, and then analyzes the text data using natural language processing (NLP) technology to determine the urgency of the call content.

[0773] 3. Collaboration with emotion engine

[0774] The server uses an emotion engine to analyze the user's emotions from their voice data. Based on the results of the emotion analysis, the server combines it with regular text analysis to more accurately determine the urgency of the message.

[0775] 4. Emergency notification

[0776] The server will send notifications to pre-configured emergency contacts as needed.

[0777] 5. Email and meeting summaries

[0778] The server collects new emails and meeting contents from the user's mailbox and meeting recording data, summarizes the collected data using a generation AI such as ChatGPT, and generates a summary report.

[0779] 6. Prioritize

[0780] The server takes into account the emotional data of emails and meeting contents analyzed using an emotion engine, and prioritizes them according to importance and urgency based on the summary data.

[0781] 7. Generate and send a summary report

[0782] The server compiles all new emails and summaries of meetings, including emotion data, into a single summary report at the end of the vacation, and sends it to the user's terminal.

[0783] Terminal

[0784] The terminal refers to the user's smartphone or computer, and acts as an interface with the server. The specific operations of the terminal are as follows:

[0785] 1. Display phone notifications

[0786] The device receives a notification from the server and displays a message to the user such as "AI has responded" along with the results of the emotion analysis. If necessary, details of the call content will also be displayed.

[0787] 2. View the summary report

[0788] The terminal receives a summary report sent from the server and notifies the user. The report includes a summary and priority of emails and meetings received during the vacation. It also displays the results of sentiment analysis.

[0789] User

[0790] The user is an employee who uses the system, and can be relieved from work during the vacation and return to work efficiently after the vacation. The specific operations and usage of the user are as follows:

[0791] 1. Vacation Settings

[0792] The user enters the start and end times of their vacation into the system, which prepares the server to activate the auto-answer mode.

[0793] 2. Checking during vacation

[0794] The user checks the notifications displayed on the device while on vacation. If there is an emergency, the user receives a detailed notification from the server and takes appropriate action.

[0795] 3. Resuming work after holidays

[0796] After the vacation, users can check a summary report on their device, which summarizes important emails and meeting details, allowing them to return to work smoothly based on their priorities.

[0797] Specific examples

[0798] For example, if a user sets a one-week vacation, the server will automatically respond to all calls during the vacation period, converting the calls into text and analyzing them. At the same time, the server uses an emotion engine to analyze the user's emotions from the voice data. For example, if the emotion analysis indicates a high level of urgency, the server will immediately send a notification to emergency contacts.

[0799] The server also periodically checks the user's mailbox and meeting recordings to generate summaries of new emails and meetings. Using an emotion engine, it analyzes the emotions of the collected data and prioritizes them based on importance and urgency. At the end of the vacation, a summary report containing the prioritized summaries is sent to the user's device.

[0800] When users check their device after their vacation, they will see a summary report that summarizes all the information they need, along with additional information based on sentiment analysis, allowing them to quickly start working on the most important tasks.

[0801] In this way, the system of the present invention allows users to enjoy their vacation while efficiently returning to work after their vacation. Furthermore, the emotion engine enables more accurate determination of urgency and prioritization of information.

[0802] The processing flow will be explained below.

[0803] Answering phone calls during holidays

[0804] Step 1:

[0805] The server receives the user's vacation setting information and enables the telephone auto-answer mode during the vacation period.

[0806] Step 2:

[0807] The server detects an incoming call while on vacation.

[0808] Step 3:

[0809] The server plays a pre-programmed answering message and converts the call into text using voice recognition technology.

[0810] Step 4:

[0811] The server analyzes the text of the call using natural language processing (NLP) technology to determine the urgency of the call.

[0812] Step 5:

[0813] The server uses an emotion engine to analyze emotions from the voice data, for example, detecting changes in tone and speed of the voice and using these as indicators of urgency.

[0814] Step 6:

[0815] The server combines the results of text analysis and sentiment analysis to determine the final urgency.

[0816] Step 7:

[0817] The server will send notifications to pre-configured emergency contacts as needed.

[0818] Step 8:

[0819] The server generates a summary of the response content and emotion analysis results and sends it to the user's terminal.

[0820] Step 9:

[0821] The device will notify the user that "AI has answered," along with the results of the emotion analysis. If necessary, it will also display details of the call.

[0822] Vacation email and meeting summaries

[0823] Step 1:

[0824] The server periodically checks the user's mailbox and collects new emails.

[0825] Step 2:

[0826] The server checks the user's calendar and meeting recording data to collect the corresponding meeting details.

[0827] Step 3:

[0828] The server analyzes the contents of newly arrived emails collected using ChatGPT and generates summaries.

[0829] Step 4:

[0830] The server converts the meeting recording data into text using speech recognition technology and uses ChatGPT to generate a summary that extracts the key points.

[0831] Step 5:

[0832] The server uses an emotion engine to analyze the user's emotions contained in emails and meeting content, for example, by assigning emotion labels to comments in emails and meetings.

[0833] Step 6:

[0834] The server determines the importance and urgency of emails and meetings based on the content of the generated summaries, and prioritizes them taking into account the results of sentiment analysis.

[0835] Step 7:

[0836] The server generates a priority list and sends it to the user's terminal.

[0837] Step 8:

[0838] The terminal receives the list and notifies the user.

[0839] Generate and view a summary report after the holidays

[0840] Step 1:

[0841] At the end of the vacation, the server compiles all collected new emails and summaries generated from the meeting content into a single summary report, incorporating sentiment analysis results.

[0842] Step 2:

[0843] The server transmits the generated summary report to the user's terminal.

[0844] Step 3:

[0845] The terminal notifies the user of receipt of the summary report.

[0846] Step 4:

[0847] The user can view a summary report on their device, which includes a summary and prioritization of all emails and meetings received during their vacation, as well as the results of sentiment analysis.

[0848] Step 5:

[0849] Based on the summary report, users can start responding to important emails and meeting details one by one.

[0850] Example 2

[0851] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0852] In today's business environment, employees are expected to respond efficiently to important phone calls, emails, and meeting details while on vacation. However, if work is completely neglected while on vacation, emergencies and important matters may be overlooked, resulting in a huge amount of work to be handled once the vacation returns. Furthermore, conventional systems do not perform sentiment analysis, making it difficult to determine urgency and insufficient to prioritize information. A new system is needed to solve these issues.

[0853] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0854] In this invention, the server includes: means for automatically answering calls during vacation; means for converting received call content into text using speech recognition technology; means for analyzing text data using natural language processing technology; means for analyzing emotions from the voice data; means for determining urgency based on the analyzed call content and emotion data and notifying emergency contacts as necessary; means for collecting new emails and meeting content from the user's mailbox and conference recording data; means for generating summaries of the collected data using a generative AI model; means for prioritizing from the summaries; means for generating a summary report including email and meeting summaries at the end of the vacation; and means for sending the generated summary report to the user. This allows for more efficient work during vacation and a smooth resumption of work after the vacation. Furthermore, using emotion analysis allows for more accurate determination of urgency and effective prioritization of information.

[0855] A "means for automatically answering vacation calls" is hardware or software that has the function of automatically detecting calls received while the user is on vacation and playing a pre-defined answering message.

[0856] "Means for converting received telephone calls into text using voice recognition technology" refers to hardware or software that uses voice recognition technology to convert the voice data of received telephone calls into text data.

[0857] "Means for analyzing text data using natural language processing technology" refers to hardware or software that uses natural language processing technology to analyze the content of text data obtained by speech recognition and extract useful information from it.

[0858] "Means for analyzing emotions from voice data" refers to hardware or software that utilizes emotion analysis technology to analyze the emotional state of a user or the other party from voice data or the text data converted from the voice data.

[0859] "Means for determining the urgency of a call based on the analyzed content and emotional data, and notifying emergency contacts as necessary" refers to hardware or software that evaluates the urgency of the content of a call based on the results of natural language processing and emotional analysis, and sends a notification to emergency contacts in the event of an emergency.

[0860] "Means for collecting new emails and conference contents from the user's mailbox and conference recording data" refers to hardware or software that periodically checks the user's mailbox and conference audio data to collect newly received emails and conference contents.

[0861] "Means for generating summaries of collected data using a generative AI model" refers to hardware or software capable of inputting collected email and meeting data into an AI model and generating summaries of the data.

[0862] The "means for prioritizing from summaries" refers to hardware or software capable of prioritizing the generated summary data based on importance and urgency.

[0863] "Means for generating a summary report including email and meeting summaries at the end of vacation" refers to hardware or software that compiles the data collected and summarized during the vacation into a single report at the end of the vacation.

[0864] The "means for transmitting the generated summary report to the user" is hardware or software for transmitting the generated summary report to the user's terminal after the vacation ends.

[0865] The "means for setting the start and end times of vacation" refers to an interface and system for a user to input the start and end times of vacation and record the schedule.

[0866] "Means for preparing the server to enable the auto-reply mode at the vacation start time" refers to programs and processes on the server for enabling the auto-reply mode when the vacation begins.

[0867] The "means for displaying vacation notices on a terminal" refers to hardware or software that receives notices from the server and displays them on the user's terminal.

[0868] The "means for reassessing urgency based on text data and emotion data" refers to hardware or software that reassess data obtained through natural language processing and emotion analysis to improve the accuracy of urgency.

[0869] "Means of sending notifications to emergency contacts via SMS or email when necessary" refers to a system that sends notifications to appropriate emergency contacts via SMS, email, etc. when a high level of urgency is deemed to exist.

[0870] "Means for inputting the collected data into a generative AI model to generate a summary and then integrating it with the sentiment analysis results" is hardware or software for inputting the collected data into an AI model and integrating the results with the sentiment analysis results.

[0871] The "means for using an algorithm to prioritize the generated summary data" refers to a system that applies a specific algorithm to set priorities for the summary data.

[0872] The system of this invention consists of three main components: a server, a terminal, and a user, as well as an emotion engine. Each component operates as follows:

[0873] server

[0874] The server plays a central role in this system, hosting various data processing and emotion engines. The specific hardware and software configuration is as follows:

[0875] 1. Automated telephone answering

[0876] The server uses an automated answering system such as Asterisk to automatically answer calls received while the user is on vacation. For example, when a user sets the start time of their vacation, the server launches Asterisk and plays a pre-recorded message.

[0877] 2. Transcription of call contents

[0878] The server uses the Google Cloud Speech-to-Text API to convert the call into text data. For example, when the call ends, the server saves the recording and calls the Google Cloud API to convert the voice data into text.

[0879] 3. Text Data Analysis

[0880] The server uses Python NLP libraries (such as NLTK and spaCy) to analyze the text data and determine the urgency of the call content. Specifically, it extracts important keywords and evaluates the urgency.

[0881] 4. Analysis by Emotion Engine

[0882] The server uses Azure Text Analytics for Sentiment and IBM Watson Tone Analyzer to analyze emotions from voice data, and based on the obtained emotional data, it can assess the urgency of the message with even greater accuracy.

[0883] 5. Sending emergency notifications

[0884] The server uses Twilio or SendGrid to send notifications to emergency contacts when necessary, for example, using Twilio to send SMS notifications for calls deemed to be urgent.

[0885] 6. Collect email and meeting data

[0886] The server periodically collects user mailbox and meeting recording data using the Gmail API and Microsoft Graph API. For example, it periodically calls the API every day to retrieve new emails and meeting data.

[0887] 7. Summarization using generative AI models

[0888] The server inputs the collected data into a generative AI model such as ChatGPT to generate a summary. For example, email data obtained from an API is passed to the generative AI model to obtain the summary result.

[0889] 8. Data Prioritization

[0890] The server prioritizes the summary data based on the importance and urgency of the data, based on the analysis results of the emotion engine. For example, data that is deemed "urgent" by emotion analysis has a high priority.

[0891] 9. Generate and send summary reports

[0892] The server compiles all the summary data into one summary report at the end of the vacation and sends it to the user's terminal using the SMTP protocol. For example, the server collects all the data on the day before the vacation ends and compiles it to generate a report.

[0893] Terminal

[0894] The terminal refers to the user's smartphone or computer, and functions as an interface with the server. The specific operation is as follows.

[0895] 1. Display phone notifications

[0896] The device receives a notification from the server and displays the emotion analysis results along with a message such as "AI has responded," prompting the user to confirm via the notification system.

[0897] 2. View the summary report

[0898] The device receives the summary report sent from the server and notifies the user, for example, by displaying the report through an email app or a dedicated app.

[0899] User

[0900] Users are employees who use the system, and the specific operations and usage are as follows.

[0901] 1. Vacation Settings

[0902] The user enters the start and end time of the vacation through the app, which causes the server to activate the auto-reply mode.

[0903] 2. Checking during vacation

[0904] The user checks the notifications displayed on the device while on vacation. If there is an emergency, the user receives a detailed notification from the server and takes appropriate action.

[0905] 3. Resuming work after holidays

[0906] After the vacation ends, the user checks the summary report on the terminal and starts working on important tasks based on the priority.

[0907] Specific examples

[0908] For example, if a user schedules a one-week vacation, the server will automatically answer all calls during that vacation period. When a call comes in, Asterisk detects it and plays a pre-recorded message. The call is recorded and converted to text data using Google Cloud Speech-to-Text. Next, the text data is analyzed using a Python NLP library to extract important keywords and evaluate the urgency. At the same time, Azure Text Analytics for Sentiment performs sentiment analysis of the call. If the call is deemed urgent, a notification is sent to emergency contacts using Twilio.

[0909] The server also periodically collects user mailbox and meeting recording data and generates summaries using generative AI models such as ChatGPT. The summary data is then prioritized based on the results of sentiment analysis. At the end of the vacation, a summary report containing all summaries is generated and sent to the user's device via SMTP. Users can then review the summary report after their vacation and quickly begin work on high-priority tasks.

[0910] Prompt Sentence Examples

[0911] "Please analyze my calls during my vacation and determine their urgency. Please also analyze the sentiment of the calls and send me emergency notifications if necessary. Also, please summarize emails and meetings and send me a summary report at the end."

[0912] In this way, the system of the present invention allows users to enjoy their vacation while efficiently returning to work after their vacation. The emotion engine also enables more accurate determination of urgency and prioritization of information.

[0913] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0914] Step 1: Set up your vacation time

[0915] Input: The user opens the dedicated application and enters the start and end dates and times of the vacation.

[0916] How it works: The user enters start and end dates and times into a vacation configuration form, the application validates them, and once validation is complete, the configuration information is sent to the server.

[0917] Output: The server receives the user's vacation settings and prepares to enable auto-answer mode at the vacation start time.

[0918] Step 2: Auto-answer your phone calls

[0919] Input: When the vacation begins, a call comes in to the user's phone.

[0920] How it works: The server uses Asterisk to detect incoming calls and play a pre-recorded greeting, such as "I'm on vacation right now."

[0921] Output: When a call comes in, an automatic answering message is played.

[0922] Step 3: Record and transcribe the call

[0923] Input: Audio data from the received call.

[0924] How it works: When the call ends, the call recording is saved to the server, which then calls the Google Cloud Speech-to-Text API to convert the audio data into text.

[0925] Output: The converted text data.

[0926] Step 4: Analyzing the text data

[0927] Input: The text data obtained in step 3.

[0928] How it works: The server uses Python NLP libraries (NLTK and spaCy) to analyze the text data and determine the urgency of the call. Specifically, it extracts important keywords (e.g., accident, emergency) and evaluates the urgency.

[0929] Output: Urgency assessment data of the call content.

[0930] Step 5: Analysis by Emotion Engine

[0931] Input: Text data from step 3.

[0932] How it works: The server uses Azure Text Analytics for Sentiment and IBM Watson Tone Analyzer to analyze emotions from the voice data. Based on the obtained emotional data, it can more accurately assess the urgency of the call.

[0933] Output: Sentiment analysis data of the call content.

[0934] Step 6: Send emergency notifications

[0935] Input: Urgency assessment data and sentiment analysis data from Step 4 and Step 5.

[0936] What it does: If the server determines there is an emergency, it will use Twilio or SendGrid to send a notification to emergency contacts, for example, calling the Twilio API to send an SMS notification.

[0937] Output: Notification message to emergency contacts.

[0938] Step 7: Collect email and meeting data

[0939] Input: User mailbox and meeting recording data.

[0940] How it works: The server periodically collects user mailbox and meeting recording data using the Gmail API or Microsoft Graph API. For example, it periodically calls the API every day to retrieve new emails and meeting data.

[0941] Output: Collected email and meeting data.

[0942] Step 8: Summarizing with a generative AI model

[0943] Input: Email and meeting data collected in step 7.

[0944] How it works: The server inputs the collected data into a generative AI model such as ChatGPT to generate a summary. For example, email data obtained from an API is passed to the generative AI model to obtain the summary result.

[0945] Output: The generated summary data.

[0946] Step 9: Prioritize your data

[0947] Input: Summary data from step 8 and sentiment analysis data from step 5.

[0948] How it works: The server prioritizes summary data based on importance and urgency based on the analysis results of the emotion engine. Specifically, data that is deemed "urgent" by emotion analysis is given a high priority.

[0949] Output: Prioritized summary data.

[0950] Step 10: Generate and send a summary report

[0951] Input: Prioritized summary data from step 9.

[0952] How it works: At the end of the vacation, the server compiles all the summary data into one summary report and sends it to the user's terminal using the SMTP protocol. For example, on the day before the vacation ends, the server collects all the data and generates the report.

[0953] Output: Summary report sent to the user.

[0954] Step 11: Check your vacation notices

[0955] Input: Notification data sent by the server.

[0956] How it works: The device receives a notification from the server and displays the emotion analysis results along with a message such as "AI has responded."

[0957] Output: The notification message displayed to the user.

[0958] Step 12: Resuming work after the holidays

[0959] Input: Summary report submitted in step 10.

[0960] Action: The user reviews the summary report and starts working on the most important tasks based on priority.

[0961] Output: Resume work starting with high priority tasks.

[0962] (Application example 2)

[0963] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0964] Errors and abnormal situations occurring during robotic operations in factories have a significant impact on work efficiency and productivity. Conventional robot monitoring systems often delay the detection and response of errors, preventing serious problems from being resolved quickly. Furthermore, analysis of error content and assessment of urgency are insufficient, preventing managers from being properly notified. This creates the risk of factory operations coming to a halt. Furthermore, because management reports for abnormal situations are not generated, it is difficult for managers to grasp the situation on a day-to-day basis. Another issue is that prioritization based on the urgency of errors is not performed, preventing appropriate responses.

[0965] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0966] In this invention, the server includes means for monitoring the working status of the robot, means for automatically generating a response message when the robot detects an error, means for converting the detected error into text and analyzing it, means for determining the urgency of the analyzed error, means for notifying a manager as necessary, means for recording the details and urgency of the error and generating a management report by summarizing daily abnormalities and errors, means for transmitting the generated management report, means for performing emotion analysis of the error content and prioritizing the content based on the urgency, and means for notifying the manager of events with a higher urgency first based on the priority. This enables a quick and appropriate response to errors and abnormalities, thereby improving the efficiency and safety of factory operations.

[0967] "Means for monitoring the working status of robots" refers to devices or systems that monitor the current operations and sensor information of robots operating in factories in real time and acquire their status.

[0968] "Means for automatically generating a response message" refers to a device or software that, when the robot detects an error, automatically generates a pre-set message based on the content of the error and instructs the robot to take an appropriate action.

[0969] "Means for converting detected error content into text and analyzing it" refers to a device or algorithm that converts the audio or log data of an error detected by a robot into text using natural language processing technology and analyzes that text.

[0970] The "means for determining the urgency of the analyzed error content" refers to a device or system that evaluates the severity and scope of the error based on the textualized and analyzed error information, and determines the urgency of the error.

[0971] The "means for notifying the administrator as necessary" refers to a device or service that promptly notifies the administrator of the details of an emergency error or abnormal situation when it occurs.

[0972] "Means for recording details and urgency of errors, summarizing daily abnormalities and errors, and generating management reports" refers to devices or software that store records of robot errors and abnormalities, compile and summarize daily data, and generate management reports that can be easily understood by managers.

[0973] The "means for transmitting the generated management report" refers to a device or system that transmits the management report to the administrator using email or a notification service.

[0974] "Means for performing emotional analysis of the error content and prioritizing the content based on its urgency" refers to a device or algorithm that evaluates the error content using emotional analysis technology, determines the urgency of the error based on the results, and sets priorities for response.

[0975] "Means for communicating urgent events to the administrator first based on priority" refers to devices or services that notify the administrator of urgent errors or abnormal situations on a priority basis, encouraging a rapid response.

[0976] To implement this invention, a system based on the following configuration and procedures is required: The system is mainly composed of three main components: a server, a robot, and an administrator terminal.

[0977] server

[0978] The server is the core of the system, and performs real-time data processing and analysis. Its specific functions are as follows:

[0979] 1. Monitoring the robot's working status

[0980] The server collects data from sensors and cameras installed on robots in the factory and monitors it in real time. This requires sensors and cameras to be installed on each robot. IoT technology is also used as the communication protocol for data collection.

[0981] 2. Error detection and automatic response

[0982] If the robot detects an error, the server immediately generates and sends a response message based on a pre-defined message. To achieve this, it leverages an AI model (e.g., ChatGPT).

[0983] 3. Error text and analysis

[0984] The detected error content is converted into text using speech recognition technology (e.g., Google Cloud Speech-to-Text) and then analyzed using NLP technology (e.g., Hugging Face Transformers).

[0985] 4. Urgency assessment and notification

[0986] The severity of the analyzed error is assessed using a sentiment analysis engine (e.g., IBM Watson Tone Analyzer), and if the severity is high, the administrator is notified using a notification system (e.g., Firebase Cloud Messaging or Twilio).

[0987] 5. Generate and send management reports

[0988] The server records the details and severity of the errors, and generates a management report summarizing the daily abnormalities and errors. The report is then periodically sent to the administrator.

[0989] robot

[0990] The robots operating in the factory are equipped with sensors and cameras and are responsible for sharing data with the server. If a robot detects an error, it immediately sends the data to the server.

[0991] Administrator terminal

[0992] The administrator's terminal receives and displays notifications and management reports sent from the server, allowing the administrator to quickly understand the situation and take necessary measures.

[0993] Specific examples

[0994] For example, if a factory robot detects an error on a production line, the sensor installed on the robot immediately sends the data to a server. The server uses NLP technology and a sentiment analysis engine to analyze the error and determine its level of urgency. If the level is high, it notifies the administrator using Firebase Cloud Messaging or Twilio with "Error code: 1234, Urgency: High." The server also aggregates daily error data, generates a management report, and sends it to the administrator's terminal.

[0995] An example prompt is:

[0996] "Please convert the error data collected from the robots in the factory into text and perform sentiment analysis to determine the urgency. If the urgency is high, please notify the administrator with 'Error Code: XXXX, Urgency: High' and generate a summary report."

[0997] This enables quick and appropriate responses to errors and abnormal situations, improving the efficiency and safety of factory operations.

[0998] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0999] Step 1:

[1000] The robot detects the error.

[1001] If the robot's onboard sensors or cameras detect an error or abnormality during operation, the information is immediately sent to the server, where the robot generates an error code, detailed sensor data, and a specific error message.

[1002] Input: Error data from sensors and cameras

[1003] Output: Error information sent to the server

[1004] Step 2:

[1005] The error information received by the server is converted to text.

[1006] The server converts the received error information into text data using speech recognition technology (e.g., Google Cloud Speech-to-Text). Here, sensor and camera data and error messages are recorded as text.

[1007] Input: Error information received from the robot

[1008] Output: Textual error data

[1009] Step 3:

[1010] The server parses the text data.

[1011] The server analyzes the textual error data using natural language processing (NLP) technology (e.g., Hugging Face Transformers) to identify the type and cause of the error. This analysis provides a detailed understanding of the error content.

[1012] Input: Textual error data

[1013] Output: Parsed error data

[1014] Step 4:

[1015] The server determines the urgency of the error.

[1016] Based on the analyzed error data, the server uses a sentiment analysis engine (e.g., IBM Watson Tone Analyzer) to evaluate the urgency of the error, which determines the severity of the error's impact on factory operations.

[1017] Input: Parsed error data

[1018] Output: Error data with urgency rating

[1019] Step 5:

[1020] The server notifies the administrator of any errors that are of high urgency.

[1021] If the error is deemed urgent, the server will use a notification system (e.g., Firebase Cloud Messaging or Twilio) to notify the administrator of the error details and urgency, including the error code, error message, and urgency.

[1022] Input: Error data with urgency rating

[1023] Output: Notification sent to administrator

[1024] Step 6:

[1025] The server records error data and summarizes daily anomalies and errors.

[1026] All error data and abnormal events are logged on a server, and the data is aggregated and summarized daily to generate management reports, including the frequency and urgency of errors.

[1027] Input: Error data with urgency rating

[1028] Output: Management Report

[1029] Step 7:

[1030] The server generates a management report and sends it to the administrator's terminal.

[1031] The generated management report is sent to the administrator's terminal on a regular basis, and the administrator can check it to understand any abnormalities that may have occurred.

[1032] Input: Management Report

[1033] Output: Management report sent to administrator

[1034] This enables quick and appropriate responses to errors and abnormal situations, improving the efficiency and safety of factory operations.

[1035] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1036] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1037] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1038] [Third embodiment]

[1039] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1040] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1041] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1042] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1043] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1044] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1045] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1046] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1047] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1048] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1049] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1050] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1051] The system of this invention consists of three main components: a server, a terminal, and a user. Each component operates as follows:

[1052] server

[1053] The server plays a central role in the system, hosting the AI ​​model and processing data in real time. The server has the following specific functions:

[1054] 1. Automated telephone answering

[1055] The server has the function of automatically detecting incoming calls when the user is on vacation and playing a pre-defined answering message, so that the user does not have to answer the phone directly while on vacation.

[1056] 2. Transcription and analysis of call content

[1057] The server converts the received call content into text data using voice recognition technology, and then analyzes the text data using natural language processing (NLP) technology to determine the urgency of the call content.

[1058] 3. Emergency notification

[1059] Based on the analysis results, the server will send notifications to pre-defined emergency contacts as necessary, ensuring that users only respond in truly urgent cases.

[1060] 4. Email and meeting summaries

[1061] The server collects new emails and meeting contents from the user's mailbox and meeting recording data, summarizes the collected data using a generation AI such as ChatGPT, and generates a summary report.

[1062] 5. Prioritize

[1063] The server evaluates the importance and urgency of the requests based on the generated summary data and assigns priorities to them, allowing users to return to work efficiently after their recovery.

[1064] 6. Generate and send a summary report

[1065] The server summarizes all collected data into a single summary report at the end of the vacation and sends it to the user's terminal.

[1066] Terminal

[1067] The terminal refers to the user's smartphone or computer, and acts as an interface with the server. The specific operations of the terminal are as follows:

[1068] 1. Display phone notifications

[1069] The device receives a notification from the server and displays a message to the user such as "AI has answered." If necessary, details of the call will also be displayed.

[1070] 2. View the summary report

[1071] The terminal receives the summary report sent from the server and notifies the user, which includes a summary and priority of emails and meetings received during the vacation.

[1072] User

[1073] The user is an employee who uses the system, and can be relieved from work during the vacation and return to work efficiently after the vacation. The specific operations and usage of the user are as follows:

[1074] 1. Vacation Settings

[1075] The user enters the start and end times of their vacation into the system, which prepares the server to activate the auto-answer mode.

[1076] 2. Checking during vacation

[1077] The user checks the notifications displayed on the device while on vacation. If there is an emergency, the user receives a detailed notification from the server and takes appropriate action.

[1078] 3. Resuming work after holidays

[1079] After the vacation, users can check a summary report on their device, which summarizes important emails and meeting details, allowing them to return to work smoothly based on their priorities.

[1080] Specific examples

[1081] For example, if a user sets a one-week vacation, the server will automatically answer all calls during that time, convert the calls into text, and analyze them. If the analysis determines that an emergency response is required, the server will immediately send a notification to the emergency contact.

[1082] At the same time, the server periodically checks the user's mailbox and conference recordings, generates summaries of new emails and conferences, and then prioritizes them into a summary report that is sent to the user's device at the end of the vacation.

[1083] When a user checks their device after a vacation, a summary report summarizing all the necessary information is displayed, allowing them to quickly start working on important tasks.

[1084] In this way, the system of the present invention allows users to maximize their vacation time while returning to work efficiently after their vacation.

[1085] The processing flow will be explained below.

[1086] Answering phone calls during holidays

[1087] Step 1:

[1088] The server receives the user's vacation setting information and enables the telephone auto-answer mode during the vacation period.

[1089] Step 2:

[1090] The server detects an incoming call while on vacation.

[1091] Step 3:

[1092] The server plays a pre-programmed answering message and converts the call into text using voice recognition technology.

[1093] Step 4:

[1094] The server analyzes the text of the call using natural language processing (NLP) technology to determine the urgency of the content.

[1095] Step 5:

[1096] The server will send notifications to pre-configured emergency contacts as needed.

[1097] Step 6:

[1098] The server generates a summary of the response content and sends it to the user's terminal.

[1099] Step 7:

[1100] The device will notify the user that "AI has responded" and, if necessary, display details of the response.

[1101] Vacation email and meeting summaries

[1102] Step 1:

[1103] The server periodically checks the user's mailbox and collects new emails.

[1104] Step 2:

[1105] The server checks the user's calendar and meeting recording data to collect the corresponding meeting details.

[1106] Step 3:

[1107] The server uses ChatGPT to analyze the contents of the collected new emails and generate summaries.

[1108] Step 4:

[1109] The server converts the meeting recording data into text using speech recognition technology and uses ChatGPT to generate a summary that extracts the key points.

[1110] Step 5:

[1111] The server determines the importance and urgency of emails and meetings based on the content of the generated summaries, and assigns priorities to them.

[1112] Step 6:

[1113] The server generates a priority list and sends it to the user's terminal.

[1114] Step 7:

[1115] The terminal receives the list and notifies the user.

[1116] Generate and view a summary report after the holidays

[1117] Step 1:

[1118] The server compiles all collected new emails and summaries generated from the meeting contents into one summary report at the end of the vacation.

[1119] Step 2:

[1120] The server transmits the generated summary report to the user's terminal.

[1121] Step 3:

[1122] The terminal notifies the user of receipt of the summary report.

[1123] Step 4:

[1124] The user checks the summary report on the terminal, which includes a summary and priority of all emails and meetings received during the vacation.

[1125] Step 5:

[1126] Based on the summary report, users can start responding to important emails and meeting details one by one.

[1127] Example 1

[1128] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1129] Employees are expected to be completely free from work during their vacation, while also being able to return to work quickly and efficiently after their vacation. However, traditional systems have limitations in ensuring that important communications or urgent responses are not overlooked during the vacation. Furthermore, dealing with the large amount of unprocessed emails and meeting details after the vacation takes time and effort, making it difficult to resume work smoothly.

[1130] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1131] In this invention, the server includes a means for automatically answering calls during vacation, a means for converting the contents of received calls into text data using voice recognition technology, and a means for analyzing the converted text data using natural language processing technology to determine the urgency of the call. This allows important communications and emergency responses to be made even during vacation, while allowing employees to return to work quickly and efficiently after their vacation.

[1132] "Means for automatically answering calls" refers to the ability of the system to answer calls on behalf of a vacation user and play a pre-defined voice message.

[1133] "Voice recognition technology" is a technology that analyzes voice data and converts it into text data.

[1134] "Text data" is character information converted from voice data using voice recognition technology.

[1135] "Natural language processing technology" is a technology that analyzes text data to understand its content, urgency, and intent.

[1136] The "means of determining urgency" is a function that uses natural language processing technology to analyze text data to identify important information and elements that require urgent action, and notify emergency contacts as necessary.

[1137] "Means for notifying emergency contacts" is a function that sends a notification to pre-set contacts when it is determined based on the analysis results that emergency contact is necessary.

[1138] A "mailbox" is a data storage device for receiving and storing a user's email.

[1139] "Meeting recording data" refers to data that records the audio information of a meeting.

[1140] A "generative AI model" is an artificial intelligence model that learns from large amounts of data to perform tasks such as text generation and summarization.

[1141] The "means of generating summaries" is a function that uses a generative AI model to extract key information from collected email and meeting recording data and provide it in a shortened format.

[1142] The "means for prioritizing" is a function that evaluates the importance and urgency of information based on the generated summary data and determines the order in which it should be processed.

[1143] A "Summary Report" is a report that lists key points and priorities generated from collected and summarized data.

[1144] A "terminal" is a device that allows a user to communicate with a server, and includes smartphones, computers, etc.

[1145] The "means for receiving notification" is a function by which the terminal receives information sent from the server and displays that information to the user.

[1146] This invention is based on a system that consists of three main components: a server, a terminal, and a user. Each component operates as follows:

[1147] server

[1148] The server plays a central role in this system, hosting the AI ​​model and processing data in real time. The server has the following specific functions:

[1149] 1. Automated telephone answering

[1150] The server automatically detects incoming calls from vacation users and plays pre-defined answering messages. Specifically, the server monitors the telephone line signal, and when a call comes in, it selects an appropriate voice file from templates stored in a database and plays it. For example, it could play a voice file that says, "I'm currently on vacation. I'll get back to you later."

[1151] 2. Transcription and analysis of call content

[1152] The server records the audio data of the call and converts it into text data using speech recognition technology, using technologies such as the Google Cloud Speech-to-Text API. The generated text data is then analyzed using natural language processing (NLP) technology to determine the urgency of the call content. The urgency is evaluated based on predefined rules (e.g., the presence of keywords such as "emergency" or "urgent"). For example, if the call content is something like "In the event of a fire," it is determined that an emergency response is required.

[1153] 3. Emergency notification

[1154] Based on the analysis, the server will send a notification to pre-defined emergency contacts if necessary. This notification can be sent in different ways, such as SMS, email, or push notification. For example, it may contain a message such as "A critical security issue has occurred. Your immediate attention is required."

[1155] 4. Email and meeting summaries

[1156] The server periodically checks the user's mailbox and meeting recording data. Specifically, it uses the Gmail API for the mailbox and speech recognition technology for the meeting recording. A generative AI model (e.g., ChatGPT) is used to generate a summary of the collected data. Prompts used include "Please summarize the following email" and "Please summarize the main points of this meeting." For example, a summary such as "The main topic of discussion was a progress report on a new project" can be generated.

[1157] 5. Prioritize

[1158] The server evaluates the importance and urgency of emails and meeting content based on the generated summary data, and assigns priorities to them. This prioritization is based on algorithms such as "urgency of time" and "job titles of the people involved." For example, "customer complaints" might be listed as the top priority.

[1159] 6. Generate and send a summary report

[1160] The server compiles all the summary data into a single summary report at the end of the vacation. This summary report is sent to the user's device via email or push notification. For example, it may contain a list of emails requiring replies under "Important matters during vacation," a summary of new emails, and key points from important meetings.

[1161] Terminal

[1162] The terminal refers to the user's smartphone or computer, and functions as an interface with the server. The specific operation of the terminal is as follows.

[1163] 1. Display phone notifications

[1164] The device receives notifications from the server in real time and displays them to the user. The notification content includes a message such as "AI has responded." For example, a notification may appear on the user's smartphone saying "An automatic response was made because the user is on vacation."

[1165] 2. View the summary report

[1166] The device receives the summary report sent from the server and notifies the user. The report contents can be checked from the notification center or a dedicated app, and important items are highlighted. For example, a notification may be displayed showing "two emails requiring attention" and "four important meetings."

[1167] User

[1168] The user is an employee who uses the system, and can be relieved from work during the vacation and return to work efficiently after the vacation. The specific operation and usage of the user are as follows.

[1169] 1. Vacation Settings

[1170] The user enters the system's settings screen and enters the start and end times of the vacation. The server receives this information and prepares the automatic response mode. For example, the user sets "Vacation start date: October 1, 2023" and "Vacation end date: October 7, 2023."

[1171] 2. Checking during vacation

[1172] While on vacation, users check the notifications displayed on their devices. Notifications with urgent nature will be displayed with special alerts, and users can respond accordingly. For example, if a message appears saying "You have an important notification," users can check the content and respond as necessary.

[1173] 3. Resuming work after holidays

[1174] After the vacation ends, the user checks the summary report on their device. The report summarizes important emails and meeting details, and the user can start addressing them based on priority. For example, the report shows "2 emails needing attention" and "4 important meetings," allowing the user to return to work immediately.

[1175] The system allows users to return to work quickly and efficiently without missing important communications while on vacation.

[1176] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1177] Server Processing Steps

[1178] Step 1: Auto-answer your phone

[1179] The server detects an incoming call. It receives the signal from the telephone line as input and plays a pre-defined answering message. For this, an audio file is selected from a database and played. For example, a voice message such as "I'm currently on vacation. I'll get back to you later" is played.

[1180] Step 2: Transcribe the call

[1181] The server records the audio data of the call. It receives the audio data as input and converts it into text data using speech recognition technology such as the Google Cloud Speech-to-Text API. The content of the call is saved in text format as output. For example, the content "There's a fire" is generated as text data.

[1182] Step 3: Analyzing the call

[1183] The server analyzes the generated text data using natural language processing (NLP) technology. It receives the text data as input and analyzes specific keywords and phrases to determine the urgency. The output is an assessment of the urgency level. For example, if the keyword "fire" is included, the urgency level is determined to be high.

[1184] Step 4: Send emergency notifications

[1185] Based on the analysis results, the server sends notifications to pre-configured emergency contacts as needed. It receives the urgency assessment result as input and sends notifications via SMS, email, etc. The output is a notification sent to the emergency contact. For example, a notification stating, "A critical security issue has occurred. Urgent action is required."

[1186] Step 5: Summarize emails and meetings

[1187] The server periodically checks the user's mailbox and meeting recording data. It receives email data and audio recording data as input and generates a summary using a generative AI model (e.g., ChatGPT). The output is a summary that extracts the key points. Prompts used include "Please summarize the following email" and "Please summarize the main points of this meeting." For example, a summary might be generated that reads, "The main topic was a progress report on a new project."

[1188] Step 6: Prioritize

[1189] The server evaluates the importance and urgency of the generated summary data and assigns priorities to them. It takes the summary data as input and runs a prioritization algorithm (e.g., time urgency, role of the involved parties). As output, a list of priorities is created. For example, "Customer complaints" may be listed as the highest priority.

[1190] Step 7: Generate and submit a summary report

[1191] The server compiles all the summary data into a single summary report when the vacation ends. It receives the generated summary data as input and formats it into a summary report. As output, it generates a summary report that is sent to the user's device at the end of the vacation. For example, it includes a list of emails that need replying as "important matters during vacation," a summary of new emails, and important meeting points.

[1192] Terminal processing steps

[1193] Step 1: View phone notifications

[1194] The device receives notifications from the server in real time and displays them to the user. Notifications from the server are received as input, and an application for displaying them on a smartphone or computer receives them. As output, a notification message is displayed to the user. For example, a notification saying "An automatic response has been sent because I am on vacation" is displayed.

[1195] Step 2: View the summary report

[1196] The device receives the summary report sent from the server and notifies the user. It receives the summary report data as input and displays it in a dedicated app or notification center. As output, the summary report is displayed to the user. For example, the device may be notified of "two emails that need attention" and "four important meetings."

[1197] User operation steps

[1198] Step 1: Vacation Settings

[1199] The user enters the system's settings screen and enters the start and end times of the vacation. As input, the start and end dates of the vacation are entered into the system, and the server receives this information. As output, the server-side settings are updated and the auto-response mode is ready. For example, the user sets "Vacation starts October 1, 2023" and "Vacation ends October 7, 2023."

[1200] Step 2: Vacation Check

[1201] The user checks the notifications displayed on the device while on vacation. As input, the user checks the notifications displayed on the device and takes emergency notices if necessary. As output, the user can take emergency measures. For example, if the message "You have an important notice" is displayed, the user checks the contents and takes action if necessary.

[1202] Step 3: Resuming work after the holidays

[1203] After the vacation ends, the user checks the summary report on their device. The summary report data is taken as input, and a plan is made to prioritize important matters. The output is the ability to quickly return to work based on work priorities. For example, "2 emails needing attention" and "4 important meetings" are displayed, allowing the user to quickly start dealing with the important tasks.

[1204] (Application example 1)

[1205] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1206] Modern business people are faced with the need to respond to emergencies and important tasks even while on vacation. However, answering phone calls, checking emails, and even responding to emergencies while on vacation can significantly reduce the quality of their vacation. This invention aims to solve this problem by providing a system that streamlines tasks that are treated as an extension of work even while users are on vacation, and enables rapid response to emergencies.

[1207] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1208] In this invention, the server includes means for automatically answering calls during vacation, means for converting received phone call content into text and analyzing it, means for determining the urgency of the call from the analyzed content and notifying an emergency contact as necessary, means for the user to monitor the surroundings using a visual device and detect abnormal behavior or an emergency, means for notifying the user of the detected abnormality or emergency by voice and providing instructions on its content, and means for analyzing the converted call content, assessing the urgency, and providing instructions on how to respond. This allows the user to effectively and efficiently handle work and emergencies even while on vacation.

[1209] A "system for improving work efficiency during vacation" is a system that automatically processes work on behalf of a user who is on vacation, improving efficiency after the user returns to work.

[1210] "Auto-answer" is a feature that allows the system to play a pre-defined answering message, eliminating the need for the user to answer the phone.

[1211] "Texting" is the process of converting audio data into text data, and is done using voice recognition technology.

[1212] "Analysis" is the process of using natural language processing techniques on text data to determine the intent and urgency of the content.

[1213] "Means for notifying emergency contacts" refers to a method for sending a notification to a designated emergency contact based on the results of the analyzed data.

[1214] A "visual device" is a device worn by a user that allows the user to obtain information visually, and is used as smart glasses or a head-mounted display.

[1215] The "means for detecting abnormal behavior and emergency situations" is a system that uses cameras and sensors in the visual device to monitor the user's surroundings and recognize abnormal activities and emergency situations.

[1216] "Voice notification means" is a function in which, when an emergency occurs, the system explains the situation to the user by voice and gives necessary instructions.

[1217] A "summary report" is a report that summarizes the collected data and comprehensively presents the information the user needs after completing their vacation.

[1218] The embodiment of the present invention is mainly composed of three components: a server, a terminal, and a user. This embodiment allows users to be freed from work during vacations while enabling a quick and appropriate response to emergencies.

[1219] server

[1220] The server plays a central role in this system, hosting the AI ​​model and processing data in real time. The server has the following specific functions:

[1221] 1. Auto-answer: Automatically detects incoming calls when the user is on vacation and plays a pre-defined answering message.

[1222] 2. Converting call content into text: Converting received phone calls into text data using voice recognition technology. This process is achieved using Google Speech API, etc.

[1223] 3. NLP analysis: Analyze text data using natural language processing technology (e.g., spaCy or NLTK) to determine the urgency of the call content.

[1224] 4. Emergency Notification: Based on the analysis results, notifications will be sent to pre-defined emergency contacts as necessary.

[1225] 5. Email and meeting summaries: Collect new emails and meeting contents from the user's mailbox and meeting recording data, summarize the collected data using a generative AI model (e.g., ChatGPT), and generate the results as a summary report.

[1226] 6. Prioritization: Based on the generated summary data, the importance and urgency are evaluated and prioritized.

[1227] 7. Generate and send summary report: At the end of the vacation, a summary of all collected data is compiled into a single summary report and sent to the user's device.

[1228] Terminal

[1229] The terminal refers to the user's smartphone or computer, and functions as an interface with the server. The specific operation of the terminal is as follows.

[1230] 1. Displaying phone call notification: Receives a notification from the server and displays a message to the user such as "AI has answered." If necessary, details of the call will also be displayed.

[1231] 2. Displaying summary reports: Receives summary reports sent from the server and notifies the user. The reports include summaries and priorities of emails and meetings received during vacation.

[1232] User

[1233] Users are staff who use the system, and there are specific operations and usage methods as follows.

[1234] 1. Vacation Settings: The user enters the start and end time of the vacation into the system, which prepares the server to activate the auto-answer mode.

[1235] 2. Checking while on vacation: The user checks the notifications displayed on the device while on vacation. If there is an emergency, the user receives a detailed notification from the server and takes appropriate action.

[1236] 3. Resuming work after vacation: After returning from vacation, users can check the summary report on their device. The report summarizes important emails and meeting details, allowing them to quickly start working on important tasks.

[1237] Specific examples

[1238] For example, if a user sets a one-week vacation, the server will automatically answer all incoming calls during the vacation period, converting the calls into text and analyzing them. If the analysis determines that an emergency response is required, the server will immediately send a notification to the emergency contact. At the same time, the server will regularly check the user's mailbox and meeting recordings, and generate summaries of new emails and meetings. The generated summaries are compiled into a summary report with prioritized information and sent to the user's device at the end of the vacation. When the user checks their device after the vacation, a summary report with all the necessary information will be displayed, allowing them to quickly start responding to important tasks.

[1239] Prompt Sentence Examples

[1240] "A suspicious person is detected while the user is on vacation. Real-time video and audio analysis is performed to determine the urgency. If the urgency is high, a notification is sent and response instructions are displayed."

[1241] In this way, the system of the present invention provides a concrete means for enabling users to enjoy their vacation while also returning to work efficiently after the vacation.

[1242] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1243] Step 1:

[1244] Vacation Settings

[1245] The user inputs the start and end times of the vacation into the system, which prepares the server to activate the automatic response mode. The start date and time and end date and time of the vacation are specified as input data, and the server sets the activation of the automatic response mode based on that information. Specifically, the user inputs the vacation information using the terminal interface, and the server saves the information in the database.

[1246] Step 2:

[1247] Automatic answering of incoming calls

[1248] If a call comes in while you're on vacation, the server automatically detects the call and plays a pre-defined answering message. It takes the received call's voice data as input and plays the answering message as output. Specifically, the server detects an incoming call and plays the answering message on an audio output device.

[1249] Step 3:

[1250] Transcription of call contents

[1251] The server converts the received voice data into text. In this step, it uses voice recognition technology such as Google Speech API to convert the input voice file into text data. The text data is output. Specifically, the server sends the voice data to the API and stores the converted text in a database.

[1252] Step 4:

[1253] NLP analysis

[1254] The server analyzes the text data using natural language processing (NLP) technology and evaluates its urgency. The input text is passed through an NLP module (e.g., spaCy or NLTK), and the data with an urgency rating is output. Specifically, if the analysis results indicate a high urgency, that information is reflected in the emergency notification settings.

[1255] Step 5:

[1256] emergency notification

[1257] Based on the analysis results, the server sends a notification to the pre-defined emergency contacts. The server uses the urgency assessment data as input data and sends a notification as output. Specifically, the server sends the notification to the emergency contacts via phone or message notification services.

[1258] Step 6:

[1259] Email and meeting summary generation

[1260] The server collects new emails and meeting contents from the user's mailbox and meeting recording data, and generates summaries using a generative AI model (e.g., ChatGPT). Emails and meeting records are used as input data, and summary text data is output. Specifically, the server periodically checks the mailbox and meeting recording data, and generates summaries by inputting the data into the generative AI model.

[1261] Step 7:

[1262] Prioritization

[1263] The server evaluates the importance and urgency of the generated summary data and assigns priorities to them. It uses the summary text as input data and outputs prioritized schedule data. Specifically, the server assigns a rank to each summary using an evaluation algorithm.

[1264] Step 8:

[1265] Generate and send summary reports

[1266] The server generates a summary report and sends it to the user's device. Prioritized summary data is used as input data, and a document constructed as a summary report is generated and sent as output. Specifically, the server sends the generated report to the user by email or other means.

[1267] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1268] The system of this invention consists of three main components: a server, a terminal, a user, and an emotion engine. Each component operates as follows:

[1269] server

[1270] The server plays a central role in the system, hosting the AI ​​model and emotion engine, and performing real-time data processing. The server has the following specific functions:

[1271] 1. Automated telephone answering

[1272] The server has the function of automatically detecting an incoming call when the user is on vacation and playing a pre-defined answering message.

[1273] 2. Transcription and analysis of call content

[1274] The server converts the received call content into text data using voice recognition technology, and then analyzes the text data using natural language processing (NLP) technology to determine the urgency of the call content.

[1275] 3. Collaboration with emotion engine

[1276] The server uses an emotion engine to analyze the user's emotions from their voice data. Based on the results of the emotion analysis, the server combines it with regular text analysis to more accurately determine the urgency of the message.

[1277] 4. Emergency notification

[1278] The server will send notifications to pre-configured emergency contacts as needed.

[1279] 5. Email and meeting summaries

[1280] The server collects new emails and meeting contents from the user's mailbox and meeting recording data, summarizes the collected data using a generation AI such as ChatGPT, and generates a summary report.

[1281] 6. Prioritize

[1282] The server takes into account the emotional data of emails and meeting contents analyzed using an emotion engine, and prioritizes them according to importance and urgency based on the summary data.

[1283] 7. Generate and send a summary report

[1284] The server compiles all new emails and summaries of meetings, including emotion data, into a single summary report at the end of the vacation, and sends it to the user's terminal.

[1285] Terminal

[1286] The terminal refers to the user's smartphone or computer, and acts as an interface with the server. The specific operations of the terminal are as follows:

[1287] 1. Display phone notifications

[1288] The device receives a notification from the server and displays a message to the user such as "AI has responded" along with the results of the emotion analysis. If necessary, details of the call content will also be displayed.

[1289] 2. View the summary report

[1290] The terminal receives a summary report sent from the server and notifies the user. The report includes a summary and priority of emails and meetings received during the vacation. It also displays the results of sentiment analysis.

[1291] User

[1292] The user is an employee who uses the system, and can be relieved from work during the vacation and return to work efficiently after the vacation. The specific operations and usage of the user are as follows:

[1293] 1. Vacation Settings

[1294] The user enters the start and end times of their vacation into the system, which prepares the server to activate the auto-answer mode.

[1295] 2. Checking during vacation

[1296] The user checks the notifications displayed on the device while on vacation. If there is an emergency, the user receives a detailed notification from the server and takes appropriate action.

[1297] 3. Resuming work after holidays

[1298] After the vacation, users can check a summary report on their device, which summarizes important emails and meeting details, allowing them to return to work smoothly based on their priorities.

[1299] Specific examples

[1300] For example, if a user sets a one-week vacation, the server will automatically respond to all calls during the vacation period, converting the calls into text and analyzing them. At the same time, the server uses an emotion engine to analyze the user's emotions from the voice data. For example, if the emotion analysis indicates a high level of urgency, the server will immediately send a notification to emergency contacts.

[1301] The server also periodically checks the user's mailbox and meeting recordings to generate summaries of new emails and meetings. Using an emotion engine, it analyzes the emotions of the collected data and prioritizes them based on importance and urgency. At the end of the vacation, a summary report containing the prioritized summaries is sent to the user's device.

[1302] When users check their device after their vacation, they will see a summary report that summarizes all the information they need, along with additional information based on sentiment analysis, allowing them to quickly start working on the most important tasks.

[1303] In this way, the system of the present invention allows users to enjoy their vacation while efficiently returning to work after their vacation. Furthermore, the emotion engine enables more accurate determination of urgency and prioritization of information.

[1304] The processing flow will be explained below.

[1305] Answering phone calls during holidays

[1306] Step 1:

[1307] The server receives the user's vacation setting information and enables the telephone auto-answer mode during the vacation period.

[1308] Step 2:

[1309] The server detects an incoming call while on vacation.

[1310] Step 3:

[1311] The server plays a pre-programmed answering message and converts the call into text using voice recognition technology.

[1312] Step 4:

[1313] The server analyzes the text of the call using natural language processing (NLP) technology to determine the urgency of the call.

[1314] Step 5:

[1315] The server uses an emotion engine to analyze emotions from the voice data, for example, detecting changes in tone and speed of the voice and using these as indicators of urgency.

[1316] Step 6:

[1317] The server combines the results of text analysis and sentiment analysis to determine the final urgency.

[1318] Step 7:

[1319] The server will send notifications to pre-configured emergency contacts as needed.

[1320] Step 8:

[1321] The server generates a summary of the response content and emotion analysis results and sends it to the user's terminal.

[1322] Step 9:

[1323] The device will notify the user that "AI has answered," along with the results of the emotion analysis. If necessary, it will also display details of the call.

[1324] Vacation email and meeting summaries

[1325] Step 1:

[1326] The server periodically checks the user's mailbox and collects new emails.

[1327] Step 2:

[1328] The server checks the user's calendar and meeting recording data to collect the corresponding meeting details.

[1329] Step 3:

[1330] The server analyzes the contents of newly arrived emails collected using ChatGPT and generates summaries.

[1331] Step 4:

[1332] The server converts the meeting recording data into text using speech recognition technology and uses ChatGPT to generate a summary that extracts the key points.

[1333] Step 5:

[1334] The server uses an emotion engine to analyze the user's emotions contained in emails and meeting content, for example, by assigning emotion labels to comments in emails and meetings.

[1335] Step 6:

[1336] The server determines the importance and urgency of emails and meetings based on the content of the generated summaries, and prioritizes them taking into account the results of sentiment analysis.

[1337] Step 7:

[1338] The server generates a priority list and sends it to the user's terminal.

[1339] Step 8:

[1340] The terminal receives the list and notifies the user.

[1341] Generate and view a summary report after the holidays

[1342] Step 1:

[1343] At the end of the vacation, the server compiles all collected new emails and summaries generated from the meeting content into a single summary report, incorporating sentiment analysis results.

[1344] Step 2:

[1345] The server transmits the generated summary report to the user's terminal.

[1346] Step 3:

[1347] The terminal notifies the user of receipt of the summary report.

[1348] Step 4:

[1349] The user can view a summary report on their device, which includes a summary and prioritization of all emails and meetings received during their vacation, as well as the results of sentiment analysis.

[1350] Step 5:

[1351] Based on the summary report, users can start responding to important emails and meeting details one by one.

[1352] Example 2

[1353] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1354] In today's business environment, employees are expected to respond efficiently to important phone calls, emails, and meeting details while on vacation. However, if work is completely neglected while on vacation, emergencies and important matters may be overlooked, resulting in a huge amount of work to be handled once the vacation returns. Furthermore, conventional systems do not perform sentiment analysis, making it difficult to determine urgency and insufficient to prioritize information. A new system is needed to solve these issues.

[1355] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1356] In this invention, the server includes: means for automatically answering calls during vacation; means for converting received call content into text using speech recognition technology; means for analyzing text data using natural language processing technology; means for analyzing emotions from the voice data; means for determining urgency based on the analyzed call content and emotion data and notifying emergency contacts as necessary; means for collecting new emails and meeting content from the user's mailbox and conference recording data; means for generating summaries of the collected data using a generative AI model; means for prioritizing from the summaries; means for generating a summary report including email and meeting summaries at the end of the vacation; and means for sending the generated summary report to the user. This allows for more efficient work during vacation and a smooth resumption of work after the vacation. Furthermore, using emotion analysis allows for more accurate determination of urgency and effective prioritization of information.

[1357] A "means for automatically answering vacation calls" is hardware or software that has the function of automatically detecting calls received while the user is on vacation and playing a pre-defined answering message.

[1358] "Means for converting received telephone calls into text using voice recognition technology" refers to hardware or software that uses voice recognition technology to convert the voice data of received telephone calls into text data.

[1359] "Means for analyzing text data using natural language processing technology" refers to hardware or software that uses natural language processing technology to analyze the content of text data obtained by speech recognition and extract useful information from it.

[1360] "Means for analyzing emotions from voice data" refers to hardware or software that utilizes emotion analysis technology to analyze the emotional state of a user or the other party from voice data or the text data converted from the voice data.

[1361] "Means for determining the urgency of a call based on the analyzed content and emotional data, and notifying emergency contacts as necessary" refers to hardware or software that evaluates the urgency of the content of a call based on the results of natural language processing and emotional analysis, and sends a notification to emergency contacts in the event of an emergency.

[1362] "Means for collecting new emails and conference contents from the user's mailbox and conference recording data" refers to hardware or software that periodically checks the user's mailbox and conference audio data to collect newly received emails and conference contents.

[1363] "Means for generating summaries of collected data using a generative AI model" refers to hardware or software capable of inputting collected email and meeting data into an AI model and generating summaries of the data.

[1364] The "means for prioritizing from summaries" refers to hardware or software capable of prioritizing the generated summary data based on importance and urgency.

[1365] "Means for generating a summary report including email and meeting summaries at the end of vacation" refers to hardware or software that compiles the data collected and summarized during the vacation into a single report at the end of the vacation.

[1366] The "means for transmitting the generated summary report to the user" is hardware or software for transmitting the generated summary report to the user's terminal after the vacation ends.

[1367] The "means for setting the start and end times of vacation" refers to an interface and system for a user to input the start and end times of vacation and record the schedule.

[1368] "Means for preparing the server to enable the auto-reply mode at the vacation start time" refers to programs and processes on the server for enabling the auto-reply mode when the vacation begins.

[1369] The "means for displaying vacation notices on a terminal" refers to hardware or software that receives notices from the server and displays them on the user's terminal.

[1370] The "means for reassessing urgency based on text data and emotion data" refers to hardware or software that reassess data obtained through natural language processing and emotion analysis to improve the accuracy of urgency.

[1371] "Means of sending notifications to emergency contacts via SMS or email when necessary" refers to a system that sends notifications to appropriate emergency contacts via SMS, email, etc. when a high level of urgency is deemed to exist.

[1372] "Means for inputting the collected data into a generative AI model to generate a summary and then integrating it with the sentiment analysis results" is hardware or software for inputting the collected data into an AI model and integrating the results with the sentiment analysis results.

[1373] The "means for using an algorithm to prioritize the generated summary data" refers to a system that applies a specific algorithm to set priorities for the summary data.

[1374] The system of this invention consists of three main components: a server, a terminal, and a user, as well as an emotion engine. Each component operates as follows:

[1375] server

[1376] The server plays a central role in this system, hosting various data processing and emotion engines. The specific hardware and software configuration is as follows:

[1377] 1. Automated telephone answering

[1378] The server uses an automated answering system such as Asterisk to automatically answer calls received while the user is on vacation. For example, when a user sets the start time of their vacation, the server launches Asterisk and plays a pre-recorded message.

[1379] 2. Transcription of call contents

[1380] The server uses the Google Cloud Speech-to-Text API to convert the call into text data. For example, when the call ends, the server saves the recording and calls the Google Cloud API to convert the voice data into text.

[1381] 3. Text Data Analysis

[1382] The server uses Python NLP libraries (such as NLTK and spaCy) to analyze the text data and determine the urgency of the call content. Specifically, it extracts important keywords and evaluates the urgency.

[1383] 4. Analysis by Emotion Engine

[1384] The server uses Azure Text Analytics for Sentiment and IBM Watson Tone Analyzer to analyze emotions from voice data, and based on the obtained emotional data, it can assess the urgency of the message with even greater accuracy.

[1385] 5. Sending emergency notifications

[1386] The server uses Twilio or SendGrid to send notifications to emergency contacts when necessary, for example, using Twilio to send SMS notifications for calls deemed to be urgent.

[1387] 6. Collect email and meeting data

[1388] The server periodically collects user mailbox and meeting recording data using the Gmail API and Microsoft Graph API. For example, it periodically calls the API every day to retrieve new emails and meeting data.

[1389] 7. Summarization using generative AI models

[1390] The server inputs the collected data into a generative AI model such as ChatGPT to generate a summary. For example, email data obtained from an API is passed to the generative AI model to obtain the summary result.

[1391] 8. Data Prioritization

[1392] The server prioritizes the summary data based on the importance and urgency of the data, based on the analysis results of the emotion engine. For example, data that is deemed "urgent" by emotion analysis has a high priority.

[1393] 9. Generate and send summary reports

[1394] The server compiles all the summary data into one summary report at the end of the vacation and sends it to the user's terminal using the SMTP protocol. For example, the server collects all the data on the day before the vacation ends and compiles it to generate a report.

[1395] Terminal

[1396] The terminal refers to the user's smartphone or computer, and functions as an interface with the server. The specific operation is as follows.

[1397] 1. Display phone notifications

[1398] The device receives a notification from the server and displays the emotion analysis results along with a message such as "AI has responded," prompting the user to confirm via the notification system.

[1399] 2. View the summary report

[1400] The device receives the summary report sent from the server and notifies the user, for example, by displaying the report through an email app or a dedicated app.

[1401] User

[1402] Users are employees who use the system, and the specific operations and usage are as follows.

[1403] 1. Vacation Settings

[1404] The user enters the start and end time of the vacation through the app, which causes the server to activate the auto-reply mode.

[1405] 2. Checking during vacation

[1406] The user checks the notifications displayed on the device while on vacation. If there is an emergency, the user receives a detailed notification from the server and takes appropriate action.

[1407] 3. Resuming work after holidays

[1408] After the vacation ends, the user checks the summary report on the terminal and starts working on important tasks based on the priority.

[1409] Specific examples

[1410] For example, if a user schedules a one-week vacation, the server will automatically answer all calls during that vacation period. When a call comes in, Asterisk detects it and plays a pre-recorded message. The call is recorded and converted to text data using Google Cloud Speech-to-Text. Next, the text data is analyzed using a Python NLP library to extract important keywords and evaluate the urgency. At the same time, Azure Text Analytics for Sentiment performs sentiment analysis of the call. If the call is deemed urgent, a notification is sent to emergency contacts using Twilio.

[1411] The server also periodically collects user mailbox and meeting recording data and generates summaries using generative AI models such as ChatGPT. The summary data is then prioritized based on the results of sentiment analysis. At the end of the vacation, a summary report containing all summaries is generated and sent to the user's device via SMTP. Users can then review the summary report after their vacation and quickly begin work on high-priority tasks.

[1412] Prompt Sentence Examples

[1413] "Please analyze my calls during my vacation and determine their urgency. Please also analyze the sentiment of the calls and send me emergency notifications if necessary. Also, please summarize emails and meetings and send me a summary report at the end."

[1414] In this way, the system of the present invention allows users to enjoy their vacation while efficiently returning to work after their vacation. The emotion engine also enables more accurate determination of urgency and prioritization of information.

[1415] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1416] Step 1: Set up your vacation time

[1417] Input: The user opens the dedicated application and enters the start and end dates and times of the vacation.

[1418] How it works: The user enters start and end dates and times into a vacation configuration form, the application validates them, and once validation is complete, the configuration information is sent to the server.

[1419] Output: The server receives the user's vacation settings and prepares to enable auto-answer mode at the vacation start time.

[1420] Step 2: Auto-answer your phone calls

[1421] Input: When the vacation begins, a call comes in to the user's phone.

[1422] How it works: The server uses Asterisk to detect incoming calls and play a pre-recorded greeting, such as "I'm on vacation right now."

[1423] Output: When a call comes in, an automatic answering message is played.

[1424] Step 3: Record and transcribe the call

[1425] Input: Audio data from the received call.

[1426] How it works: When the call ends, the call recording is saved to the server, which then calls the Google Cloud Speech-to-Text API to convert the audio data into text.

[1427] Output: The converted text data.

[1428] Step 4: Analyzing the text data

[1429] Input: The text data obtained in step 3.

[1430] How it works: The server uses Python NLP libraries (NLTK and spaCy) to analyze the text data and determine the urgency of the call. Specifically, it extracts important keywords (e.g., accident, emergency) and evaluates the urgency.

[1431] Output: Urgency assessment data of the call content.

[1432] Step 5: Analysis by Emotion Engine

[1433] Input: Text data from step 3.

[1434] How it works: The server uses Azure Text Analytics for Sentiment and IBM Watson Tone Analyzer to analyze emotions from the voice data. Based on the obtained emotional data, it can more accurately assess the urgency of the call.

[1435] Output: Sentiment analysis data of the call content.

[1436] Step 6: Send emergency notifications

[1437] Input: Urgency assessment data and sentiment analysis data from Step 4 and Step 5.

[1438] What it does: If the server determines there is an emergency, it will use Twilio or SendGrid to send a notification to emergency contacts, for example, calling the Twilio API to send an SMS notification.

[1439] Output: Notification message to emergency contacts.

[1440] Step 7: Collect email and meeting data

[1441] Input: User mailbox and meeting recording data.

[1442] How it works: The server periodically collects user mailbox and meeting recording data using the Gmail API or Microsoft Graph API. For example, it periodically calls the API every day to retrieve new emails and meeting data.

[1443] Output: Collected email and meeting data.

[1444] Step 8: Summarizing with a generative AI model

[1445] Input: Email and meeting data collected in step 7.

[1446] How it works: The server inputs the collected data into a generative AI model such as ChatGPT to generate a summary. For example, email data obtained from an API is passed to the generative AI model to obtain the summary result.

[1447] Output: The generated summary data.

[1448] Step 9: Prioritize your data

[1449] Input: Summary data from step 8 and sentiment analysis data from step 5.

[1450] How it works: The server prioritizes summary data based on importance and urgency based on the analysis results of the emotion engine. Specifically, data that is deemed "urgent" by emotion analysis is given a high priority.

[1451] Output: Prioritized summary data.

[1452] Step 10: Generate and send a summary report

[1453] Input: Prioritized summary data from step 9.

[1454] How it works: At the end of the vacation, the server compiles all the summary data into one summary report and sends it to the user's terminal using the SMTP protocol. For example, on the day before the vacation ends, the server collects all the data and generates the report.

[1455] Output: Summary report sent to the user.

[1456] Step 11: Check your vacation notices

[1457] Input: Notification data sent by the server.

[1458] How it works: The device receives a notification from the server and displays the emotion analysis results along with a message such as "AI has responded."

[1459] Output: The notification message displayed to the user.

[1460] Step 12: Resuming work after the holidays

[1461] Input: Summary report submitted in step 10.

[1462] Action: The user reviews the summary report and starts working on the most important tasks based on priority.

[1463] Output: Resume work starting with high priority tasks.

[1464] (Application example 2)

[1465] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1466] Errors and abnormal situations occurring during robotic operations in factories have a significant impact on work efficiency and productivity. Conventional robot monitoring systems often delay the detection and response of errors, preventing serious problems from being resolved quickly. Furthermore, analysis of error content and assessment of urgency are insufficient, preventing managers from being properly notified. This creates the risk of factory operations coming to a halt. Furthermore, because management reports for abnormal situations are not generated, it is difficult for managers to grasp the situation on a day-to-day basis. Another issue is that prioritization based on the urgency of errors is not performed, preventing appropriate responses.

[1467] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1468] In this invention, the server includes means for monitoring the working status of the robot, means for automatically generating a response message when the robot detects an error, means for converting the detected error into text and analyzing it, means for determining the urgency of the analyzed error, means for notifying a manager as necessary, means for recording the details and urgency of the error and generating a management report by summarizing daily abnormalities and errors, means for transmitting the generated management report, means for performing emotion analysis of the error content and prioritizing the content based on the urgency, and means for notifying the manager of events with a higher urgency first based on the priority. This enables a quick and appropriate response to errors and abnormalities, thereby improving the efficiency and safety of factory operations.

[1469] "Means for monitoring the working status of robots" refers to devices or systems that monitor the current operations and sensor information of robots operating in factories in real time and acquire their status.

[1470] "Means for automatically generating a response message" refers to a device or software that, when the robot detects an error, automatically generates a pre-set message based on the content of the error and instructs the robot to take an appropriate action.

[1471] "Means for converting detected error content into text and analyzing it" refers to a device or algorithm that converts the audio or log data of an error detected by a robot into text using natural language processing technology and analyzes that text.

[1472] The "means for determining the urgency of the analyzed error content" refers to a device or system that evaluates the severity and scope of the error based on the textualized and analyzed error information, and determines the urgency of the error.

[1473] The "means for notifying the administrator as necessary" refers to a device or service that promptly notifies the administrator of the details of an emergency error or abnormal situation when it occurs.

[1474] "Means for recording details and urgency of errors, summarizing daily abnormalities and errors, and generating management reports" refers to devices or software that store records of robot errors and abnormalities, compile and summarize daily data, and generate management reports that can be easily understood by managers.

[1475] The "means for transmitting the generated management report" refers to a device or system that transmits the management report to the administrator using email or a notification service.

[1476] "Means for performing emotional analysis of the error content and prioritizing the content based on its urgency" refers to a device or algorithm that evaluates the error content using emotional analysis technology, determines the urgency of the error based on the results, and sets priorities for response.

[1477] "Means for communicating urgent events to the administrator first based on priority" refers to devices or services that notify the administrator of urgent errors or abnormal situations on a priority basis, encouraging a rapid response.

[1478] To implement this invention, a system based on the following configuration and procedures is required: The system is mainly composed of three main components: a server, a robot, and an administrator terminal.

[1479] server

[1480] The server is the core of the system, and performs real-time data processing and analysis. Its specific functions are as follows:

[1481] 1. Monitoring the robot's working status

[1482] The server collects data from sensors and cameras installed on robots in the factory and monitors it in real time. This requires sensors and cameras to be installed on each robot. IoT technology is also used as the communication protocol for data collection.

[1483] 2. Error detection and automatic response

[1484] If the robot detects an error, the server immediately generates and sends a response message based on a pre-defined message. To achieve this, it leverages an AI model (e.g., ChatGPT).

[1485] 3. Error text and analysis

[1486] The detected error content is converted into text using speech recognition technology (e.g., Google Cloud Speech-to-Text) and then analyzed using NLP technology (e.g., Hugging Face Transformers).

[1487] 4. Urgency assessment and notification

[1488] The severity of the analyzed error is assessed using a sentiment analysis engine (e.g., IBM Watson Tone Analyzer), and if the severity is high, the administrator is notified using a notification system (e.g., Firebase Cloud Messaging or Twilio).

[1489] 5. Generate and send management reports

[1490] The server records the details and severity of the errors, and generates a management report summarizing the daily abnormalities and errors. The report is then periodically sent to the administrator.

[1491] robot

[1492] The robots operating in the factory are equipped with sensors and cameras and are responsible for sharing data with the server. If a robot detects an error, it immediately sends the data to the server.

[1493] Administrator terminal

[1494] The administrator's terminal receives and displays notifications and management reports sent from the server, allowing the administrator to quickly understand the situation and take necessary measures.

[1495] Specific examples

[1496] For example, if a factory robot detects an error on a production line, the sensor installed on the robot immediately sends the data to a server. The server uses NLP technology and a sentiment analysis engine to analyze the error and determine its level of urgency. If the level is high, it notifies the administrator using Firebase Cloud Messaging or Twilio with "Error code: 1234, Urgency: High." The server also aggregates daily error data, generates a management report, and sends it to the administrator's terminal.

[1497] An example prompt is:

[1498] "Please convert the error data collected from the robots in the factory into text and perform sentiment analysis to determine the urgency. If the urgency is high, please notify the administrator with 'Error Code: XXXX, Urgency: High' and generate a summary report."

[1499] This enables quick and appropriate responses to errors and abnormal situations, improving the efficiency and safety of factory operations.

[1500] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1501] Step 1:

[1502] The robot detects the error.

[1503] If the robot's onboard sensors or cameras detect an error or abnormality during operation, the information is immediately sent to the server, where the robot generates an error code, detailed sensor data, and a specific error message.

[1504] Input: Error data from sensors and cameras

[1505] Output: Error information sent to the server

[1506] Step 2:

[1507] The error information received by the server is converted to text.

[1508] The server converts the received error information into text data using speech recognition technology (e.g., Google Cloud Speech-to-Text). Here, sensor and camera data and error messages are recorded as text.

[1509] Input: Error information received from the robot

[1510] Output: Textual error data

[1511] Step 3:

[1512] The server parses the text data.

[1513] The server analyzes the textual error data using natural language processing (NLP) technology (e.g., Hugging Face Transformers) to identify the type and cause of the error. This analysis provides a detailed understanding of the error content.

[1514] Input: Textual error data

[1515] Output: Parsed error data

[1516] Step 4:

[1517] The server determines the urgency of the error.

[1518] Based on the analyzed error data, the server uses a sentiment analysis engine (e.g., IBM Watson Tone Analyzer) to evaluate the urgency of the error, which determines the severity of the error's impact on factory operations.

[1519] Input: Parsed error data

[1520] Output: Error data with urgency rating

[1521] Step 5:

[1522] The server notifies the administrator of any errors that are of high urgency.

[1523] If the error is deemed urgent, the server will use a notification system (e.g., Firebase Cloud Messaging or Twilio) to notify the administrator of the error details and urgency, including the error code, error message, and urgency.

[1524] Input: Error data with urgency rating

[1525] Output: Notification sent to administrator

[1526] Step 6:

[1527] The server records error data and summarizes daily anomalies and errors.

[1528] All error data and abnormal events are logged on a server, and the data is aggregated and summarized daily to generate management reports, including the frequency and urgency of errors.

[1529] Input: Error data with urgency rating

[1530] Output: Management Report

[1531] Step 7:

[1532] The server generates a management report and sends it to the administrator's terminal.

[1533] The generated management report is sent to the administrator's terminal on a regular basis, and the administrator can check it to understand any abnormalities that may have occurred.

[1534] Input: Management Report

[1535] Output: Management report sent to administrator

[1536] This enables quick and appropriate responses to errors and abnormal situations, improving the efficiency and safety of factory operations.

[1537] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1538] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1539] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1540] [Fourth embodiment]

[1541] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1542] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1543] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1544] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1545] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1546] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1547] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1548] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1549] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1550] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1551] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1552] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1553] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1554] The system of this invention consists of three main components: a server, a terminal, and a user. Each component operates as follows:

[1555] server

[1556] The server plays a central role in the system, hosting the AI ​​model and processing data in real time. The server has the following specific functions:

[1557] 1. Automated telephone answering

[1558] The server has the function of automatically detecting incoming calls when the user is on vacation and playing a pre-defined answering message, so that the user does not have to answer the phone directly while on vacation.

[1559] 2. Transcription and analysis of call content

[1560] The server converts the received call content into text data using voice recognition technology, and then analyzes the text data using natural language processing (NLP) technology to determine the urgency of the call content.

[1561] 3. Emergency notification

[1562] Based on the analysis results, the server will send notifications to pre-defined emergency contacts as necessary, ensuring that users only respond in truly urgent cases.

[1563] 4. Email and meeting summaries

[1564] The server collects new emails and meeting contents from the user's mailbox and meeting recording data, summarizes the collected data using a generation AI such as ChatGPT, and generates a summary report.

[1565] 5. Prioritize

[1566] The server evaluates the importance and urgency of the requests based on the generated summary data and assigns priorities to them, allowing users to return to work efficiently after their recovery.

[1567] 6. Generate and send a summary report

[1568] The server summarizes all collected data into a single summary report at the end of the vacation and sends it to the user's terminal.

[1569] Terminal

[1570] The terminal refers to the user's smartphone or computer, and acts as an interface with the server. The specific operations of the terminal are as follows:

[1571] 1. Display phone notifications

[1572] The device receives a notification from the server and displays a message to the user such as "AI has answered." If necessary, details of the call will also be displayed.

[1573] 2. View the summary report

[1574] The terminal receives the summary report sent from the server and notifies the user, which includes a summary and priority of emails and meetings received during the vacation.

[1575] User

[1576] The user is an employee who uses the system, and can be relieved from work during the vacation and return to work efficiently after the vacation. The specific operations and usage of the user are as follows:

[1577] 1. Vacation Settings

[1578] The user enters the start and end times of their vacation into the system, which prepares the server to activate the auto-answer mode.

[1579] 2. Checking during vacation

[1580] The user checks the notifications displayed on the device while on vacation. If there is an emergency, the user receives a detailed notification from the server and takes appropriate action.

[1581] 3. Resuming work after holidays

[1582] After the vacation, users can check a summary report on their device, which summarizes important emails and meeting details, allowing them to return to work smoothly based on their priorities.

[1583] Specific examples

[1584] For example, if a user sets a one-week vacation, the server will automatically answer all calls during that time, convert the calls into text, and analyze them. If the analysis determines that an emergency response is required, the server will immediately send a notification to the emergency contact.

[1585] At the same time, the server periodically checks the user's mailbox and conference recordings, generates summaries of new emails and conferences, and then prioritizes them into a summary report that is sent to the user's device at the end of the vacation.

[1586] When a user checks their device after a vacation, a summary report summarizing all the necessary information is displayed, allowing them to quickly start working on important tasks.

[1587] In this way, the system of the present invention allows users to maximize their vacation time while returning to work efficiently after their vacation.

[1588] The processing flow will be explained below.

[1589] Answering phone calls during holidays

[1590] Step 1:

[1591] The server receives the user's vacation setting information and enables the telephone auto-answer mode during the vacation period.

[1592] Step 2:

[1593] The server detects an incoming call while on vacation.

[1594] Step 3:

[1595] The server plays a pre-programmed answering message and converts the call into text using voice recognition technology.

[1596] Step 4:

[1597] The server analyzes the text of the call using natural language processing (NLP) technology to determine the urgency of the content.

[1598] Step 5:

[1599] The server will send notifications to pre-configured emergency contacts as needed.

[1600] Step 6:

[1601] The server generates a summary of the response content and sends it to the user's terminal.

[1602] Step 7:

[1603] The device will notify the user that "AI has responded" and, if necessary, display details of the response.

[1604] Vacation email and meeting summaries

[1605] Step 1:

[1606] The server periodically checks the user's mailbox and collects new emails.

[1607] Step 2:

[1608] The server checks the user's calendar and meeting recording data to collect the corresponding meeting details.

[1609] Step 3:

[1610] The server uses ChatGPT to analyze the contents of the collected new emails and generate summaries.

[1611] Step 4:

[1612] The server converts the meeting recording data into text using speech recognition technology and uses ChatGPT to generate a summary that extracts the key points.

[1613] Step 5:

[1614] The server determines the importance and urgency of emails and meetings based on the content of the generated summaries, and assigns priorities to them.

[1615] Step 6:

[1616] The server generates a priority list and sends it to the user's terminal.

[1617] Step 7:

[1618] The terminal receives the list and notifies the user.

[1619] Generate and view a summary report after the holidays

[1620] Step 1:

[1621] The server compiles all collected new emails and summaries generated from the meeting contents into one summary report at the end of the vacation.

[1622] Step 2:

[1623] The server transmits the generated summary report to the user's terminal.

[1624] Step 3:

[1625] The terminal notifies the user of receipt of the summary report.

[1626] Step 4:

[1627] The user checks the summary report on the terminal, which includes a summary and priority of all emails and meetings received during the vacation.

[1628] Step 5:

[1629] Based on the summary report, users can start responding to important emails and meeting details one by one.

[1630] Example 1

[1631] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1632] Employees are expected to be completely free from work during their vacation, while also being able to return to work quickly and efficiently after their vacation. However, traditional systems have limitations in ensuring that important communications or urgent responses are not overlooked during the vacation. Furthermore, dealing with the large amount of unprocessed emails and meeting details after the vacation takes time and effort, making it difficult to resume work smoothly.

[1633] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1634] In this invention, the server includes a means for automatically answering calls during vacation, a means for converting the contents of received calls into text data using voice recognition technology, and a means for analyzing the converted text data using natural language processing technology to determine the urgency of the call. This allows important communications and emergency responses to be made even during vacation, while allowing employees to return to work quickly and efficiently after their vacation.

[1635] "Means for automatically answering calls" refers to the ability of the system to answer calls on behalf of a vacation user and play a pre-defined voice message.

[1636] "Voice recognition technology" is a technology that analyzes voice data and converts it into text data.

[1637] "Text data" is character information converted from voice data using voice recognition technology.

[1638] "Natural language processing technology" is a technology that analyzes text data to understand its content, urgency, and intent.

[1639] The "means of determining urgency" is a function that uses natural language processing technology to analyze text data to identify important information and elements that require urgent action, and notify emergency contacts as necessary.

[1640] "Means for notifying emergency contacts" is a function that sends a notification to pre-set contacts when it is determined based on the analysis results that emergency contact is necessary.

[1641] A "mailbox" is a data storage device for receiving and storing a user's email.

[1642] "Meeting recording data" refers to data that records the audio information of a meeting.

[1643] A "generative AI model" is an artificial intelligence model that learns from large amounts of data to perform tasks such as text generation and summarization.

[1644] The "means of generating summaries" is a function that uses a generative AI model to extract key information from collected email and meeting recording data and provide it in a shortened format.

[1645] The "means for prioritizing" is a function that evaluates the importance and urgency of information based on the generated summary data and determines the order in which it should be processed.

[1646] A "Summary Report" is a report that lists key points and priorities generated from collected and summarized data.

[1647] A "terminal" is a device that allows a user to communicate with a server, and includes smartphones, computers, etc.

[1648] The "means for receiving notification" is a function by which the terminal receives information sent from the server and displays that information to the user.

[1649] This invention is based on a system that consists of three main components: a server, a terminal, and a user. Each component operates as follows:

[1650] server

[1651] The server plays a central role in this system, hosting the AI ​​model and processing data in real time. The server has the following specific functions:

[1652] 1. Automated telephone answering

[1653] The server automatically detects incoming calls from vacation users and plays pre-defined answering messages. Specifically, the server monitors the telephone line signal, and when a call comes in, it selects an appropriate voice file from templates stored in a database and plays it. For example, it could play a voice file that says, "I'm currently on vacation. I'll get back to you later."

[1654] 2. Transcription and analysis of call content

[1655] The server records the audio data of the call and converts it into text data using speech recognition technology, using technologies such as the Google Cloud Speech-to-Text API. The generated text data is then analyzed using natural language processing (NLP) technology to determine the urgency of the call content. The urgency is evaluated based on predefined rules (e.g., the presence of keywords such as "emergency" or "urgent"). For example, if the call content is something like "In the event of a fire," it is determined that an emergency response is required.

[1656] 3. Emergency notification

[1657] Based on the analysis, the server will send a notification to pre-defined emergency contacts if necessary. This notification can be sent in different ways, such as SMS, email, or push notification. For example, it may contain a message such as "A critical security issue has occurred. Your immediate attention is required."

[1658] 4. Email and meeting summaries

[1659] The server periodically checks the user's mailbox and meeting recording data. Specifically, it uses the Gmail API for the mailbox and speech recognition technology for the meeting recording. A generative AI model (e.g., ChatGPT) is used to generate a summary of the collected data. Prompts used include "Please summarize the following email" and "Please summarize the main points of this meeting." For example, a summary such as "The main topic of discussion was a progress report on a new project" can be generated.

[1660] 5. Prioritize

[1661] The server evaluates the importance and urgency of emails and meeting content based on the generated summary data, and assigns priorities to them. This prioritization is based on algorithms such as "urgency of time" and "job titles of the people involved." For example, "customer complaints" might be listed as the top priority.

[1662] 6. Generate and send a summary report

[1663] The server compiles all the summary data into a single summary report at the end of the vacation. This summary report is sent to the user's device via email or push notification. For example, it may contain a list of emails requiring replies under "Important matters during vacation," a summary of new emails, and key points from important meetings.

[1664] Terminal

[1665] The terminal refers to the user's smartphone or computer, and functions as an interface with the server. The specific operation of the terminal is as follows.

[1666] 1. Display phone notifications

[1667] The device receives notifications from the server in real time and displays them to the user. The notification content includes a message such as "AI has responded." For example, a notification may appear on the user's smartphone saying "An automatic response was made because the user is on vacation."

[1668] 2. View the summary report

[1669] The device receives the summary report sent from the server and notifies the user. The report contents can be checked from the notification center or a dedicated app, and important items are highlighted. For example, a notification may be displayed showing "two emails requiring attention" and "four important meetings."

[1670] User

[1671] The user is an employee who uses the system, and can be relieved from work during the vacation and return to work efficiently after the vacation. The specific operation and usage of the user are as follows.

[1672] 1. Vacation Settings

[1673] The user enters the system's settings screen and enters the start and end times of the vacation. The server receives this information and prepares the automatic response mode. For example, the user sets "Vacation start date: October 1, 2023" and "Vacation end date: October 7, 2023."

[1674] 2. Checking during vacation

[1675] While on vacation, users check the notifications displayed on their devices. Notifications with urgent nature will be displayed with special alerts, and users can respond accordingly. For example, if a message appears saying "You have an important notification," users can check the content and respond as necessary.

[1676] 3. Resuming work after holidays

[1677] After the vacation ends, the user checks the summary report on their device. The report summarizes important emails and meeting details, and the user can start addressing them based on priority. For example, the report shows "2 emails needing attention" and "4 important meetings," allowing the user to return to work immediately.

[1678] The system allows users to return to work quickly and efficiently without missing important communications while on vacation.

[1679] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1680] Server Processing Steps

[1681] Step 1: Auto-answer your phone

[1682] The server detects an incoming call. It receives the signal from the telephone line as input and plays a pre-defined answering message. For this, an audio file is selected from a database and played. For example, a voice message such as "I'm currently on vacation. I'll get back to you later" is played.

[1683] Step 2: Transcribe the call

[1684] The server records the audio data of the call. It receives the audio data as input and converts it into text data using speech recognition technology such as the Google Cloud Speech-to-Text API. The content of the call is saved in text format as output. For example, the content "There's a fire" is generated as text data.

[1685] Step 3: Analyzing the call

[1686] The server analyzes the generated text data using natural language processing (NLP) technology. It receives the text data as input and analyzes specific keywords and phrases to determine the urgency. The output is an assessment of the urgency level. For example, if the keyword "fire" is included, the urgency level is determined to be high.

[1687] Step 4: Send emergency notifications

[1688] Based on the analysis results, the server sends notifications to pre-configured emergency contacts as needed. It receives the urgency assessment result as input and sends notifications via SMS, email, etc. The output is a notification sent to the emergency contact. For example, a notification stating, "A critical security issue has occurred. Urgent action is required."

[1689] Step 5: Summarize emails and meetings

[1690] The server periodically checks the user's mailbox and meeting recording data. It receives email data and audio recording data as input and generates a summary using a generative AI model (e.g., ChatGPT). The output is a summary that extracts the key points. Prompts used include "Please summarize the following email" and "Please summarize the main points of this meeting." For example, a summary might be generated that reads, "The main topic was a progress report on a new project."

[1691] Step 6: Prioritize

[1692] The server evaluates the importance and urgency of the generated summary data and assigns priorities to them. It takes the summary data as input and runs a prioritization algorithm (e.g., time urgency, role of the involved parties). As output, a list of priorities is created. For example, "Customer complaints" may be listed as the highest priority.

[1693] Step 7: Generate and submit a summary report

[1694] The server compiles all the summary data into a single summary report when the vacation ends. It receives the generated summary data as input and formats it into a summary report. As output, it generates a summary report that is sent to the user's device at the end of the vacation. For example, it includes a list of emails that need replying as "important matters during vacation," a summary of new emails, and important meeting points.

[1695] Terminal processing steps

[1696] Step 1: View phone notifications

[1697] The device receives notifications from the server in real time and displays them to the user. Notifications from the server are received as input, and an application for displaying them on a smartphone or computer receives them. As output, a notification message is displayed to the user. For example, a notification saying "An automatic response has been sent because I am on vacation" is displayed.

[1698] Step 2: View the summary report

[1699] The device receives the summary report sent from the server and notifies the user. It receives the summary report data as input and displays it in a dedicated app or notification center. As output, the summary report is displayed to the user. For example, the device may be notified of "two emails that need attention" and "four important meetings."

[1700] User operation steps

[1701] Step 1: Vacation Settings

[1702] The user enters the system's settings screen and enters the start and end times of the vacation. As input, the start and end dates of the vacation are entered into the system, and the server receives this information. As output, the server-side settings are updated and the auto-response mode is ready. For example, the user sets "Vacation starts October 1, 2023" and "Vacation ends October 7, 2023."

[1703] Step 2: Vacation Check

[1704] The user checks the notifications displayed on the device while on vacation. As input, the user checks the notifications displayed on the device and takes emergency notices if necessary. As output, the user can take emergency measures. For example, if the message "You have an important notice" is displayed, the user checks the contents and takes action if necessary.

[1705] Step 3: Resuming work after the holidays

[1706] After the vacation ends, the user checks the summary report on their device. The summary report data is taken as input, and a plan is made to prioritize important matters. The output is the ability to quickly return to work based on work priorities. For example, "2 emails needing attention" and "4 important meetings" are displayed, allowing the user to quickly start dealing with the important tasks.

[1707] (Application example 1)

[1708] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1709] Modern business people are faced with the need to respond to emergencies and important tasks even while on vacation. However, answering phone calls, checking emails, and even responding to emergencies while on vacation can significantly reduce the quality of their vacation. This invention aims to solve this problem by providing a system that streamlines tasks that are treated as an extension of work even while users are on vacation, and enables rapid response to emergencies.

[1710] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1711] In this invention, the server includes means for automatically answering calls during vacation, means for converting received phone call content into text and analyzing it, means for determining the urgency of the call from the analyzed content and notifying an emergency contact as necessary, means for the user to monitor the surroundings using a visual device and detect abnormal behavior or an emergency, means for notifying the user of the detected abnormality or emergency by voice and providing instructions on its content, and means for analyzing the converted call content, assessing the urgency, and providing instructions on how to respond. This allows the user to effectively and efficiently handle work and emergencies even while on vacation.

[1712] A "system for improving work efficiency during vacation" is a system that automatically processes work on behalf of a user who is on vacation, improving efficiency after the user returns to work.

[1713] "Auto-answer" is a feature that allows the system to play a pre-defined answering message, eliminating the need for the user to answer the phone.

[1714] "Texting" is the process of converting audio data into text data, and is done using voice recognition technology.

[1715] "Analysis" is the process of using natural language processing techniques on text data to determine the intent and urgency of the content.

[1716] "Means for notifying emergency contacts" refers to a method for sending a notification to a designated emergency contact based on the results of the analyzed data.

[1717] A "visual device" is a device worn by a user that allows the user to obtain information visually, and is used as smart glasses or a head-mounted display.

[1718] The "means for detecting abnormal behavior and emergency situations" is a system that uses cameras and sensors in the visual device to monitor the user's surroundings and recognize abnormal activities and emergency situations.

[1719] "Voice notification means" is a function in which, when an emergency occurs, the system explains the situation to the user by voice and gives necessary instructions.

[1720] A "summary report" is a report that summarizes the collected data and comprehensively presents the information the user needs after completing their vacation.

[1721] The embodiment of the present invention is mainly composed of three components: a server, a terminal, and a user. This embodiment allows users to be freed from work during vacations while enabling a quick and appropriate response to emergencies.

[1722] server

[1723] The server plays a central role in this system, hosting the AI ​​model and processing data in real time. The server has the following specific functions:

[1724] 1. Auto-answer: Automatically detects incoming calls when the user is on vacation and plays a pre-defined answering message.

[1725] 2. Converting call content into text: Converting received phone calls into text data using voice recognition technology. This process is achieved using Google Speech API, etc.

[1726] 3. NLP analysis: Analyze text data using natural language processing technology (e.g., spaCy or NLTK) to determine the urgency of the call content.

[1727] 4. Emergency Notification: Based on the analysis results, notifications will be sent to pre-defined emergency contacts as necessary.

[1728] 5. Email and meeting summaries: Collect new emails and meeting contents from the user's mailbox and meeting recording data, summarize the collected data using a generative AI model (e.g., ChatGPT), and generate the results as a summary report.

[1729] 6. Prioritization: Based on the generated summary data, the importance and urgency are evaluated and prioritized.

[1730] 7. Generate and send summary report: At the end of the vacation, a summary of all collected data is compiled into a single summary report and sent to the user's device.

[1731] Terminal

[1732] The terminal refers to the user's smartphone or computer, and functions as an interface with the server. The specific operation of the terminal is as follows.

[1733] 1. Displaying phone call notification: Receives a notification from the server and displays a message to the user such as "AI has answered." If necessary, details of the call will also be displayed.

[1734] 2. Displaying summary reports: Receives summary reports sent from the server and notifies the user. The reports include summaries and priorities of emails and meetings received during vacation.

[1735] User

[1736] Users are staff who use the system, and there are specific operations and usage methods as follows.

[1737] 1. Vacation Settings: The user enters the start and end time of the vacation into the system, which prepares the server to activate the auto-answer mode.

[1738] 2. Checking while on vacation: The user checks the notifications displayed on the device while on vacation. If there is an emergency, the user receives a detailed notification from the server and takes appropriate action.

[1739] 3. Resuming work after vacation: After returning from vacation, users can check the summary report on their device. The report summarizes important emails and meeting details, allowing them to quickly start working on important tasks.

[1740] Specific examples

[1741] For example, if a user sets a one-week vacation, the server will automatically answer all incoming calls during the vacation period, converting the calls into text and analyzing them. If the analysis determines that an emergency response is required, the server will immediately send a notification to the emergency contact. At the same time, the server will regularly check the user's mailbox and meeting recordings, and generate summaries of new emails and meetings. The generated summaries are compiled into a summary report with prioritized information and sent to the user's device at the end of the vacation. When the user checks their device after the vacation, a summary report with all the necessary information will be displayed, allowing them to quickly start responding to important tasks.

[1742] Prompt Sentence Examples

[1743] "A suspicious person is detected while the user is on vacation. Real-time video and audio analysis is performed to determine the urgency. If the urgency is high, a notification is sent and response instructions are displayed."

[1744] In this way, the system of the present invention provides a concrete means for enabling users to enjoy their vacation while also returning to work efficiently after the vacation.

[1745] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1746] Step 1:

[1747] Vacation Settings

[1748] The user inputs the start and end times of the vacation into the system, which prepares the server to activate the automatic response mode. The start date and time and end date and time of the vacation are specified as input data, and the server sets the activation of the automatic response mode based on that information. Specifically, the user inputs the vacation information using the terminal interface, and the server saves the information in the database.

[1749] Step 2:

[1750] Automatic answering of incoming calls

[1751] If a call comes in while you're on vacation, the server automatically detects the call and plays a pre-defined answering message. It takes the received call's voice data as input and plays the answering message as output. Specifically, the server detects an incoming call and plays the answering message on an audio output device.

[1752] Step 3:

[1753] Transcription of call contents

[1754] The server converts the received voice data into text. In this step, it uses voice recognition technology such as Google Speech API to convert the input voice file into text data. The text data is output. Specifically, the server sends the voice data to the API and stores the converted text in a database.

[1755] Step 4:

[1756] NLP analysis

[1757] The server analyzes the text data using natural language processing (NLP) technology and evaluates its urgency. The input text is passed through an NLP module (e.g., spaCy or NLTK), and the data with an urgency rating is output. Specifically, if the analysis results indicate a high urgency, that information is reflected in the emergency notification settings.

[1758] Step 5:

[1759] emergency notification

[1760] Based on the analysis results, the server sends a notification to the pre-defined emergency contacts. The server uses the urgency assessment data as input data and sends a notification as output. Specifically, the server sends the notification to the emergency contacts via phone or message notification services.

[1761] Step 6:

[1762] Email and meeting summary generation

[1763] The server collects new emails and meeting contents from the user's mailbox and meeting recording data, and generates summaries using a generative AI model (e.g., ChatGPT). Emails and meeting records are used as input data, and summary text data is output. Specifically, the server periodically checks the mailbox and meeting recording data, and generates summaries by inputting the data into the generative AI model.

[1764] Step 7:

[1765] Prioritization

[1766] The server evaluates the importance and urgency of the generated summary data and assigns priorities to them. It uses the summary text as input data and outputs prioritized schedule data. Specifically, the server assigns a rank to each summary using an evaluation algorithm.

[1767] Step 8:

[1768] Generate and send summary reports

[1769] The server generates a summary report and sends it to the user's device. Prioritized summary data is used as input data, and a document constructed as a summary report is generated and sent as output. Specifically, the server sends the generated report to the user by email or other means.

[1770] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1771] The system of this invention consists of three main components: a server, a terminal, a user, and an emotion engine. Each component operates as follows:

[1772] server

[1773] The server plays a central role in the system, hosting the AI ​​model and emotion engine, and performing real-time data processing. The server has the following specific functions:

[1774] 1. Automated telephone answering

[1775] The server has the function of automatically detecting an incoming call when the user is on vacation and playing a pre-defined answering message.

[1776] 2. Transcription and analysis of call content

[1777] The server converts the received call content into text data using voice recognition technology, and then analyzes the text data using natural language processing (NLP) technology to determine the urgency of the call content.

[1778] 3. Collaboration with emotion engine

[1779] The server uses an emotion engine to analyze the user's emotions from their voice data. Based on the results of the emotion analysis, the server combines it with regular text analysis to more accurately determine the urgency of the message.

[1780] 4. Emergency notification

[1781] The server will send notifications to pre-configured emergency contacts as needed.

[1782] 5. Email and meeting summaries

[1783] The server collects new emails and meeting contents from the user's mailbox and meeting recording data, summarizes the collected data using a generation AI such as ChatGPT, and generates a summary report.

[1784] 6. Prioritize

[1785] The server takes into account the emotional data of emails and meeting contents analyzed using an emotion engine, and prioritizes them according to importance and urgency based on the summary data.

[1786] 7. Generate and send a summary report

[1787] The server compiles all new emails and summaries of meetings, including emotion data, into a single summary report at the end of the vacation, and sends it to the user's terminal.

[1788] Terminal

[1789] The terminal refers to the user's smartphone or computer, and acts as an interface with the server. The specific operations of the terminal are as follows:

[1790] 1. Display phone notifications

[1791] The device receives a notification from the server and displays a message to the user such as "AI has responded" along with the results of the emotion analysis. If necessary, details of the call content will also be displayed.

[1792] 2. View the summary report

[1793] The terminal receives a summary report sent from the server and notifies the user. The report includes a summary and priority of emails and meetings received during the vacation. It also displays the results of sentiment analysis.

[1794] User

[1795] The user is an employee who uses the system, and can be relieved from work during the vacation and return to work efficiently after the vacation. The specific operations and usage of the user are as follows:

[1796] 1. Vacation Settings

[1797] The user enters the start and end times of their vacation into the system, which prepares the server to activate the auto-answer mode.

[1798] 2. Checking during vacation

[1799] The user checks the notifications displayed on the device while on vacation. If there is an emergency, the user receives a detailed notification from the server and takes appropriate action.

[1800] 3. Resuming work after holidays

[1801] After the vacation, users can check a summary report on their device, which summarizes important emails and meeting details, allowing them to return to work smoothly based on their priorities.

[1802] Specific examples

[1803] For example, if a user sets a one-week vacation, the server will automatically respond to all calls during the vacation period, converting the calls into text and analyzing them. At the same time, the server uses an emotion engine to analyze the user's emotions from the voice data. For example, if the emotion analysis indicates a high level of urgency, the server will immediately send a notification to emergency contacts.

[1804] The server also periodically checks the user's mailbox and meeting recordings to generate summaries of new emails and meetings. Using an emotion engine, it analyzes the emotions of the collected data and prioritizes them based on importance and urgency. At the end of the vacation, a summary report containing the prioritized summaries is sent to the user's device.

[1805] When users check their device after their vacation, they will see a summary report that summarizes all the information they need, along with additional information based on sentiment analysis, allowing them to quickly start working on the most important tasks.

[1806] In this way, the system of the present invention allows users to enjoy their vacation while efficiently returning to work after their vacation. Furthermore, the emotion engine enables more accurate determination of urgency and prioritization of information.

[1807] The processing flow will be explained below.

[1808] Answering phone calls during holidays

[1809] Step 1:

[1810] The server receives the user's vacation setting information and enables the telephone auto-answer mode during the vacation period.

[1811] Step 2:

[1812] The server detects an incoming call while on vacation.

[1813] Step 3:

[1814] The server plays a pre-programmed answering message and converts the call into text using voice recognition technology.

[1815] Step 4:

[1816] The server analyzes the text of the call using natural language processing (NLP) technology to determine the urgency of the call.

[1817] Step 5:

[1818] The server uses an emotion engine to analyze emotions from the voice data, for example, detecting changes in tone and speed of the voice and using these as indicators of urgency.

[1819] Step 6:

[1820] The server combines the results of text analysis and sentiment analysis to determine the final urgency.

[1821] Step 7:

[1822] The server will send notifications to pre-configured emergency contacts as needed.

[1823] Step 8:

[1824] The server generates a summary of the response content and emotion analysis results and sends it to the user's terminal.

[1825] Step 9:

[1826] The device will notify the user that "AI has answered," along with the results of the emotion analysis. If necessary, it will also display details of the call.

[1827] Vacation email and meeting summaries

[1828] Step 1:

[1829] The server periodically checks the user's mailbox and collects new emails.

[1830] Step 2:

[1831] The server checks the user's calendar and meeting recording data to collect the corresponding meeting details.

[1832] Step 3:

[1833] The server analyzes the contents of newly arrived emails collected using ChatGPT and generates summaries.

[1834] Step 4:

[1835] The server converts the meeting recording data into text using speech recognition technology and uses ChatGPT to generate a summary that extracts the key points.

[1836] Step 5:

[1837] The server uses an emotion engine to analyze the user's emotions contained in emails and meeting content, for example, by assigning emotion labels to comments in emails and meetings.

[1838] Step 6:

[1839] The server determines the importance and urgency of emails and meetings based on the content of the generated summaries, and prioritizes them taking into account the results of sentiment analysis.

[1840] Step 7:

[1841] The server generates a priority list and sends it to the user's terminal.

[1842] Step 8:

[1843] The terminal receives the list and notifies the user.

[1844] Generate and view a summary report after the holidays

[1845] Step 1:

[1846] At the end of the vacation, the server compiles all collected new emails and summaries generated from the meeting content into a single summary report, incorporating sentiment analysis results.

[1847] Step 2:

[1848] The server transmits the generated summary report to the user's terminal.

[1849] Step 3:

[1850] The terminal notifies the user of receipt of the summary report.

[1851] Step 4:

[1852] The user can view a summary report on their device, which includes a summary and prioritization of all emails and meetings received during their vacation, as well as the results of sentiment analysis.

[1853] Step 5:

[1854] Based on the summary report, users can start responding to important emails and meeting details one by one.

[1855] Example 2

[1856] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1857] In today's business environment, employees are expected to respond efficiently to important phone calls, emails, and meeting details while on vacation. However, if work is completely neglected while on vacation, emergencies and important matters may be overlooked, resulting in a huge amount of work to be handled once the vacation returns. Furthermore, conventional systems do not perform sentiment analysis, making it difficult to determine urgency and insufficient to prioritize information. A new system is needed to solve these issues.

[1858] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1859] In this invention, the server includes: means for automatically answering calls during vacation; means for converting received call content into text using speech recognition technology; means for analyzing text data using natural language processing technology; means for analyzing emotions from the voice data; means for determining urgency based on the analyzed call content and emotion data and notifying emergency contacts as necessary; means for collecting new emails and meeting content from the user's mailbox and conference recording data; means for generating summaries of the collected data using a generative AI model; means for prioritizing from the summaries; means for generating a summary report including email and meeting summaries at the end of the vacation; and means for sending the generated summary report to the user. This allows for more efficient work during vacation and a smooth resumption of work after the vacation. Furthermore, using emotion analysis allows for more accurate determination of urgency and effective prioritization of information.

[1860] A "means for automatically answering vacation calls" is hardware or software that has the function of automatically detecting calls received while the user is on vacation and playing a pre-defined answering message.

[1861] "Means for converting received telephone calls into text using voice recognition technology" refers to hardware or software that uses voice recognition technology to convert the voice data of received telephone calls into text data.

[1862] "Means for analyzing text data using natural language processing technology" refers to hardware or software that uses natural language processing technology to analyze the content of text data obtained by speech recognition and extract useful information from it.

[1863] "Means for analyzing emotions from voice data" refers to hardware or software that utilizes emotion analysis technology to analyze the emotional state of a user or the other party from voice data or the text data converted from the voice data.

[1864] "Means for determining the urgency of a call based on the analyzed content and emotional data, and notifying emergency contacts as necessary" refers to hardware or software that evaluates the urgency of the content of a call based on the results of natural language processing and emotional analysis, and sends a notification to emergency contacts in the event of an emergency.

[1865] "Means for collecting new emails and conference contents from the user's mailbox and conference recording data" refers to hardware or software that periodically checks the user's mailbox and conference audio data to collect newly received emails and conference contents.

[1866] "Means for generating summaries of collected data using a generative AI model" refers to hardware or software capable of inputting collected email and meeting data into an AI model and generating summaries of the data.

[1867] The "means for prioritizing from summaries" refers to hardware or software capable of prioritizing the generated summary data based on importance and urgency.

[1868] "Means for generating a summary report including email and meeting summaries at the end of vacation" refers to hardware or software that compiles the data collected and summarized during the vacation into a single report at the end of the vacation.

[1869] The "means for transmitting the generated summary report to the user" is hardware or software for transmitting the generated summary report to the user's terminal after the vacation ends.

[1870] The "means for setting the start and end times of vacation" refers to an interface and system for a user to input the start and end times of vacation and record the schedule.

[1871] "Means for preparing the server to enable the auto-reply mode at the vacation start time" refers to programs and processes on the server for enabling the auto-reply mode when the vacation begins.

[1872] The "means for displaying vacation notices on a terminal" refers to hardware or software that receives notices from the server and displays them on the user's terminal.

[1873] The "means for reassessing urgency based on text data and emotion data" refers to hardware or software that reassess data obtained through natural language processing and emotion analysis to improve the accuracy of urgency.

[1874] "Means of sending notifications to emergency contacts via SMS or email when necessary" refers to a system that sends notifications to appropriate emergency contacts via SMS, email, etc. when a high level of urgency is deemed to exist.

[1875] "Means for inputting the collected data into a generative AI model to generate a summary and then integrating it with the sentiment analysis results" is hardware or software for inputting the collected data into an AI model and integrating the results with the sentiment analysis results.

[1876] The "means for using an algorithm to prioritize the generated summary data" refers to a system that applies a specific algorithm to set priorities for the summary data.

[1877] The system of this invention consists of three main components: a server, a terminal, and a user, as well as an emotion engine. Each component operates as follows:

[1878] server

[1879] The server plays a central role in this system, hosting various data processing and emotion engines. The specific hardware and software configuration is as follows:

[1880] 1. Automated telephone answering

[1881] The server uses an automated answering system such as Asterisk to automatically answer calls received while the user is on vacation. For example, when a user sets the start time of their vacation, the server launches Asterisk and plays a pre-recorded message.

[1882] 2. Transcription of call contents

[1883] The server uses the Google Cloud Speech-to-Text API to convert the call into text data. For example, when the call ends, the server saves the recording and calls the Google Cloud API to convert the voice data into text.

[1884] 3. Text Data Analysis

[1885] The server uses Python NLP libraries (such as NLTK and spaCy) to analyze the text data and determine the urgency of the call content. Specifically, it extracts important keywords and evaluates the urgency.

[1886] 4. Analysis by Emotion Engine

[1887] The server uses Azure Text Analytics for Sentiment and IBM Watson Tone Analyzer to analyze emotions from voice data, and based on the obtained emotional data, it can assess the urgency of the message with even greater accuracy.

[1888] 5. Sending emergency notifications

[1889] The server uses Twilio or SendGrid to send notifications to emergency contacts when necessary, for example, using Twilio to send SMS notifications for calls deemed to be urgent.

[1890] 6. Collect email and meeting data

[1891] The server periodically collects user mailbox and meeting recording data using the Gmail API and Microsoft Graph API. For example, it periodically calls the API every day to retrieve new emails and meeting data.

[1892] 7. Summarization using generative AI models

[1893] The server inputs the collected data into a generative AI model such as ChatGPT to generate a summary. For example, email data obtained from an API is passed to the generative AI model to obtain the summary result.

[1894] 8. Data Prioritization

[1895] The server prioritizes the summary data based on the importance and urgency of the data, based on the analysis results of the emotion engine. For example, data that is deemed "urgent" by emotion analysis has a high priority.

[1896] 9. Generate and send summary reports

[1897] The server compiles all the summary data into one summary report at the end of the vacation and sends it to the user's terminal using the SMTP protocol. For example, the server collects all the data on the day before the vacation ends and compiles it to generate a report.

[1898] Terminal

[1899] The terminal refers to the user's smartphone or computer, and functions as an interface with the server. The specific operation is as follows.

[1900] 1. Display phone notifications

[1901] The device receives a notification from the server and displays the emotion analysis results along with a message such as "AI has responded," prompting the user to confirm via the notification system.

[1902] 2. View the summary report

[1903] The device receives the summary report sent from the server and notifies the user, for example, by displaying the report through an email app or a dedicated app.

[1904] User

[1905] Users are employees who use the system, and the specific operations and usage are as follows.

[1906] 1. Vacation Settings

[1907] The user enters the start and end time of the vacation through the app, which causes the server to activate the auto-reply mode.

[1908] 2. Checking during vacation

[1909] The user checks the notifications displayed on the device while on vacation. If there is an emergency, the user receives a detailed notification from the server and takes appropriate action.

[1910] 3. Resuming work after holidays

[1911] After the vacation ends, the user checks the summary report on the terminal and starts working on important tasks based on the priority.

[1912] Specific examples

[1913] For example, if a user schedules a one-week vacation, the server will automatically answer all calls during that vacation period. When a call comes in, Asterisk detects it and plays a pre-recorded message. The call is recorded and converted to text data using Google Cloud Speech-to-Text. Next, the text data is analyzed using a Python NLP library to extract important keywords and evaluate the urgency. At the same time, Azure Text Analytics for Sentiment performs sentiment analysis of the call. If the call is deemed urgent, a notification is sent to emergency contacts using Twilio.

[1914] The server also periodically collects user mailbox and meeting recording data and generates summaries using generative AI models such as ChatGPT. The summary data is then prioritized based on the results of sentiment analysis. At the end of the vacation, a summary report containing all summaries is generated and sent to the user's device via SMTP. Users can then review the summary report after their vacation and quickly begin work on high-priority tasks.

[1915] Prompt Sentence Examples

[1916] "Please analyze my calls during my vacation and determine their urgency. Please also analyze the sentiment of the calls and send me emergency notifications if necessary. Also, please summarize emails and meetings and send me a summary report at the end."

[1917] In this way, the system of the present invention allows users to enjoy their vacation while efficiently returning to work after their vacation. The emotion engine also enables more accurate determination of urgency and prioritization of information.

[1918] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1919] Step 1: Set up your vacation time

[1920] Input: The user opens the dedicated application and enters the start and end dates and times of the vacation.

[1921] How it works: The user enters start and end dates and times into a vacation configuration form, the application validates them, and once validation is complete, the configuration information is sent to the server.

[1922] Output: The server receives the user's vacation settings and prepares to enable auto-answer mode at the vacation start time.

[1923] Step 2: Auto-answer your phone calls

[1924] Input: When the vacation begins, a call comes in to the user's phone.

[1925] How it works: The server uses Asterisk to detect incoming calls and play a pre-recorded greeting, such as "I'm on vacation right now."

[1926] Output: When a call comes in, an automatic answering message is played.

[1927] Step 3: Record and transcribe the call 【192...

Claims

1. A system for streamlining work during holidays, A means of automatically answering calls during vacation time; A means of converting received phone calls into text and analyzing them; A method to determine the urgency of the call based on the analyzed content and notify emergency contacts as necessary. A system including:

2. 10. The system of claim 1, means for collecting new emails and meeting contents from a user's mailbox and meeting recording data; means for generating a summary of the collected data; A means of prioritizing from the summary; The system further comprises:

3. 3. The system according to claim 1 or 2, A means for generating a summary report at the end of vacation that includes email and meeting summaries; means for transmitting the generated summary report to a user; The system further comprises:

Images