system

Abstract

We provide the system. [Solution] A means of collecting communication data received from the user's device during vacation, A means for generating and summarizing important information from the aforementioned communication data using AI technology, A means for converting audio data and video data into text information, and for analyzing the text information to generate a summary, A method for performing sentiment analysis and prioritizing the display of negative information, A means of providing the summarized information in an individualized manner based on the user's relevant business activities, Analyzing delivery instructions, customer feedback, and information on new stores in the area accumulated during the holiday period, and providing a means to present important information to delivery personnel, A system that includes this.

Description

Technical Field

[0005] ,

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, and includes steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When returning to work after a long vacation, there is a problem that a huge amount of unread emails and messages accumulate, and it is difficult to quickly organize and grasp this information, which hinders the smooth return of workers to their duties. Furthermore, there is a burden of having to check the content of online meetings during the vacation at a later date. Current information processing technologies do not sufficiently provide means for efficiently summarizing this information and presenting important information based on priority.

Means for Solving the Problems

[0005] The present invention provides a means for collecting emails and messages received from a user's terminal device during vacation, a means for summarizing important information from these emails and messages using generative artificial intelligence technology, and a means for converting and analyzing audio and video data into text data to generate a summary. Furthermore, it provides a system that improves the efficiency of returning to work by performing sentiment analysis to prioritize the display of negative information and providing personalized summary information based on the user's relevant work content. In addition, it ensures the security of information by encrypting and securely managing the information obtained from the data collection and controlling access rights.

[0006] A "user" is an individual or group that utilizes an information processing system and is authorized to use the data and information within the system.

[0007] A "terminal device" refers to a computer device, smartphone, or other information and communication equipment equipped by a user, which has the function of data collection and information display.

[0008] "Email" refers to informational messages, including text and files, that are sent and received via the internet or communication networks.

[0009] A "message" refers to information sent and received via business communication tools other than email, and includes text and multimedia content.

[0010] "Generative artificial intelligence technology" refers to technology that autonomously generates output that conforms to certain conditions based on input data, and in particular, includes technology that includes information summarization using natural language processing.

[0011] "Audio data" refers to digital data containing sound information spoken by a user or other person, and typically includes the content of conversations or meetings.

[0012] "Video data" refers to data that records visual and auditory information in digital format, and includes the content of meetings and presentations, including video and audio.

[0013] "Character data" refers to information data expressed in text format, and typically includes linguistic information that can be read visually.

[0014] "Sentiment analysis" is a technique that analyzes the emotional and opinion tendencies contained in text and audio data, identifying positive or negative sentiment.

[0015] "Summary information" refers to information that concisely summarizes the important elements extracted from a vast amount of data, helping users to quickly understand it.

[0016] "Personalization" is the process of customizing information to suit a specific user based on their work content and past behavior.

[0017] Encryption is a method of making the contents of data unreadable to third parties, and it is a fundamental security technology for ensuring safety.

[0018] "Access control" is a procedure in information systems that manages the permissions for specific users or processes to use resources and data. [Brief explanation of the drawing]

[0019] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Mode for Carrying Out the Invention

[0020] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described according to the accompanying drawings.

[0021] First, the terms used in the following description will be explained.

[0022] In the following embodiments, the signed processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Furthermore, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), and APU (Accelerated Processing Unit).

[0023] In the following embodiments, signed RAM (Random Access Memory) is a memory that temporarily stores information and is used as work memory by the processor.

[0024] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0025] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0026] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0027] [First Embodiment]

[0028] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0029] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0030] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0031] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0032] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0033] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0034] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0035] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0036] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0037] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0038] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0039] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0040] This invention is an information processing system designed to enable users to efficiently return to work after taking extended leave. The following provides a detailed description of an embodiment of this system.

[0041] First, users need to configure their system access permissions in advance to allow data processing during their vacation period. This allows the server to collect necessary information from the mail server and communication tools while the user is away.

[0042] The server accesses the user's mailbox and collects all unread emails. Similarly, it collects messages received during the holiday from communication tools. The server efficiently retrieves this information using APIs and protocols.

[0043] Next, the server uploads audio and video data from the terminals to the cloud as needed, and converts it into text data using speech recognition technology. This converted text data serves as the basis for understanding the meeting results and important discussion points.

[0044] The server uses generative artificial intelligence technology to generate summaries from the collected text and character data. In generating these summaries, elements such as project progress and important action items are extracted from emails and messages and summarized concisely. Sentiment analysis is also performed simultaneously, and the system is configured to prioritize notifying users of particularly negative content.

[0045] The summarized information is provided in a customized format based on each user's work history and project status. For example, users who prioritize information related to a specific project will see a summary of that project first.

[0046] Sufficient consideration has been given to the security of information; the servers encrypt the data they collect and store, and strict access controls are in place to prevent unauthorized access and information leaks by third parties.

[0047] As a concrete example, consider a user returning to work after a two-week vacation. During this time, there are 100 emails, 20 messages, and 5 online meeting recordings. The server collects all of this data and uses generative artificial intelligence to create a summary. One summary might extract the crucial information, "Project X's deadline has been moved up," and present it to the user. As a result, the user can quickly grasp important changes and resume work efficiently.

[0048] In this way, users can reduce the stress of information overload even after returning from vacation and continue their work flow smoothly.

[0049] The following describes the processing flow.

[0050] Step 1:

[0051] Based on the user's pre-configured settings, the server connects to email servers and business communication tools via APIs to automatically collect unread emails and messages received during vacation.

[0052] Step 2:

[0053] The server converts the collected text data into a format that is easy for the natural language processing engine to analyze. The HTML portion of emails is converted to text format, the language of the message is detected, and preparations for analysis are made according to the language characteristics.

[0054] Step 3:

[0055] The device synchronizes the audio and video data of the online meeting to cloud storage. This prepares the server to apply speech recognition technology to generate text data from the audio data.

[0056] Step 4:

[0057] The server uses natural language processing to analyze the collected and transformed text data, extract key information, and generate a summary. It identifies key points and tasks and creates a concise summary.

[0058] Step 5:

[0059] The server performs sentiment analysis on the generated text data and summaries. It measures sentiment based on the content of emails and messages, and configures the system to prioritize notifying users of information containing negative sentiment.

[0060] Step 6:

[0061] The server compares the user's past work information and project data, and customizes the summarized information. It prioritizes and organizes information that is particularly relevant to the user's job, and performs filtering to present it in an individualized format.

[0062] Step 7:

[0063] The terminal displays customized summary information received from the server to the user in a dashboard format, helping the user easily access important information.

[0064] Step 8:

[0065] The server ensures the confidentiality and security of information by encrypting and securely storing all data and implementing strict access controls.

[0066] (Example 1)

[0067] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0068] In today's business environment, when users are absent due to extended leave, there is a need to efficiently organize large amounts of unread communication data and information, and to quickly return to work. Current methods risk information overload upon return, particularly important information being overlooked. This invention aims to provide a system that allows users to smoothly resume work after leave and efficiently grasp important information.

[0069] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0070] In this invention, the server includes means for collecting electronic communication data received from the user's computer during the vacation period, means for summarizing important information from the electronic communication data using generative artificial intelligence technology, and means for converting audio and video information into coded data, and analyzing the coded data to generate a summary. This enables the user to quickly and efficiently access important information and return to work even after their vacation.

[0071] A "user's computing device" refers to a computer or smart device used by an individual for sending, receiving, and processing data.

[0072] "Electronic communication data" refers to data including text, images, and other associated information sent and received through email and messaging applications.

[0073] "Generative artificial intelligence technology" refers to AI technology designed to analyze input information and automatically perform specific tasks such as summarizing and inference.

[0074] "Audio and video information" refers to human conversations, images, videos, and other multimedia data collected in these formats.

[0075] "Coded data" refers to data obtained by converting audio or video information into a text format that can be analyzed using algorithms.

[0076] "Analysis" is the process of examining collected data to extract specific information or patterns.

[0077] This information processing system is designed to help users smoothly return to work after taking time off during vacation. It utilizes computing power, online communication, and generative artificial intelligence technology to efficiently collect, process, and provide relevant information.

[0078] First, before taking a vacation, users configure access permissions on their computing device to allow the system to collect data. This enables the server to connect to the user's computer and automatically collect electronic communication data even while they are on vacation. For example, it can collect data sent and received via email and messaging apps.

[0079] After collecting this data, the server transfers the audio and video information to the cloud and converts it into coded data using speech recognition and video analysis technologies. This process uses specific encoding techniques, and the converted data is analyzed by generative artificial intelligence technology. This AI technology has the ability to summarize important information from emails and chat messages, helping users quickly understand the information.

[0080] For example, consider a scenario where a user receives 50 emails and 10 meeting minutes while on vacation. The server collects all this information and uses an AI model to create a summary. The resulting summary extracts specific information such as "Project Y is progressing well" and provides it to the user. This saves the user the trouble of sifting through information upon returning from vacation, allowing them to quickly return to work.

[0081] Furthermore, this system can be customized based on the user's work content and project progress, providing individually optimized information. Information about projects of particular interest to the user appears to be prioritized over other information.

[0082] An example of a prompt message would be something like, "Please summarize project progress and important tasks from emails and messages received during your vacation," which would then be entered into the system.

[0083] This invention provides an effective means for users to efficiently return to their work and grasp important information without omission.

[0084] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0085] Step 1:

[0086] Users configure their system access permissions before going on vacation. They log in to their computing device and register their email account and API keys for business communication tools. This prepares the server to access the user's account and automatically collect data. The input requires user authentication information and API keys, and the output provides the access permission status.

[0087] Step 2:

[0088] The server connects to the user's computer and collects electronic communication data received during the vacation. Specifically, it retrieves email data using the IMAP protocol and collects message data using the communication tool's API. Electronic communication data is provided as input, and unread message content and metadata are stored as output.

[0089] Step 3:

[0090] The device uploads the user's voice and video information to the cloud. This information is then converted into coded data using speech recognition and video analysis technologies. Specifically, it converts audio files to MP3 format and video files to MP4 format, saving them to the cloud. The input is audio and video files, and the output is coded text data.

[0091] Step 4:

[0092] The server receives coded data from the cloud and uses a generative AI model to summarize key information. A specific algorithm is used to extract project progress information and important tasks from the coded data. The input is coded data, and the output is a concise summary.

[0093] Step 5:

[0094] The server customizes the generated summaries based on individual work content and project progress. It is configured to prioritize displaying project-related elements and also performs sentiment analysis to highlight negative information. The input is a summary text, and the output is a customized information display.

[0095] Step 6:

[0096] The server protects and securely manages all collected data using cryptographic techniques. Access restrictions are implemented to prevent unauthorized access to the data. Specifically, information stored in the database is encrypted using AES-256. The input is the data before encryption, and the output is the encrypted data.

[0097] (Application Example 1)

[0098] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0099] Users returning to work after extended leave tend to be overwhelmed by the large amount of communication data and new information accumulated during their time off. As a result, it is difficult for them to catch up with the workflow in a short period of time. To address this problem, there is a need for information processing systems that enable users to return to work quickly and efficiently. In the food delivery industry in particular, it is necessary to appropriately grasp the latest delivery instructions, customer feedback, and new store opening information.

[0100] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0101] In this invention, the server includes means for collecting communication data received from the user's device during the vacation, means for summarizing important information from the communication data using AI technology, and means for analyzing delivery instructions, customer feedback, and new store information in the area that have accumulated during the vacation, and presenting important information to the delivery personnel. This enables the user to efficiently return to the workflow.

[0102] "User's equipment" refers to communication devices used to collect data received during vacation.

[0103] "Communication data" refers to a collection of information, including emails, messages, audio data, and video data.

[0104] "Generative AI technology" refers to artificial intelligence technology used to summarize important information from data.

[0105] "Methods of summarization" refer to the process of extracting important information from collected data and presenting it in a concise format.

[0106] "Sentiment analysis" refers to analytical techniques that identify emotions within data and highlight negative elements in particular.

[0107] "Personalized delivery methods" refer to the process of customizing summary information according to the user's work content to provide information efficiently.

[0108] "Delivery instructions" refers to specific directives or orders used in delivery operations.

[0109] "Customer feedback" refers to the feedback and opinions collected from customers who have used the service.

[0110] "New store information" refers to detailed information about stores that have recently opened in the area.

[0111] A "delivery driver" refers to a person responsible for delivery services in food delivery services.

[0112] The system for implementing this invention consists of a server for data processing and a terminal used by the user. The server comprehensively manages communication data collected from the user's terminal during vacation, namely emails, messages, voice data, and video data. The collected data is extracted and summarized using generative AI technology to extract important information. The generated summary incorporates a mechanism that performs sentiment analysis to prioritize the extraction of negative information.

[0113] The server uses programming languages ​​such as Python for data processing and computation, and utilizes RESTful APIs and natural language processing libraries (e.g., spaCy, transformers). Data security is ensured by security software that encrypts data and controls access (e.g., SSL / TLS protocol).

[0114] As a concrete example, consider a scenario where a delivery driver returns from a two-week vacation. The server collects 30 delivery instructions, 15 customer comments, and 5 pieces of new store information received during the vacation, and summarizes them using a generative AI. Finally, targeted information such as "a new store has opened" and "positive customer feedback has been received" is presented to the delivery driver as important information.

[0115] An example of a specific input prompt for the generating AI model would be: "Summarize the key changes and performance metrics from the following datasets: List of customer feedback, List of new store information." This would allow delivery drivers to quickly understand the latest information and resume operations smoothly.

[0116] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0117] Step 1:

[0118] The server collects communication data received from the user's device during their vacation. Inputs include emails, messages, audio data, and video data, while output is an integrated dataset of these. The server retrieves the data via an API and stores it in a database.

[0119] Step 2:

[0120] The server converts audio and video data into text. Input is audio and video files, and output is text. Speech recognition software is used to generate text from audio, and text is extracted from video as needed.

[0121] Step 3:

[0122] The server analyzes the collected text information using AI technology and generates a summary. The input is the text information converted in step 2 and the collected emails and messages, and the output is the summarized text. A prompt sentence is entered into the AI ​​model, and it extracts important information to create a summary.

[0123] Step 4:

[0124] The server analyzes the sentiment of the summarized information and prioritizes extracting negative information. The input is the summarized text generated in step 3, and the output is the summarized text with negative elements emphasized. Natural language processing techniques are used to analyze the sentiment of the text.

[0125] Step 5:

[0126] The server personalizes the summary information based on the user's work content and provides it to the terminal. The input is the summary text highlighted in step 4, and the output is the personalized summary information. Information priorities are set according to the user's work history and project status.

[0127] Step 6:

[0128] Users view summary information through their terminals. The output is personalized summary information. Based on the presented summary, users can quickly grasp the current work situation and decide on their next actions.

[0129] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0130] This invention is an information processing system that enables users to quickly and efficiently return to work after a long vacation, and is particularly characterized by its optimization of information presentation, taking into account the user's emotions. Specific embodiments of the present invention are described below.

[0131] First, the user installs the system and authorizes data collection from their terminal device during their vacation. Based on this authorization, the server accesses email servers and business communication tools to collect unread emails and messages received. Furthermore, the terminal connects with online meeting systems to upload audio and video data to the cloud, which the server then converts into text data using speech recognition technology.

[0132] Next, the server uses generative artificial intelligence technology to summarize the content of the collected emails and messages. The summarization employs techniques to extract particularly important tasks and decisions and organize the information concisely. It also incorporates an emotion engine to analyze the emotional elements in the data and prioritize notifications if negative information is present.

[0133] Furthermore, this system can recognize the user's emotional state in real time. The terminal is equipped with sensors that detect the user's facial expressions and tone of voice, and the server uses this information to activate an emotion engine. If the user is stressed, the server adjusts the order in which summary information is presented, reducing the user's psychological burden by delaying less important information.

[0134] For example, if a user is overwhelmed by 100 unread emails after returning to work, the system will detect that the user's facial expression indicates stress and first display only the most important messages requiring immediate attention. Then, as the user's facial expression relaxes, it will sequentially display the other messages. This reduces the user's information processing burden and efficiently helps them return to work.

[0135] Furthermore, the server encrypts data and strictly controls access to ensure security, thereby managing user data safely. This configuration provides the present invention with an environment in which users can use the system with peace of mind.

[0136] By combining these functions, the present invention aims to alleviate the stress caused by information overload when users return to work after vacation, thereby creating a more comfortable business environment.

[0137] The following describes the processing flow.

[0138] Step 1:

[0139] Users configure system settings before their vacation, granting the server access to email servers and business communication tools. This enables data collection during their vacation.

[0140] Step 2:

[0141] The server connects to the user's mail server and business communication tools via API to collect all emails and messages received during the holiday period. The scope of collection can be adjusted as needed based on user preferences.

[0142] Step 3:

[0143] The device synchronizes the audio and video data of the online meeting to cloud storage. This prepares the server to use speech recognition technology to convert the audio data into text data.

[0144] Step 4:

[0145] The server generates summaries using generative artificial intelligence techniques from collected emails, messages, and converted text data. It extracts key points related to high-priority information and tasks, creating concise summaries.

[0146] Step 5:

[0147] The server uses an emotion engine to perform sentiment analysis on the collected data. It identifies sections containing particularly negative emotions and prioritizes notifying the user, enabling quicker action.

[0148] Step 6:

[0149] The device uses sensors to detect the user's emotional state in real time and sends that information to a server. The system determines the user's stress level based on their facial expressions and tone of voice.

[0150] Step 7:

[0151] The server adjusts the order in which summary information is presented based on the user's emotional state. If the user is stressed, only important messages requiring immediate attention will be displayed first, while other information will be presented later.

[0152] Step 8:

[0153] The server encrypts and securely stores the data it collects and processes. Furthermore, strict access control is implemented to ensure that only authorized users can access the data.

[0154] Through this process, the system can support users in smoothly returning to work after vacation and reduce stress caused by information overload.

[0155] (Example 2)

[0156] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0157] In today's business environment, when users return from extended leave, they often face information overload due to a large volume of accumulated electronic communications and notifications. This situation raises concerns about user efficiency and mental burden upon returning to work. Furthermore, the presentation of information without considering the user's emotional state can increase stress upon returning to work.

[0158] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0159] In this invention, the server includes means for collecting electronic communications and notifications acquired from the user's information device during vacation; means for summarizing important elements from the electronic communications and notifications using a generative artificial intelligence method; means for converting audio and video information into text information, analyzing the text information, and creating a summary; means for performing sentiment analysis and prioritizing the presentation of negative elements; and means for detecting the user's emotional state and adjusting the order in which information is presented. This reduces the user's information burden and enables the presentation of information that is in line with their emotional state.

[0160] A "user" is an individual or organization that uses this system to manage information and efficiently process that information upon returning to work.

[0161] "Information device" is a general term for electronic devices that are operated by users to send, receive, and process data.

[0162] "Electronic communications" refers to digital information, such as emails and messages, that is sent and received over a network.

[0163] A "notification" is a means of communication from a system to inform a user of specific information or a message.

[0164] "Generative artificial intelligence methods" are artificial intelligence technologies that analyze data and use the results to create new summaries and suggestions.

[0165] A "summary" is a concise compilation of information, extracting the most important elements from the data.

[0166] "Audio information" refers to data that records sounds emitted as human voices in digital format.

[0167] "Visual information" refers to data recorded in digital format, consisting of visually represented moving images and still images.

[0168] "Text information" refers to information expressed as a string of characters, including character data converted from audio and video.

[0169] "Analysis" is the process of evaluating and interpreting data according to a specific purpose to derive valuable information.

[0170] "Emotional analysis" is a technology that infers and judges an individual's emotions and psychological state from data.

[0171] "Negative elements" refer to information that may cause discomfort or stress to the user.

[0172] "Emotional state" refers to a user's psychological and emotional condition, and is a factor that influences their daily activities.

[0173] "Adjusting the order in which information is presented" means changing the order in which information is presented in order to make it more efficient and effective depending on the user's situation.

[0174] This invention is an information processing system designed to facilitate information processing in the workplace when a user returns from a long leave of absence. The system aims to efficiently collect, summarize, and present information related to business communication while taking into account the user's emotional state.

[0175] First, users access the system using their own information devices and authorize the collection of electronic communications and notifications received during their vacation. These information devices include computers, smartphones, and tablet devices, which are used to operate the system.

[0176] Next, the server efficiently collects electronic communications and notifications based on information authorized by the user. This communication data includes emails and chat messages. The collected data is further processed within the server.

[0177] The server uses generative artificial intelligence techniques to automatically summarize important elements from the collected information. In this process, particularly important tasks and decisions are extracted, prioritizing information useful to the user. The summarization is performed using generative AI technologies such as OpenAI® models.

[0178] The device detects the user's emotional state in real time based on factors such as skin color and voice tone. This allows the server to determine whether the user is stressed. If the user indicates stress, the server adjusts the order in which information is presented, prioritizing less important information to reduce psychological burden.

[0179] Furthermore, audio and video information are also processed. The server can convert audio data used in online meetings into text using Google's (registered trademark) speech recognition technology, and then analyze it to provide useful information.

[0180] For example, if a user is overwhelmed by a large number of unread emails after returning to work, this system will immediately prioritize and display the most urgent emails. This allows the user to efficiently resume their work.

[0181] Examples of prompt statements to input into a generative AI model include the following:

[0182] "To help users returning to work after extended leave efficiently process unread emails, devise a method to summarize and prioritize important emails. Consider the user's current emotional state and propose an appropriate order for presenting the information."

[0183] In this way, the present invention helps users reduce information overload while using the system with peace of mind, and supports a smooth return to work.

[0184] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0185] Step 1:

[0186] The user logs into their information device and authorizes the system to collect electronic communications and notifications received during their vacation. Input requires the user's authentication information and permission settings. Output is the authorized access information sent to the server. Based on this access information, the server determines the user's permitted scope.

[0187] Step 2:

[0188] The server uses access information obtained from users to collect electronic communications and notifications into the system. Specifically, it calls APIs of mail servers and chat tools to retrieve unread messages. The inputs used are user access information and API information of the target servers. As output, the server stores unread communication data.

[0189] Step 3:

[0190] The server uses a generative AI model to perform summarization processing to extract important elements from the collected electronic communication data. The communication data collected in step 2 is required as input. Summarized information is generated as output. Here, the AI ​​model prioritizes extracting tasks and important decisions within the emails.

[0191] Step 4:

[0192] The device collects user facial expression data and voice tone to detect the user's emotional state in real time. Sensor data from the camera and microphone are used as input. The output is the user's emotion analysis result, which is sent to the server. Based on this information, the server determines the user's emotional state.

[0193] Step 5:

[0194] The server adjusts the order in which information is presented based on the user's emotional state. If the user is showing signs of stress, the server re-evaluates the information to be presented preferentially and displays the most important content first. Summarized information and the results of the emotional analysis are required as input. The adjusted order of information presentation is determined as output and sent to the terminal.

[0195] Step 6:

[0196] The terminal receives pre-configured information from the server and presents it to the user. The user can then resume their work based on this information. The input is ordered information from the server. The output is a visual display of information on the user's screen. Specifically, information is displayed in a way that suits the user's mood, improving work efficiency.

[0197] Through the steps outlined above, this system aims to prevent users from becoming overwhelmed with information, reduce the burden upon returning to work, and enable efficient information processing.

[0198] (Application Example 2)

[0199] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0200] When returning to work after vacation, individuals may be overwhelmed by the sheer volume of information, hindering their ability to quickly and efficiently resume their duties. Furthermore, when using public services, citizens may lack prior knowledge of waiting times and congestion levels, hindering their efficient use of resources. These situations increase psychological burden and contribute to overall decreased work efficiency. Therefore, there is a need to provide means that enable users to act smoothly and with less burden when using work and daily public services.

[0201] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0202] In this invention, the server includes means for collecting communication information received from a communication terminal during vacation, means for summarizing important information from the communication information using artificial intelligence technology, and means for performing emotion recognition and adjusting the order in which information is presented to reduce psychological burden. This supports users in efficiently returning to work and enables optimization of the timing of users' use of public functions.

[0203] A "communication terminal" is an electronic device capable of sending and receiving information, and is a device used by users to acquire and transmit information.

[0204] "Communication information" refers to information sent and received in digital format, such as emails and messages.

[0205] "Artificial intelligence technology" refers to technologies that enable computer systems to learn, reason, and solve problems in a manner similar to human intelligence.

[0206] "Emotion recognition" is a technology that analyzes data such as a user's facial expressions and voice to determine their emotions and psychological state.

[0207] "Psychological burden" refers to mental stress and pressure, which can be a cause of information overload and hinder efficient action.

[0208] "Adjusting the order in which information is presented" means changing the display order of information so that users can process it effectively.

[0209] A "server" is a computer system that processes and provides information on a network in response to client requests.

[0210] "Public functions" refer to institutions and facilities that provide public services, such as administrative offices and public facilities.

[0211] "Usage data" refers to data that shows the current usage status of facilities and services, such as the level of congestion and the number of users.

[0212] "Recommended information" refers to instructions and advice provided to help users take the best course of action.

[0213] The system implementing this invention primarily consists of a server, communication terminals, and information from public functions. The server summarizes communication information collected from the communication terminals using artificial intelligence technology and performs user emotion recognition. This allows users to obtain timely information suitable for using business and public functions while minimizing psychological burden.

[0214] The server performs these processes over the network. The software used includes a backend application written in Python and Google Cloud's AutoML, and scalable data processing is possible by leveraging Amazon Web Services (AWS®) cloud services.

[0215] As a concrete example, consider a scenario where users need to know in advance about waiting times and congestion levels when visiting a government office. Data collected by the user's communication device is uploaded to a server, where it undergoes sentiment analysis, and information regarding congestion levels is prioritized and summarized. As a result, the user is recommended the optimal time to visit via the application.

[0216] An example of a prompt that effectively utilizes a generative AI model is, "Please provide a wait time summary to optimize the timing of your visit, taking into account the current emotional state." This prompt allows the AI ​​model to properly recognize emotions and summarize information, enabling an improved user experience.

[0217] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0218] Step 1:

[0219] The terminal collects communication information. The user grants permission to the terminal, which then collects emails and messages and sends this data to the server. The input data is emails and messages, and the output is the data sent to the server. Specifically, the user launches the terminal application and performs an operation to authorize data collection.

[0220] Step 2:

[0221] The server summarizes communication information using artificial intelligence technology. The server analyzes the received data and uses a generative AI model to summarize important information. The input is data from emails and messages sent from terminals, and the output is summarized information. Specifically, the server runs Python on the cloud and uses Google Cloud's AutoML to summarize the data.

[0222] Step 3:

[0223] The server recognizes the user's emotions. The server receives data from the user's voice and facial expressions sent from the terminal and performs emotion recognition. The input is data from the user's facial expressions and voice, and the output is information about the user's emotional state. Specifically, the server executes an emotion analysis algorithm to evaluate the user's psychological state.

[0224] Step 4:

[0225] The server adjusts the order in which information is presented. Based on the emotion recognition results, the server adjusts the priority of the summarized information and presents it in the optimal order to minimize user stress. The input is information about the emotional state, and the output is the adjusted summarized information. Specifically, the server rearranges the information and prepares it for transmission to the terminal.

[0226] Step 5:

[0227] The terminal provides the user with customized information. The terminal displays the customized information received from the server on the user's screen. The input is the summarized information sent from the server, and the output is the information display provided to the user. Specifically, the terminal displays a notification and prepares an interface for the user to confirm the information.

[0228] This entire process allows users to process information efficiently and stress-free, optimizing their use of public services and other functions.

[0229] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0230] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0231] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0232] [Second Embodiment]

[0233] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0234] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0235] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0236] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0237] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0238] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0239] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0240] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0241] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0242] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0243] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0244] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0245] This invention is an information processing system designed to enable users to efficiently return to work after taking extended leave. The following provides a detailed description of an embodiment of this system.

[0246] First, users need to configure their system access permissions in advance to allow data processing during their vacation period. This allows the server to collect necessary information from the mail server and communication tools while the user is away.

[0247] The server accesses the user's mailbox and collects all unread emails. Similarly, it collects messages received during the holiday from communication tools. The server efficiently retrieves this information using APIs and protocols.

[0248] Next, the server uploads audio and video data from the terminals to the cloud as needed, and converts it into text data using speech recognition technology. This converted text data serves as the basis for understanding the meeting results and important discussion points.

[0249] The server uses generative artificial intelligence technology to generate summaries from the collected text and character data. In generating these summaries, elements such as project progress and important action items are extracted from emails and messages and summarized concisely. Sentiment analysis is also performed simultaneously, and the system is configured to prioritize notifying users of particularly negative content.

[0250] The summarized information is provided in a customized format based on each user's work history and project status. For example, users who prioritize information related to a specific project will see a summary of that project first.

[0251] Sufficient consideration has been given to the security of information; the servers encrypt the data they collect and store, and strict access controls are in place to prevent unauthorized access and information leaks by third parties.

[0252] As a concrete example, consider a user returning to work after a two-week vacation. During this time, there are 100 emails, 20 messages, and 5 online meeting recordings. The server collects all of this data and uses generative artificial intelligence to create a summary. One summary might extract the crucial information, "Project X's deadline has been moved up," and present it to the user. As a result, the user can quickly grasp important changes and resume work efficiently.

[0253] In this way, users can reduce the stress of information overload even after returning from vacation and continue their work flow smoothly.

[0254] The following describes the processing flow.

[0255] Step 1:

[0256] Based on the user's pre-configured settings, the server connects to email servers and business communication tools via APIs to automatically collect unread emails and messages received during vacation.

[0257] Step 2:

[0258] The server converts the collected text data into a format that is easy for the natural language processing engine to analyze. The HTML portion of emails is converted to text format, the language of the message is detected, and preparations for analysis are made according to the language characteristics.

[0259] Step 3:

[0260] The device synchronizes the audio and video data of the online meeting to cloud storage. This prepares the server to apply speech recognition technology to generate text data from the audio data.

[0261] Step 4:

[0262] The server uses natural language processing to analyze the collected and transformed text data, extract key information, and generate a summary. It identifies key points and tasks and creates a concise summary.

[0263] Step 5:

[0264] The server performs sentiment analysis on the generated text data and summaries. It measures sentiment based on the content of emails and messages, and configures the system to prioritize notifying users of information containing negative sentiment.

[0265] Step 6:

[0266] The server compares the user's past work information and project data, and customizes the summarized information. It prioritizes and organizes information that is particularly relevant to the user's job, and performs filtering to present it in an individualized format.

[0267] Step 7:

[0268] The terminal displays customized summary information received from the server to the user in a dashboard format, helping the user easily access important information.

[0269] Step 8:

[0270] The server ensures the confidentiality and security of information by encrypting and securely storing all data and implementing strict access controls.

[0271] (Example 1)

[0272] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0273] In today's business environment, when users are absent due to extended leave, there is a need to efficiently organize large amounts of unread communication data and information, and to quickly return to work. Current methods risk information overload upon return, particularly important information being overlooked. This invention aims to provide a system that allows users to smoothly resume work after leave and efficiently grasp important information.

[0274] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0275] In this invention, the server includes means for collecting electronic communication data received from the user's computer during the vacation period, means for summarizing important information from the electronic communication data using generative artificial intelligence technology, and means for converting audio and video information into coded data, and analyzing the coded data to generate a summary. This enables the user to quickly and efficiently access important information and return to work even after their vacation.

[0276] A "user's computing device" refers to a computer or smart device used by an individual for sending, receiving, and processing data.

[0277] "Electronic communication data" refers to data including text, images, and other associated information sent and received through email and messaging applications.

[0278] "Generative artificial intelligence technology" refers to AI technology designed to analyze input information and automatically perform specific tasks such as summarizing and inference.

[0279] "Audio and video information" refers to human conversations, images, videos, and other multimedia data collected in these formats.

[0280] "Coded data" refers to data obtained by converting audio or video information into a text format that can be analyzed using algorithms.

[0281] "Analysis" is the process of examining collected data to extract specific information or patterns.

[0282] This information processing system is designed to help users smoothly return to work after taking time off during vacation. It utilizes computing power, online communication, and generative artificial intelligence technology to efficiently collect, process, and provide relevant information.

[0283] First, before taking a vacation, the user sets access permissions on the computing device they use so that the system can collect data. As a result, the server can connect to the user's computing device and automatically collect electronic communication data even during the vacation. For example, it collects data transmitted and received through email and messaging apps.

[0284] After the server collects this data, it transfers voice information and video information to the cloud and converts them into coded data using voice recognition and video analysis technologies. In this process, specific encoding technologies are used, and the converted data is analyzed by generative artificial intelligence technology. This AI technology has the function of summarizing important information from emails and chat messages, which helps the user quickly understand the information.

[0285] For example, consider the case where the user receives 50 emails and 10 meeting records during the vacation. The server collects all this information and creates a summary using an AI model. As the generated summary, specific information such as "The progress of Project Y is good" is extracted and provided to the user. This saves the user the trouble of making choices about information after the vacation and allows them to quickly resume work.

[0286] In addition, since this system can be customized based on the user's work content and project progress, optimized information is provided individually. Information related to projects that the user is particularly interested in seems to be provided with priority over other information.

[0287] As an example of the prompt text, input to the system is made in a form such as "Please summarize the project progress and important tasks from the emails and messages received during the vacation."

[0288] This invention provides an effective means for the user to efficiently resume work and comprehensively grasp important information without omission.

[0289] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0290] Step 1:

[0291] Users configure their system access permissions before going on vacation. They log in to their computing device and register their email account and API keys for business communication tools. This prepares the server to access the user's account and automatically collect data. The input requires user authentication information and API keys, and the output provides the access permission status.

[0292] Step 2:

[0293] The server connects to the user's computer and collects electronic communication data received during the vacation. Specifically, it retrieves email data using the IMAP protocol and collects message data using the communication tool's API. Electronic communication data is provided as input, and unread message content and metadata are stored as output.

[0294] Step 3:

[0295] The device uploads the user's voice and video information to the cloud. This information is then converted into coded data using speech recognition and video analysis technologies. Specifically, it converts audio files to MP3 format and video files to MP4 format, saving them to the cloud. The input is audio and video files, and the output is coded text data.

[0296] Step 4:

[0297] The server receives coded data from the cloud and uses a generative AI model to summarize key information. A specific algorithm is used to extract project progress information and important tasks from the coded data. The input is coded data, and the output is a concise summary.

[0298] Step 5:

[0299] The server customizes the generated summaries based on individual work content and project progress. It is configured to prioritize displaying project-related elements and also performs sentiment analysis to highlight negative information. The input is a summary text, and the output is a customized information display.

[0300] Step 6:

[0301] The server protects and securely manages all collected data using cryptographic techniques. Access restrictions are implemented to prevent unauthorized access to the data. Specifically, information stored in the database is encrypted using AES-256. The input is the data before encryption, and the output is the encrypted data.

[0302] (Application Example 1)

[0303] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0304] Users returning to work after extended leave tend to be overwhelmed by the large amount of communication data and new information accumulated during their time off. As a result, it is difficult for them to catch up with the workflow in a short period of time. To address this problem, there is a need for information processing systems that enable users to return to work quickly and efficiently. In the food delivery industry in particular, it is necessary to appropriately grasp the latest delivery instructions, customer feedback, and new store opening information.

[0305] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0306] In this invention, the server includes means for collecting communication data received during the user's vacation from the user's device, means for summarizing important information from the communication data using generative AI technology, and means for analyzing delivery instructions, customer opinions, and new store information within the region that have accumulated during the vacation and presenting important matters to the delivery staff. As a result, the user can efficiently return to the business process.

[0307] The "user's device" refers to a communication device used to collect data received during the vacation.

[0308] The "communication data" refers to a collection of information including emails, messages, voice data, and video data.

[0309] The "generative AI technology" refers to artificial intelligence technology used to summarize important information from data.

[0310] The "means for summarizing" refers to a process of extracting important information from the collected data and providing it in a concise form.

[0311] "Sentiment analysis" refers to an analysis method for identifying the sentiment within data and particularly emphasizing negative elements.

[0312] The "means for providing individually" refers to a process of customizing summary information according to the user's business content and providing efficient information.

[0313] The "delivery instruction" refers to specific commands or orders in the delivery business.

[0314] The "customer opinion" refers to feedback or impressions collected from customers who have used the service.

[0315] [[ID=A]]The "new store information" refers to detailed information about newly opened stores within the region.

[0316] The "delivery staff" refers to the person in charge of the delivery business in the food delivery service.

[0317] The system for implementing this invention consists of a server for data processing and a terminal used by the user. The server comprehensively manages communication data collected from the user's terminal during vacation, namely emails, messages, voice data, and video data. The collected data is extracted and summarized using generative AI technology to extract important information. The generated summary incorporates a mechanism that performs sentiment analysis to prioritize the extraction of negative information.

[0318] The server uses programming languages ​​such as Python for data processing and computation, and utilizes RESTful APIs and natural language processing libraries (e.g., spaCy, transformers). Data security is ensured by security software that encrypts data and controls access (e.g., SSL / TLS protocol).

[0319] As a concrete example, consider a scenario where a delivery driver returns from a two-week vacation. The server collects 30 delivery instructions, 15 customer comments, and 5 pieces of new store information received during the vacation, and summarizes them using a generative AI. Finally, targeted information such as "a new store has opened" and "positive customer feedback has been received" is presented to the delivery driver as important information.

[0320] An example of a specific input prompt for the generating AI model would be: "Summarize the key changes and performance metrics from the following datasets: List of customer feedback, List of new store information." This would allow delivery drivers to quickly understand the latest information and resume operations smoothly.

[0321] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0322] Step 1:

[0323] The server collects communication data received from the user's device during their vacation. Inputs include emails, messages, audio data, and video data, while output is an integrated dataset of these. The server retrieves the data via an API and stores it in a database.

[0324] Step 2:

[0325] The server converts audio and video data into text. Input is audio and video files, and output is text. Speech recognition software is used to generate text from audio, and text is extracted from video as needed.

[0326] Step 3:

[0327] The server analyzes the collected text information using AI technology and generates a summary. The input is the text information converted in step 2 and the collected emails and messages, and the output is the summarized text. A prompt sentence is entered into the AI ​​model, and it extracts important information to create a summary.

[0328] Step 4:

[0329] The server analyzes the sentiment of the summarized information and prioritizes extracting negative information. The input is the summarized text generated in step 3, and the output is the summarized text with negative elements emphasized. Natural language processing techniques are used to analyze the sentiment of the text.

[0330] Step 5:

[0331] The server personalizes the summary information based on the user's work content and provides it to the terminal. The input is the summary text highlighted in step 4, and the output is the personalized summary information. Information priorities are set according to the user's work history and project status.

[0332] Step 6:

[0333] Users view summary information through their terminals. The output is personalized summary information. Based on the presented summary, users can quickly grasp the current work situation and decide on their next actions.

[0334] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0335] This invention is an information processing system that enables users to quickly and efficiently return to work after a long vacation, and is particularly characterized by its optimization of information presentation, taking into account the user's emotions. Specific embodiments of the present invention are described below.

[0336] First, the user installs the system and authorizes data collection from their terminal device during their vacation. Based on this authorization, the server accesses email servers and business communication tools to collect unread emails and messages received. Furthermore, the terminal connects with online meeting systems to upload audio and video data to the cloud, which the server then converts into text data using speech recognition technology.

[0337] Next, the server uses generative artificial intelligence technology to summarize the content of the collected emails and messages. The summarization employs techniques to extract particularly important tasks and decisions and organize the information concisely. It also incorporates an emotion engine to analyze the emotional elements in the data and prioritize notifications if negative information is present.

[0338] Furthermore, this system can recognize the user's emotional state in real time. The terminal is equipped with sensors that detect the user's facial expressions and tone of voice, and the server uses this information to activate an emotion engine. If the user is stressed, the server adjusts the order in which summary information is presented, reducing the user's psychological burden by delaying less important information.

[0339] For example, if a user is overwhelmed by 100 unread emails after returning to work, the system will detect that the user's facial expression indicates stress and first display only the most important messages requiring immediate attention. Then, as the user's facial expression relaxes, it will sequentially display the other messages. This reduces the user's information processing burden and efficiently helps them return to work.

[0340] Furthermore, the server encrypts data and strictly controls access to ensure security, thereby managing user data safely. This configuration provides the present invention with an environment in which users can use the system with peace of mind.

[0341] By combining these functions, the present invention aims to alleviate the stress caused by information overload when users return to work after vacation, thereby creating a more comfortable business environment.

[0342] The following describes the processing flow.

[0343] Step 1:

[0344] Users configure system settings before their vacation, granting the server access to email servers and business communication tools. This enables data collection during their vacation.

[0345] Step 2:

[0346] The server connects to the user's mail server and business communication tools via API to collect all emails and messages received during the holiday period. The scope of collection can be adjusted as needed based on user preferences.

[0347] Step 3:

[0348] The device synchronizes the audio and video data of the online meeting to cloud storage. This prepares the server to use speech recognition technology to convert the audio data into text data.

[0349] Step 4:

[0350] The server generates summaries using generative artificial intelligence techniques from collected emails, messages, and converted text data. It extracts key points related to high-priority information and tasks, creating concise summaries.

[0351] Step 5:

[0352] The server uses an emotion engine to perform sentiment analysis on the collected data. It identifies sections containing particularly negative emotions and prioritizes notifying the user, enabling quicker action.

[0353] Step 6:

[0354] The device uses sensors to detect the user's emotional state in real time and sends that information to a server. The system determines the user's stress level based on their facial expressions and tone of voice.

[0355] Step 7:

[0356] The server adjusts the order in which summary information is presented based on the user's emotional state. If the user is stressed, only important messages requiring immediate attention will be displayed first, while other information will be presented later.

[0357] Step 8:

[0358] The server encrypts and securely stores the data it collects and processes. Furthermore, strict access control is implemented to ensure that only authorized users can access the data.

[0359] Through this process, the system can support users in smoothly returning to work after vacation and reduce stress caused by information overload.

[0360] (Example 2)

[0361] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0362] In today's business environment, when users return from extended leave, they often face information overload due to a large volume of accumulated electronic communications and notifications. This situation raises concerns about user efficiency and mental burden upon returning to work. Furthermore, the presentation of information without considering the user's emotional state can increase stress upon returning to work.

[0363] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0364] In this invention, the server includes means for collecting electronic communications and notifications acquired from the user's information device during vacation; means for summarizing important elements from the electronic communications and notifications using a generative artificial intelligence method; means for converting audio and video information into text information, analyzing the text information, and creating a summary; means for performing sentiment analysis and prioritizing the presentation of negative elements; and means for detecting the user's emotional state and adjusting the order in which information is presented. This reduces the user's information burden and enables the presentation of information that is in line with their emotional state.

[0365] A "user" is an individual or organization that uses this system to manage information and efficiently process that information upon returning to work.

[0366] "Information device" is a general term for electronic devices that are operated by users to send, receive, and process data.

[0367] "Electronic communications" refers to digital information, such as emails and messages, that is sent and received over a network.

[0368] A "notification" is a means of communication from a system to inform a user of specific information or a message.

[0369] "Generative artificial intelligence methods" are artificial intelligence technologies that analyze data and use the results to create new summaries and suggestions.

[0370] A "summary" is a concise compilation of information, extracting the most important elements from the data.

[0371] "Audio information" refers to data that records sounds emitted as human voices in digital format.

[0372] "Visual information" refers to data recorded in digital format, consisting of visually represented moving images and still images.

[0373] "Text information" refers to information expressed as a string of characters, including character data converted from audio and video.

[0374] "Analysis" is the process of evaluating and interpreting data according to a specific purpose to derive valuable information.

[0375] "Emotional analysis" is a technology that infers and judges an individual's emotions and psychological state from data.

[0376] "Negative elements" refer to information that may cause discomfort or stress to the user.

[0377] "Emotional state" refers to a user's psychological and emotional condition, and is a factor that influences their daily activities.

[0378] "Adjusting the order in which information is presented" means changing the order in which information is presented in order to make it more efficient and effective depending on the user's situation.

[0379] This invention is an information processing system designed to facilitate information processing in the workplace when a user returns from a long leave of absence. The system aims to efficiently collect, summarize, and present information related to business communication while taking into account the user's emotional state.

[0380] First, users access the system using their own information devices and authorize the collection of electronic communications and notifications received during their vacation. These information devices include computers, smartphones, and tablet devices, which are used to operate the system.

[0381] Next, the server efficiently collects electronic communications and notifications based on information authorized by the user. This communication data includes emails and chat messages. The collected data is further processed within the server.

[0382] The server uses generative artificial intelligence techniques to automatically summarize key elements from the collected information. In this process, particularly important tasks and decisions are extracted, prioritizing information useful to the user. The summarization is performed using generative AI technologies such as OpenAI models.

[0383] The device detects the user's emotional state in real time based on factors such as skin color and voice tone. This allows the server to determine whether the user is stressed. If the user indicates stress, the server adjusts the order in which information is presented, prioritizing less important information to reduce psychological burden.

[0384] Furthermore, audio and video information are also processed. The server can convert audio data used in online meetings into text using Google's speech recognition technology, and then analyze it to provide useful information.

[0385] For example, if a user is overwhelmed by a large number of unread emails after returning to work, this system will immediately prioritize and display the most urgent emails. This allows the user to efficiently resume their work.

[0386] Examples of prompt statements to input into a generative AI model include the following:

[0387] "To help users returning to work after extended leave efficiently process unread emails, devise a method to summarize and prioritize important emails. Consider the user's current emotional state and propose an appropriate order for presenting the information."

[0388] In this way, the present invention helps users reduce information overload while using the system with peace of mind, and supports a smooth return to work.

[0389] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0390] Step 1:

[0391] The user logs into their information device and authorizes the system to collect electronic communications and notifications received during their vacation. Input requires the user's authentication information and permission settings. Output is the authorized access information sent to the server. Based on this access information, the server determines the user's permitted scope.

[0392] Step 2:

[0393] The server uses access information obtained from users to collect electronic communications and notifications into the system. Specifically, it calls APIs of mail servers and chat tools to retrieve unread messages. The inputs used are user access information and API information of the target servers. As output, the server stores unread communication data.

[0394] Step 3:

[0395] The server uses a generative AI model to perform summarization processing to extract important elements from the collected electronic communication data. The communication data collected in step 2 is required as input. Summarized information is generated as output. Here, the AI ​​model prioritizes extracting tasks and important decisions within the emails.

[0396] Step 4:

[0397] The device collects user facial expression data and voice tone to detect the user's emotional state in real time. Sensor data from the camera and microphone are used as input. The output is the user's emotion analysis result, which is sent to the server. Based on this information, the server determines the user's emotional state.

[0398] Step 5:

[0399] The server adjusts the order in which information is presented based on the user's emotional state. If the user is showing signs of stress, the server re-evaluates the information to be presented preferentially and displays the most important content first. Summarized information and the results of the emotional analysis are required as input. The adjusted order of information presentation is determined as output and sent to the terminal.

[0400] Step 6:

[0401] The terminal receives pre-configured information from the server and presents it to the user. The user can then resume their work based on this information. The input is ordered information from the server. The output is a visual display of information on the user's screen. Specifically, information is displayed in a way that suits the user's mood, improving work efficiency.

[0402] Through the steps outlined above, this system aims to prevent users from becoming overwhelmed with information, reduce the burden upon returning to work, and enable efficient information processing.

[0403] (Application Example 2)

[0404] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0405] When returning to work after vacation, individuals may be overwhelmed by the sheer volume of information, hindering their ability to quickly and efficiently resume their duties. Furthermore, when using public services, citizens may lack prior knowledge of waiting times and congestion levels, hindering their efficient use of resources. These situations increase psychological burden and contribute to overall decreased work efficiency. Therefore, there is a need to provide means that enable users to act smoothly and with less burden when using work and daily public services.

[0406] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0407] In this invention, the server includes means for collecting communication information received from a communication terminal during vacation, means for summarizing important information from the communication information using artificial intelligence technology, and means for performing emotion recognition and adjusting the order in which information is presented to reduce psychological burden. This supports users in efficiently returning to work and enables optimization of the timing of users' use of public functions.

[0408] A "communication terminal" is an electronic device capable of sending and receiving information, and is a device used by users to acquire and transmit information.

[0409] "Communication information" refers to information sent and received in digital format, such as emails and messages.

[0410] "Artificial intelligence technology" refers to technologies that enable computer systems to learn, reason, and solve problems in a manner similar to human intelligence.

[0411] "Emotion recognition" is a technology that analyzes data such as a user's facial expressions and voice to determine their emotions and psychological state.

[0412] "Psychological burden" refers to mental stress and pressure, which can be a cause of information overload and hinder efficient action.

[0413] "Adjusting the order in which information is presented" means changing the display order of information so that users can process it effectively.

[0414] A "server" is a computer system that processes and provides information on a network in response to client requests.

[0415] "Public functions" refer to institutions and facilities that provide public services, such as administrative offices and public facilities.

[0416] "Usage data" refers to data that shows the current usage status of facilities and services, such as the level of congestion and the number of users.

[0417] "Recommended information" refers to instructions and advice provided to help users take the best course of action.

[0418] The system implementing this invention primarily consists of a server, communication terminals, and information from public functions. The server summarizes communication information collected from the communication terminals using artificial intelligence technology and performs user emotion recognition. This allows users to obtain timely information suitable for using business and public functions while minimizing psychological burden.

[0419] The server performs these processes over the network. The software used includes a backend application written in Python and Google Cloud's AutoML, and scalable data processing is possible by leveraging Amazon Web Services (AWS) cloud services.

[0420] As a concrete example, consider a scenario where users need to know in advance about waiting times and congestion levels when visiting a government office. Data collected by the user's communication device is uploaded to a server, where it undergoes sentiment analysis, and information regarding congestion levels is prioritized and summarized. As a result, the user is recommended the optimal time to visit via the application.

[0421] An example of a prompt that effectively utilizes a generative AI model is, "Please provide a wait time summary to optimize the timing of your visit, taking into account the current emotional state." This prompt allows the AI ​​model to properly recognize emotions and summarize information, enabling an improved user experience.

[0422] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0423] Step 1:

[0424] The terminal collects communication information. The user grants permission to the terminal, which then collects emails and messages and sends this data to the server. The input data is emails and messages, and the output is the data sent to the server. Specifically, the user launches the terminal application and performs an operation to authorize data collection.

[0425] Step 2:

[0426] The server summarizes communication information using artificial intelligence technology. The server analyzes the received data and uses a generative AI model to summarize important information. The input is data from emails and messages sent from terminals, and the output is summarized information. Specifically, the server runs Python on the cloud and uses Google Cloud's AutoML to summarize the data.

[0427] Step 3:

[0428] The server recognizes the user's emotions. The server receives data from the user's voice and facial expressions sent from the terminal and performs emotion recognition. The input is data from the user's facial expressions and voice, and the output is information about the user's emotional state. Specifically, the server executes an emotion analysis algorithm to evaluate the user's psychological state.

[0429] Step 4:

[0430] The server adjusts the order in which information is presented. Based on the emotion recognition results, the server adjusts the priority of the summarized information and presents it in the optimal order to minimize user stress. The input is information about the emotional state, and the output is the adjusted summarized information. Specifically, the server rearranges the information and prepares it for transmission to the terminal.

[0431] Step 5:

[0432] The terminal provides the user with customized information. The terminal displays the customized information received from the server on the user's screen. The input is the summarized information sent from the server, and the output is the information display provided to the user. Specifically, the terminal displays a notification and prepares an interface for the user to confirm the information.

[0433] This entire process allows users to process information efficiently and stress-free, optimizing their use of public services and other functions.

[0434] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0435] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0436] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0437] [Third Embodiment]

[0438] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0439] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0440] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0441] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0442] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0443] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0444] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0445] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0446] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0447] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0448] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0449] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0450] This invention is an information processing system designed to enable users to efficiently return to work after taking extended leave. The following provides a detailed description of an embodiment of this system.

[0451] First, users need to configure their system access permissions in advance to allow data processing during their vacation period. This allows the server to collect necessary information from the mail server and communication tools while the user is away.

[0452] The server accesses the user's mailbox and collects all unread emails. Similarly, it collects messages received during the holiday from communication tools. The server efficiently retrieves this information using APIs and protocols.

[0453] Next, the server uploads audio and video data from the terminals to the cloud as needed, and converts it into text data using speech recognition technology. This converted text data serves as the basis for understanding the meeting results and important discussion points.

[0454] The server uses generative artificial intelligence technology to generate summaries from the collected text and character data. In generating these summaries, elements such as project progress and important action items are extracted from emails and messages and summarized concisely. Sentiment analysis is also performed simultaneously, and the system is configured to prioritize notifying users of particularly negative content.

[0455] The summarized information is provided in a customized format based on each user's work history and project status. For example, users who prioritize information related to a specific project will see a summary of that project first.

[0456] Sufficient consideration has been given to the security of information; the servers encrypt the data they collect and store, and strict access controls are in place to prevent unauthorized access and information leaks by third parties.

[0457] As a concrete example, consider a user returning to work after a two-week vacation. During this time, there are 100 emails, 20 messages, and 5 online meeting recordings. The server collects all of this data and uses generative artificial intelligence to create a summary. One summary might extract the crucial information, "Project X's deadline has been moved up," and present it to the user. As a result, the user can quickly grasp important changes and resume work efficiently.

[0458] In this way, users can reduce the stress of information overload even after returning from vacation and continue their work flow smoothly.

[0459] The following describes the processing flow.

[0460] Step 1:

[0461] Based on the user's pre-configured settings, the server connects to email servers and business communication tools via APIs to automatically collect unread emails and messages received during vacation.

[0462] Step 2:

[0463] The server converts the collected text data into a format that is easy for the natural language processing engine to analyze. The HTML portion of emails is converted to text format, the language of the message is detected, and preparations for analysis are made according to the language characteristics.

[0464] Step 3:

[0465] The device synchronizes the audio and video data of the online meeting to cloud storage. This prepares the server to apply speech recognition technology to generate text data from the audio data.

[0466] Step 4:

[0467] The server uses natural language processing to analyze the collected and transformed text data, extract key information, and generate a summary. It identifies key points and tasks and creates a concise summary.

[0468] Step 5:

[0469] The server performs sentiment analysis on the generated text data and summaries. It measures sentiment based on the content of emails and messages, and configures the system to prioritize notifying users of information containing negative sentiment.

[0470] Step 6:

[0471] The server compares the user's past work information and project data, and customizes the summarized information. It prioritizes and organizes information that is particularly relevant to the user's job, and performs filtering to present it in an individualized format.

[0472] Step 7:

[0473] The terminal displays customized summary information received from the server to the user in a dashboard format, helping the user easily access important information.

[0474] Step 8:

[0475] The server ensures the confidentiality and security of information by encrypting and securely storing all data and implementing strict access controls.

[0476] (Example 1)

[0477] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0478] In today's business environment, when users are absent due to extended leave, there is a need to efficiently organize large amounts of unread communication data and information, and to quickly return to work. Current methods risk information overload upon return, particularly important information being overlooked. This invention aims to provide a system that allows users to smoothly resume work after leave and efficiently grasp important information.

[0479] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0480] In this invention, the server includes means for collecting electronic communication data received from the user's computer during the vacation period, means for summarizing important information from the electronic communication data using generative artificial intelligence technology, and means for converting audio and video information into coded data, and analyzing the coded data to generate a summary. This enables the user to quickly and efficiently access important information and return to work even after their vacation.

[0481] A "user's computing device" refers to a computer or smart device used by an individual for sending, receiving, and processing data.

[0482] "Electronic communication data" refers to data including text, images, and other associated information sent and received through email and messaging applications.

[0483] "Generative artificial intelligence technology" refers to AI technology designed to analyze input information and automatically perform specific tasks such as summarizing and inference.

[0484] "Audio and video information" refers to human conversations, images, videos, and other multimedia data collected in these formats.

[0485] "Coded data" refers to data obtained by converting audio or video information into a text format that can be analyzed using algorithms.

[0486] "Analysis" is the process of examining collected data to extract specific information or patterns.

[0487] This information processing system is designed to help users smoothly return to work after taking time off during vacation. It utilizes computing power, online communication, and generative artificial intelligence technology to efficiently collect, process, and provide relevant information.

[0488] First, before taking a vacation, users configure access permissions on their computing device to allow the system to collect data. This enables the server to connect to the user's computer and automatically collect electronic communication data even while they are on vacation. For example, it can collect data sent and received via email and messaging apps.

[0489] After collecting this data, the server transfers the audio and video information to the cloud and converts it into coded data using speech recognition and video analysis technologies. This process uses specific encoding techniques, and the converted data is analyzed by generative artificial intelligence technology. This AI technology has the ability to summarize important information from emails and chat messages, helping users quickly understand the information.

[0490] For example, consider a scenario where a user receives 50 emails and 10 meeting minutes while on vacation. The server collects all this information and uses an AI model to create a summary. The resulting summary extracts specific information such as "Project Y is progressing well" and provides it to the user. This saves the user the trouble of sifting through information upon returning from vacation, allowing them to quickly return to work.

[0491] Furthermore, this system can be customized based on the user's work content and project progress, providing individually optimized information. Information about projects of particular interest to the user appears to be prioritized over other information.

[0492] An example of a prompt message would be something like, "Please summarize project progress and important tasks from emails and messages received during your vacation," which would then be entered into the system.

[0493] This invention provides an effective means for users to efficiently return to their work and grasp important information without omission.

[0494] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0495] Step 1:

[0496] Users configure their system access permissions before going on vacation. They log in to their computing device and register their email account and API keys for business communication tools. This prepares the server to access the user's account and automatically collect data. The input requires user authentication information and API keys, and the output provides the access permission status.

[0497] Step 2:

[0498] The server connects to the user's computer and collects electronic communication data received during the vacation. Specifically, it retrieves email data using the IMAP protocol and collects message data using the communication tool's API. Electronic communication data is provided as input, and unread message content and metadata are stored as output.

[0499] Step 3:

[0500] The device uploads the user's voice and video information to the cloud. This information is then converted into coded data using speech recognition and video analysis technologies. Specifically, it converts audio files to MP3 format and video files to MP4 format, saving them to the cloud. The input is audio and video files, and the output is coded text data.

[0501] Step 4:

[0502] The server receives coded data from the cloud and uses a generative AI model to summarize key information. A specific algorithm is used to extract project progress information and important tasks from the coded data. The input is coded data, and the output is a concise summary.

[0503] Step 5:

[0504] The server customizes the generated summaries based on individual work content and project progress. It is configured to prioritize displaying project-related elements and also performs sentiment analysis to highlight negative information. The input is a summary text, and the output is a customized information display.

[0505] Step 6:

[0506] The server protects and securely manages all collected data using cryptographic techniques. Access restrictions are implemented to prevent unauthorized access to the data. Specifically, information stored in the database is encrypted using AES-256. The input is the data before encryption, and the output is the encrypted data.

[0507] (Application Example 1)

[0508] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0509] Users returning to work after extended leave tend to be overwhelmed by the large amount of communication data and new information accumulated during their time off. As a result, it is difficult for them to catch up with the workflow in a short period of time. To address this problem, there is a need for information processing systems that enable users to return to work quickly and efficiently. In the food delivery industry in particular, it is necessary to appropriately grasp the latest delivery instructions, customer feedback, and new store opening information.

[0510] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0511] In this invention, the server includes means for collecting communication data received from the user's device during the vacation, means for summarizing important information from the communication data using AI technology, and means for analyzing delivery instructions, customer feedback, and new store information in the area that have accumulated during the vacation, and presenting important information to the delivery personnel. This enables the user to efficiently return to the workflow.

[0512] "User's equipment" refers to communication devices used to collect data received during vacation.

[0513] "Communication data" refers to a collection of information, including emails, messages, audio data, and video data.

[0514] "Generative AI technology" refers to artificial intelligence technology used to summarize important information from data.

[0515] "Methods of summarization" refer to the process of extracting important information from collected data and presenting it in a concise format.

[0516] "Sentiment analysis" refers to analytical techniques that identify emotions within data and highlight negative elements in particular.

[0517] "Personalized delivery methods" refer to the process of customizing summary information according to the user's work content to provide information efficiently.

[0518] "Delivery instructions" refers to specific directives or orders used in delivery operations.

[0519] "Customer feedback" refers to the feedback and opinions collected from customers who have used the service.

[0520] "New store information" refers to detailed information about stores that have recently opened in the area.

[0521] A "delivery driver" refers to a person responsible for delivery services in food delivery services.

[0522] The system for implementing this invention consists of a server for data processing and a terminal used by the user. The server comprehensively manages communication data collected from the user's terminal during vacation, namely emails, messages, voice data, and video data. The collected data is extracted and summarized using generative AI technology to extract important information. The generated summary incorporates a mechanism that performs sentiment analysis to prioritize the extraction of negative information.

[0523] The server uses programming languages ​​such as Python for data processing and computation, and utilizes RESTful APIs and natural language processing libraries (e.g., spaCy, transformers). Data security is ensured by security software that encrypts data and controls access (e.g., SSL / TLS protocol).

[0524] As a concrete example, consider a scenario where a delivery driver returns from a two-week vacation. The server collects 30 delivery instructions, 15 customer comments, and 5 pieces of new store information received during the vacation, and summarizes them using a generative AI. Finally, targeted information such as "a new store has opened" and "positive customer feedback has been received" is presented to the delivery driver as important information.

[0525] An example of a specific input prompt for the generating AI model would be: "Summarize the key changes and performance metrics from the following datasets: List of customer feedback, List of new store information." This would allow delivery drivers to quickly understand the latest information and resume operations smoothly.

[0526] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0527] Step 1:

[0528] The server collects communication data received from the user's device during their vacation. Inputs include emails, messages, audio data, and video data, while output is an integrated dataset of these. The server retrieves the data via an API and stores it in a database.

[0529] Step 2:

[0530] The server converts audio and video data into text. Input is audio and video files, and output is text. Speech recognition software is used to generate text from audio, and text is extracted from video as needed.

[0531] Step 3:

[0532] The server analyzes the collected text information using AI technology and generates a summary. The input is the text information converted in step 2 and the collected emails and messages, and the output is the summarized text. A prompt sentence is entered into the AI ​​model, and it extracts important information to create a summary.

[0533] Step 4:

[0534] The server analyzes the sentiment of the summarized information and prioritizes extracting negative information. The input is the summarized text generated in step 3, and the output is the summarized text with negative elements emphasized. Natural language processing techniques are used to analyze the sentiment of the text.

[0535] Step 5:

[0536] The server personalizes the summary information based on the user's work content and provides it to the terminal. The input is the summary text highlighted in step 4, and the output is the personalized summary information. Information priorities are set according to the user's work history and project status.

[0537] Step 6:

[0538] Users view summary information through their terminals. The output is personalized summary information. Based on the presented summary, users can quickly grasp the current work situation and decide on their next actions.

[0539] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0540] This invention is an information processing system that enables users to quickly and efficiently return to work after a long vacation, and is particularly characterized by its optimization of information presentation, taking into account the user's emotions. Specific embodiments of the present invention are described below.

[0541] First, the user installs the system and authorizes data collection from their terminal device during their vacation. Based on this authorization, the server accesses email servers and business communication tools to collect unread emails and messages received. Furthermore, the terminal connects with online meeting systems to upload audio and video data to the cloud, which the server then converts into text data using speech recognition technology.

[0542] Next, the server uses generative artificial intelligence technology to summarize the content of the collected emails and messages. The summarization employs techniques to extract particularly important tasks and decisions and organize the information concisely. It also incorporates an emotion engine to analyze the emotional elements in the data and prioritize notifications if negative information is present.

[0543] Furthermore, this system can recognize the user's emotional state in real time. The terminal is equipped with sensors that detect the user's facial expressions and tone of voice, and the server uses this information to activate an emotion engine. If the user is stressed, the server adjusts the order in which summary information is presented, reducing the user's psychological burden by delaying less important information.

[0544] For example, if a user is overwhelmed by 100 unread emails after returning to work, the system will detect that the user's facial expression indicates stress and first display only the most important messages requiring immediate attention. Then, as the user's facial expression relaxes, it will sequentially display the other messages. This reduces the user's information processing burden and efficiently helps them return to work.

[0545] Furthermore, the server encrypts data and strictly controls access to ensure security, thereby managing user data safely. This configuration provides the present invention with an environment in which users can use the system with peace of mind.

[0546] By combining these functions, the present invention aims to alleviate the stress caused by information overload when users return to work after vacation, thereby creating a more comfortable business environment.

[0547] The following describes the processing flow.

[0548] Step 1:

[0549] Users configure system settings before their vacation, granting the server access to email servers and business communication tools. This enables data collection during their vacation.

[0550] Step 2:

[0551] The server connects to the user's mail server and business communication tools via API to collect all emails and messages received during the holiday period. The scope of collection can be adjusted as needed based on user preferences.

[0552] Step 3:

[0553] The device synchronizes the audio and video data of the online meeting to cloud storage. This prepares the server to use speech recognition technology to convert the audio data into text data.

[0554] Step 4:

[0555] The server generates summaries using generative artificial intelligence techniques from collected emails, messages, and converted text data. It extracts key points related to high-priority information and tasks, creating concise summaries.

[0556] Step 5:

[0557] The server uses an emotion engine to perform sentiment analysis on the collected data. It identifies sections containing particularly negative emotions and prioritizes notifying the user, enabling quicker action.

[0558] Step 6:

[0559] The device uses sensors to detect the user's emotional state in real time and sends that information to a server. The system determines the user's stress level based on their facial expressions and tone of voice.

[0560] Step 7:

[0561] The server adjusts the order in which summary information is presented based on the user's emotional state. If the user is stressed, only important messages requiring immediate attention will be displayed first, while other information will be presented later.

[0562] Step 8:

[0563] The server encrypts and securely stores the data it collects and processes. Furthermore, strict access control is implemented to ensure that only authorized users can access the data.

[0564] Through this process, the system can support users in smoothly returning to work after vacation and reduce stress caused by information overload.

[0565] (Example 2)

[0566] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0567] In today's business environment, when users return from extended leave, they often face information overload due to a large volume of accumulated electronic communications and notifications. This situation raises concerns about user efficiency and mental burden upon returning to work. Furthermore, the presentation of information without considering the user's emotional state can increase stress upon returning to work.

[0568] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0569] In this invention, the server includes means for collecting electronic communications and notifications acquired from the user's information device during vacation; means for summarizing important elements from the electronic communications and notifications using a generative artificial intelligence method; means for converting audio and video information into text information, analyzing the text information, and creating a summary; means for performing sentiment analysis and prioritizing the presentation of negative elements; and means for detecting the user's emotional state and adjusting the order in which information is presented. This reduces the user's information burden and enables the presentation of information that is in line with their emotional state.

[0570] A "user" is an individual or organization that uses this system to manage information and efficiently process that information upon returning to work.

[0571] "Information device" is a general term for electronic devices that are operated by users to send, receive, and process data.

[0572] "Electronic communications" refers to digital information, such as emails and messages, that is sent and received over a network.

[0573] A "notification" is a means of communication from a system to inform a user of specific information or a message.

[0574] "Generative artificial intelligence methods" are artificial intelligence technologies that analyze data and use the results to create new summaries and suggestions.

[0575] A "summary" is a concise compilation of information, extracting the most important elements from the data.

[0576] "Audio information" refers to data that records sounds emitted as human voices in digital format.

[0577] "Visual information" refers to data recorded in digital format, consisting of visually represented moving images and still images.

[0578] "Text information" refers to information expressed as a string of characters, including character data converted from audio and video.

[0579] "Analysis" is the process of evaluating and interpreting data according to a specific purpose to derive valuable information.

[0580] "Emotional analysis" is a technology that infers and judges an individual's emotions and psychological state from data.

[0581] "Negative elements" refer to information that may cause discomfort or stress to the user.

[0582] "Emotional state" refers to a user's psychological and emotional condition, and is a factor that influences their daily activities.

[0583] "Adjusting the order in which information is presented" means changing the order in which information is presented in order to make it more efficient and effective depending on the user's situation.

[0584] This invention is an information processing system designed to facilitate information processing in the workplace when a user returns from a long leave of absence. The system aims to efficiently collect, summarize, and present information related to business communication while taking into account the user's emotional state.

[0585] First, users access the system using their own information devices and authorize the collection of electronic communications and notifications received during their vacation. These information devices include computers, smartphones, and tablet devices, which are used to operate the system.

[0586] Next, the server efficiently collects electronic communications and notifications based on information authorized by the user. This communication data includes emails and chat messages. The collected data is further processed within the server.

[0587] The server uses generative artificial intelligence techniques to automatically summarize key elements from the collected information. In this process, particularly important tasks and decisions are extracted, prioritizing information useful to the user. The summarization is performed using generative AI technologies such as OpenAI models.

[0588] The device detects the user's emotional state in real time based on factors such as skin color and voice tone. This allows the server to determine whether the user is stressed. If the user indicates stress, the server adjusts the order in which information is presented, prioritizing less important information to reduce psychological burden.

[0589] Furthermore, audio and video information are also processed. The server can convert audio data used in online meetings into text using Google's speech recognition technology, and then analyze it to provide useful information.

[0590] For example, if a user is overwhelmed by a large number of unread emails after returning to work, this system will immediately prioritize and display the most urgent emails. This allows the user to efficiently resume their work.

[0591] Examples of prompt statements to input into a generative AI model include the following:

[0592] "To help users returning to work after extended leave efficiently process unread emails, devise a method to summarize and prioritize important emails. Consider the user's current emotional state and propose an appropriate order for presenting the information."

[0593] In this way, the present invention helps users reduce information overload while using the system with peace of mind, and supports a smooth return to work.

[0594] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0595] Step 1:

[0596] The user logs into their information device and authorizes the system to collect electronic communications and notifications received during their vacation. Input requires the user's authentication information and permission settings. Output is the authorized access information sent to the server. Based on this access information, the server determines the user's permitted scope.

[0597] Step 2:

[0598] The server uses access information obtained from users to collect electronic communications and notifications into the system. Specifically, it calls APIs of mail servers and chat tools to retrieve unread messages. The inputs used are user access information and API information of the target servers. As output, the server stores unread communication data.

[0599] Step 3:

[0600] The server uses a generative AI model to perform summarization processing to extract important elements from the collected electronic communication data. The communication data collected in step 2 is required as input. Summarized information is generated as output. Here, the AI ​​model prioritizes extracting tasks and important decisions within the emails.

[0601] Step 4:

[0602] The device collects user facial expression data and voice tone to detect the user's emotional state in real time. Sensor data from the camera and microphone are used as input. The output is the user's emotion analysis result, which is sent to the server. Based on this information, the server determines the user's emotional state.

[0603] Step 5:

[0604] The server adjusts the order in which information is presented based on the user's emotional state. If the user is showing signs of stress, the server re-evaluates the information to be presented preferentially and displays the most important content first. Summarized information and the results of the emotional analysis are required as input. The adjusted order of information presentation is determined as output and sent to the terminal.

[0605] Step 6:

[0606] The terminal receives pre-configured information from the server and presents it to the user. The user can then resume their work based on this information. The input is ordered information from the server. The output is a visual display of information on the user's screen. Specifically, information is displayed in a way that suits the user's mood, improving work efficiency.

[0607] Through the steps outlined above, this system aims to prevent users from becoming overwhelmed with information, reduce the burden upon returning to work, and enable efficient information processing.

[0608] (Application Example 2)

[0609] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0610] When returning to work after vacation, individuals may be overwhelmed by the sheer volume of information, hindering their ability to quickly and efficiently resume their duties. Furthermore, when using public services, citizens may lack prior knowledge of waiting times and congestion levels, hindering their efficient use of resources. These situations increase psychological burden and contribute to overall decreased work efficiency. Therefore, there is a need to provide means that enable users to act smoothly and with less burden when using work and daily public services.

[0611] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0612] In this invention, the server includes means for collecting communication information received from a communication terminal during vacation, means for summarizing important information from the communication information using artificial intelligence technology, and means for performing emotion recognition and adjusting the order in which information is presented to reduce psychological burden. This supports users in efficiently returning to work and enables optimization of the timing of users' use of public functions.

[0613] A "communication terminal" is an electronic device capable of sending and receiving information, and is a device used by users to acquire and transmit information.

[0614] "Communication information" refers to information sent and received in digital format, such as emails and messages.

[0615] "Artificial intelligence technology" refers to technologies that enable computer systems to learn, reason, and solve problems in a manner similar to human intelligence.

[0616] "Emotion recognition" is a technology that analyzes data such as a user's facial expressions and voice to determine their emotions and psychological state.

[0617] "Psychological burden" refers to mental stress and pressure, which can be a cause of information overload and hinder efficient action.

[0618] "Adjusting the order in which information is presented" means changing the display order of information so that users can process it effectively.

[0619] A "server" is a computer system that processes and provides information on a network in response to client requests.

[0620] "Public functions" refer to institutions and facilities that provide public services, such as administrative offices and public facilities.

[0621] "Usage data" refers to data that shows the current usage status of facilities and services, such as the level of congestion and the number of users.

[0622] "Recommended information" refers to instructions and advice provided to help users take the best course of action.

[0623] The system implementing this invention primarily consists of a server, communication terminals, and information from public functions. The server summarizes communication information collected from the communication terminals using artificial intelligence technology and performs user emotion recognition. This allows users to obtain timely information suitable for using business and public functions while minimizing psychological burden.

[0624] The server performs these processes over the network. The software used includes a backend application written in Python and Google Cloud's AutoML, and scalable data processing is possible by leveraging Amazon Web Services (AWS) cloud services.

[0625] As a concrete example, consider a scenario where users need to know in advance about waiting times and congestion levels when visiting a government office. Data collected by the user's communication device is uploaded to a server, where it undergoes sentiment analysis, and information regarding congestion levels is prioritized and summarized. As a result, the user is recommended the optimal time to visit via the application.

[0626] An example of a prompt that effectively utilizes a generative AI model is, "Please provide a wait time summary to optimize the timing of your visit, taking into account the current emotional state." This prompt allows the AI ​​model to properly recognize emotions and summarize information, enabling an improved user experience.

[0627] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0628] Step 1:

[0629] The terminal collects communication information. The user grants permission to the terminal, which then collects emails and messages and sends this data to the server. The input data is emails and messages, and the output is the data sent to the server. Specifically, the user launches the terminal application and performs an operation to authorize data collection.

[0630] Step 2:

[0631] The server summarizes communication information using artificial intelligence technology. The server analyzes the received data and uses a generative AI model to summarize important information. The input is data from emails and messages sent from terminals, and the output is summarized information. Specifically, the server runs Python on the cloud and uses Google Cloud's AutoML to summarize the data.

[0632] Step 3:

[0633] The server recognizes the user's emotions. The server receives data from the user's voice and facial expressions sent from the terminal and performs emotion recognition. The input is data from the user's facial expressions and voice, and the output is information about the user's emotional state. Specifically, the server executes an emotion analysis algorithm to evaluate the user's psychological state.

[0634] Step 4:

[0635] The server adjusts the order in which information is presented. Based on the emotion recognition results, the server adjusts the priority of the summarized information and presents it in the optimal order to minimize user stress. The input is information about the emotional state, and the output is the adjusted summarized information. Specifically, the server rearranges the information and prepares it for transmission to the terminal.

[0636] Step 5:

[0637] The terminal provides the user with customized information. The terminal displays the customized information received from the server on the user's screen. The input is the summarized information sent from the server, and the output is the information display provided to the user. Specifically, the terminal displays a notification and prepares an interface for the user to confirm the information.

[0638] This entire process allows users to process information efficiently and stress-free, optimizing their use of public services and other functions.

[0639] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0640] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0641] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0642] [Fourth Embodiment]

[0643] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0644] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0645] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0646] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0647] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0648] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0649] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0650] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0651] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0652] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0653] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0654] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0655] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0656] This invention is an information processing system designed to enable users to efficiently return to work after taking extended leave. The following provides a detailed description of an embodiment of this system.

[0657] First, users need to configure their system access permissions in advance to allow data processing during their vacation period. This allows the server to collect necessary information from the mail server and communication tools while the user is away.

[0658] The server accesses the user's mailbox and collects all unread emails. Similarly, it collects messages received during the holiday from communication tools. The server efficiently retrieves this information using APIs and protocols.

[0659] Next, the server uploads audio and video data from the terminals to the cloud as needed, and converts it into text data using speech recognition technology. This converted text data serves as the basis for understanding the meeting results and important discussion points.

[0660] The server uses generative artificial intelligence technology to generate summaries from the collected text and character data. In generating these summaries, elements such as project progress and important action items are extracted from emails and messages and summarized concisely. Sentiment analysis is also performed simultaneously, and the system is configured to prioritize notifying users of particularly negative content.

[0661] The summarized information is provided in a customized format based on each user's work history and project status. For example, users who prioritize information related to a specific project will see a summary of that project first.

[0662] Sufficient consideration has been given to the security of information; the servers encrypt the data they collect and store, and strict access controls are in place to prevent unauthorized access and information leaks by third parties.

[0663] As a concrete example, consider a user returning to work after a two-week vacation. During this time, there are 100 emails, 20 messages, and 5 online meeting recordings. The server collects all of this data and uses generative artificial intelligence to create a summary. One summary might extract the crucial information, "Project X's deadline has been moved up," and present it to the user. As a result, the user can quickly grasp important changes and resume work efficiently.

[0664] In this way, users can reduce the stress of information overload even after returning from vacation and continue their work flow smoothly.

[0665] The following describes the processing flow.

[0666] Step 1:

[0667] Based on the user's pre-configured settings, the server connects to email servers and business communication tools via APIs to automatically collect unread emails and messages received during vacation.

[0668] Step 2:

[0669] The server converts the collected text data into a format that is easy for the natural language processing engine to analyze. The HTML portion of emails is converted to text format, the language of the message is detected, and preparations for analysis are made according to the language characteristics.

[0670] Step 3:

[0671] The device synchronizes the audio and video data of the online meeting to cloud storage. This prepares the server to apply speech recognition technology to generate text data from the audio data.

[0672] Step 4:

[0673] The server uses natural language processing to analyze the collected and transformed text data, extract key information, and generate a summary. It identifies key points and tasks and creates a concise summary.

[0674] Step 5:

[0675] The server performs sentiment analysis on the generated text data and summaries. It measures sentiment based on the content of emails and messages, and configures the system to prioritize notifying users of information containing negative sentiment.

[0676] Step 6:

[0677] The server compares the user's past work information and project data, and customizes the summarized information. It prioritizes and organizes information that is particularly relevant to the user's job, and performs filtering to present it in an individualized format.

[0678] Step 7:

[0679] The terminal displays customized summary information received from the server to the user in a dashboard format, helping the user easily access important information.

[0680] Step 8:

[0681] The server ensures the confidentiality and security of information by encrypting and securely storing all data and implementing strict access controls.

[0682] (Example 1)

[0683] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0684] In today's business environment, when users are absent due to extended leave, there is a need to efficiently organize large amounts of unread communication data and information, and to quickly return to work. Current methods risk information overload upon return, particularly important information being overlooked. This invention aims to provide a system that allows users to smoothly resume work after leave and efficiently grasp important information.

[0685] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0686] In this invention, the server includes means for collecting electronic communication data received from the user's computer during the vacation period, means for summarizing important information from the electronic communication data using generative artificial intelligence technology, and means for converting audio and video information into coded data, and analyzing the coded data to generate a summary. This enables the user to quickly and efficiently access important information and return to work even after their vacation.

[0687] A "user's computing device" refers to a computer or smart device used by an individual for sending, receiving, and processing data.

[0688] "Electronic communication data" refers to data including text, images, and other associated information sent and received through email and messaging applications.

[0689] "Generative artificial intelligence technology" refers to AI technology designed to analyze input information and automatically perform specific tasks such as summarizing and inference.

[0690] "Audio and video information" refers to human conversations, images, videos, and other multimedia data collected in these formats.

[0691] "Coded data" refers to data obtained by converting audio or video information into a text format that can be analyzed using algorithms.

[0692] "Analysis" is the process of examining collected data to extract specific information or patterns.

[0693] This information processing system is designed to help users smoothly return to work after taking time off during vacation. It utilizes computing power, online communication, and generative artificial intelligence technology to efficiently collect, process, and provide relevant information.

[0694] First, before taking a vacation, users configure access permissions on their computing device to allow the system to collect data. This enables the server to connect to the user's computer and automatically collect electronic communication data even while they are on vacation. For example, it can collect data sent and received via email and messaging apps.

[0695] After collecting this data, the server transfers the audio and video information to the cloud and converts it into coded data using speech recognition and video analysis technologies. This process uses specific encoding techniques, and the converted data is analyzed by generative artificial intelligence technology. This AI technology has the ability to summarize important information from emails and chat messages, helping users quickly understand the information.

[0696] For example, consider a scenario where a user receives 50 emails and 10 meeting minutes while on vacation. The server collects all this information and uses an AI model to create a summary. The resulting summary extracts specific information such as "Project Y is progressing well" and provides it to the user. This saves the user the trouble of sifting through information upon returning from vacation, allowing them to quickly return to work.

[0697] Furthermore, this system can be customized based on the user's work content and project progress, providing individually optimized information. Information about projects of particular interest to the user appears to be prioritized over other information.

[0698] An example of a prompt message would be something like, "Please summarize project progress and important tasks from emails and messages received during your vacation," which would then be entered into the system.

[0699] This invention provides an effective means for users to efficiently return to their work and grasp important information without omission.

[0700] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0701] Step 1:

[0702] Users configure their system access permissions before going on vacation. They log in to their computing device and register their email account and API keys for business communication tools. This prepares the server to access the user's account and automatically collect data. The input requires user authentication information and API keys, and the output provides the access permission status.

[0703] Step 2:

[0704] The server connects to the user's computer and collects electronic communication data received during the vacation. Specifically, it retrieves email data using the IMAP protocol and collects message data using the communication tool's API. Electronic communication data is provided as input, and unread message content and metadata are stored as output.

[0705] Step 3:

[0706] The device uploads the user's voice and video information to the cloud. This information is then converted into coded data using speech recognition and video analysis technologies. Specifically, it converts audio files to MP3 format and video files to MP4 format, saving them to the cloud. The input is audio and video files, and the output is coded text data.

[0707] Step 4:

[0708] The server receives coded data from the cloud and uses a generative AI model to summarize key information. A specific algorithm is used to extract project progress information and important tasks from the coded data. The input is coded data, and the output is a concise summary.

[0709] Step 5:

[0710] The server customizes the generated summaries based on individual work content and project progress. It is configured to prioritize displaying project-related elements and also performs sentiment analysis to highlight negative information. The input is a summary text, and the output is a customized information display.

[0711] Step 6:

[0712] The server protects and securely manages all collected data using cryptographic techniques. Access restrictions are implemented to prevent unauthorized access to the data. Specifically, information stored in the database is encrypted using AES-256. The input is the data before encryption, and the output is the encrypted data.

[0713] (Application Example 1)

[0714] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0715] Users returning to work after extended leave tend to be overwhelmed by the large amount of communication data and new information accumulated during their time off. As a result, it is difficult for them to catch up with the workflow in a short period of time. To address this problem, there is a need for information processing systems that enable users to return to work quickly and efficiently. In the food delivery industry in particular, it is necessary to appropriately grasp the latest delivery instructions, customer feedback, and new store opening information.

[0716] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0717] In this invention, the server includes means for collecting communication data received from the user's device during the vacation, means for summarizing important information from the communication data using AI technology, and means for analyzing delivery instructions, customer feedback, and new store information in the area that have accumulated during the vacation, and presenting important information to the delivery personnel. This enables the user to efficiently return to the workflow.

[0718] "User's equipment" refers to communication devices used to collect data received during vacation.

[0719] "Communication data" refers to a collection of information, including emails, messages, audio data, and video data.

[0720] "Generative AI technology" refers to artificial intelligence technology used to summarize important information from data.

[0721] "Methods of summarization" refer to the process of extracting important information from collected data and presenting it in a concise format.

[0722] "Sentiment analysis" refers to analytical techniques that identify emotions within data and highlight negative elements in particular.

[0723] "Personalized delivery methods" refer to the process of customizing summary information according to the user's work content to provide information efficiently.

[0724] "Delivery instructions" refers to specific directives or orders used in delivery operations.

[0725] "Customer feedback" refers to the feedback and opinions collected from customers who have used the service.

[0726] "New store information" refers to detailed information about stores that have recently opened in the area.

[0727] A "delivery driver" refers to a person responsible for delivery services in food delivery services.

[0728] The system for implementing this invention consists of a server for data processing and a terminal used by the user. The server comprehensively manages communication data collected from the user's terminal during vacation, namely emails, messages, voice data, and video data. The collected data is extracted and summarized using generative AI technology to extract important information. The generated summary incorporates a mechanism that performs sentiment analysis to prioritize the extraction of negative information.

[0729] The server uses programming languages ​​such as Python for data processing and computation, and utilizes RESTful APIs and natural language processing libraries (e.g., spaCy, transformers). Data security is ensured by security software that encrypts data and controls access (e.g., SSL / TLS protocol).

[0730] As a concrete example, consider a scenario where a delivery driver returns from a two-week vacation. The server collects 30 delivery instructions, 15 customer comments, and 5 pieces of new store information received during the vacation, and summarizes them using a generative AI. Finally, targeted information such as "a new store has opened" and "positive customer feedback has been received" is presented to the delivery driver as important information.

[0731] An example of a specific input prompt for the generating AI model would be: "Summarize the key changes and performance metrics from the following datasets: List of customer feedback, List of new store information." This would allow delivery drivers to quickly understand the latest information and resume operations smoothly.

[0732] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0733] Step 1:

[0734] The server collects communication data received from the user's device during their vacation. Inputs include emails, messages, audio data, and video data, while output is an integrated dataset of these. The server retrieves the data via an API and stores it in a database.

[0735] Step 2:

[0736] The server converts audio and video data into text. Input is audio and video files, and output is text. Speech recognition software is used to generate text from audio, and text is extracted from video as needed.

[0737] Step 3:

[0738] The server analyzes the collected text information using AI technology and generates a summary. The input is the text information converted in step 2 and the collected emails and messages, and the output is the summarized text. A prompt sentence is entered into the AI ​​model, and it extracts important information to create a summary.

[0739] Step 4:

[0740] The server analyzes the sentiment of the summarized information and prioritizes extracting negative information. The input is the summarized text generated in step 3, and the output is the summarized text with negative elements emphasized. Natural language processing techniques are used to analyze the sentiment of the text.

[0741] Step 5:

[0742] The server personalizes the summary information based on the user's work content and provides it to the terminal. The input is the summary text highlighted in step 4, and the output is the personalized summary information. Information priorities are set according to the user's work history and project status.

[0743] Step 6:

[0744] Users view summary information through their terminals. The output is personalized summary information. Based on the presented summary, users can quickly grasp the current work situation and decide on their next actions.

[0745] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0746] This invention is an information processing system that enables users to quickly and efficiently return to work after a long vacation, and is particularly characterized by its optimization of information presentation, taking into account the user's emotions. Specific embodiments of the present invention are described below.

[0747] First, the user installs the system and authorizes data collection from their terminal device during their vacation. Based on this authorization, the server accesses email servers and business communication tools to collect unread emails and messages received. Furthermore, the terminal connects with online meeting systems to upload audio and video data to the cloud, which the server then converts into text data using speech recognition technology.

[0748] Next, the server uses generative artificial intelligence technology to summarize the content of the collected emails and messages. The summarization employs techniques to extract particularly important tasks and decisions and organize the information concisely. It also incorporates an emotion engine to analyze the emotional elements in the data and prioritize notifications if negative information is present.

[0749] Furthermore, this system can recognize the user's emotional state in real time. The terminal is equipped with sensors that detect the user's facial expressions and tone of voice, and the server uses this information to activate an emotion engine. If the user is stressed, the server adjusts the order in which summary information is presented, reducing the user's psychological burden by delaying less important information.

[0750] For example, if a user is overwhelmed by 100 unread emails after returning to work, the system will detect that the user's facial expression indicates stress and first display only the most important messages requiring immediate attention. Then, as the user's facial expression relaxes, it will sequentially display the other messages. This reduces the user's information processing burden and efficiently helps them return to work.

[0751] Furthermore, the server encrypts data and strictly controls access to ensure security, thereby managing user data safely. This configuration provides the present invention with an environment in which users can use the system with peace of mind.

[0752] By combining these functions, the present invention aims to alleviate the stress caused by information overload when users return to work after vacation, thereby creating a more comfortable business environment.

[0753] The following describes the processing flow.

[0754] Step 1:

[0755] Users configure system settings before their vacation, granting the server access to email servers and business communication tools. This enables data collection during their vacation.

[0756] Step 2:

[0757] The server connects to the user's mail server and business communication tools via API to collect all emails and messages received during the holiday period. The scope of collection can be adjusted as needed based on user preferences.

[0758] Step 3:

[0759] The device synchronizes the audio and video data of the online meeting to cloud storage. This prepares the server to use speech recognition technology to convert the audio data into text data.

[0760] Step 4:

[0761] The server generates summaries using generative artificial intelligence techniques from collected emails, messages, and converted text data. It extracts key points related to high-priority information and tasks, creating concise summaries.

[0762] Step 5:

[0763] The server uses an emotion engine to perform sentiment analysis on the collected data. It identifies sections containing particularly negative emotions and prioritizes notifying the user, enabling quicker action.

[0764] Step 6:

[0765] The device uses sensors to detect the user's emotional state in real time and sends that information to a server. The system determines the user's stress level based on their facial expressions and tone of voice.

[0766] Step 7:

[0767] The server adjusts the order in which summary information is presented based on the user's emotional state. If the user is stressed, only important messages requiring immediate attention will be displayed first, while other information will be presented later.

[0768] Step 8:

[0769] The server encrypts and securely stores the data it collects and processes. Furthermore, strict access control is implemented to ensure that only authorized users can access the data.

[0770] Through this process, the system can support users in smoothly returning to work after vacation and reduce stress caused by information overload.

[0771] (Example 2)

[0772] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0773] In today's business environment, when users return from extended leave, they often face information overload due to a large volume of accumulated electronic communications and notifications. This situation raises concerns about user efficiency and mental burden upon returning to work. Furthermore, the presentation of information without considering the user's emotional state can increase stress upon returning to work.

[0774] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0775] In this invention, the server includes means for collecting electronic communications and notifications acquired from the user's information device during vacation; means for summarizing important elements from the electronic communications and notifications using a generative artificial intelligence method; means for converting audio and video information into text information, analyzing the text information, and creating a summary; means for performing sentiment analysis and prioritizing the presentation of negative elements; and means for detecting the user's emotional state and adjusting the order in which information is presented. This reduces the user's information burden and enables the presentation of information that is in line with their emotional state.

[0776] A "user" is an individual or organization that uses this system to manage information and efficiently process that information upon returning to work.

[0777] "Information device" is a general term for electronic devices that are operated by users to send, receive, and process data.

[0778] "Electronic communications" refers to digital information, such as emails and messages, that is sent and received over a network.

[0779] A "notification" is a means of communication from a system to inform a user of specific information or a message.

[0780] "Generative artificial intelligence methods" are artificial intelligence technologies that analyze data and use the results to create new summaries and suggestions.

[0781] A "summary" is a concise compilation of information, extracting the most important elements from the data.

[0782] "Audio information" refers to data that records sounds emitted as human voices in digital format.

[0783] "Visual information" refers to data recorded in digital format, consisting of visually represented moving images and still images.

[0784] "Text information" refers to information expressed as a string of characters, including character data converted from audio and video.

[0785] "Analysis" is the process of evaluating and interpreting data according to a specific purpose to derive valuable information.

[0786] "Emotional analysis" is a technology that infers and judges an individual's emotions and psychological state from data.

[0787] "Negative elements" refer to information that may cause discomfort or stress to the user.

[0788] "Emotional state" refers to a user's psychological and emotional condition, and is a factor that influences their daily activities.

[0789] "Adjusting the order in which information is presented" means changing the order in which information is presented in order to make it more efficient and effective depending on the user's situation.

[0790] This invention is an information processing system designed to facilitate information processing in the workplace when a user returns from a long leave of absence. The system aims to efficiently collect, summarize, and present information related to business communication while taking into account the user's emotional state.

[0791] First, users access the system using their own information devices and authorize the collection of electronic communications and notifications received during their vacation. These information devices include computers, smartphones, and tablet devices, which are used to operate the system.

[0792] Next, the server efficiently collects electronic communications and notifications based on information authorized by the user. This communication data includes emails and chat messages. The collected data is further processed within the server.

[0793] The server uses generative artificial intelligence techniques to automatically summarize key elements from the collected information. In this process, particularly important tasks and decisions are extracted, prioritizing information useful to the user. The summarization is performed using generative AI technologies such as OpenAI models.

[0794] The device detects the user's emotional state in real time based on factors such as skin color and voice tone. This allows the server to determine whether the user is stressed. If the user indicates stress, the server adjusts the order in which information is presented, prioritizing less important information to reduce psychological burden.

[0795] Furthermore, audio and video information are also processed. The server can convert audio data used in online meetings into text using Google's speech recognition technology, and then analyze it to provide useful information.

[0796] For example, if a user is overwhelmed by a large number of unread emails after returning to work, this system will immediately prioritize and display the most urgent emails. This allows the user to efficiently resume their work.

[0797] Examples of prompt statements to input into a generative AI model include the following:

[0798] "To help users returning to work after extended leave efficiently process unread emails, devise a method to summarize and prioritize important emails. Consider the user's current emotional state and propose an appropriate order for presenting the information."

[0799] In this way, the present invention helps users reduce information overload while using the system with peace of mind, and supports a smooth return to work.

[0800] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0801] Step 1:

[0802] The user logs into their information device and authorizes the system to collect electronic communications and notifications received during their vacation. Input requires the user's authentication information and permission settings. Output is the authorized access information sent to the server. Based on this access information, the server determines the user's permitted scope.

[0803] Step 2:

[0804] The server uses access information obtained from users to collect electronic communications and notifications into the system. Specifically, it calls APIs of mail servers and chat tools to retrieve unread messages. The inputs used are user access information and API information of the target servers. As output, the server stores unread communication data.

[0805] Step 3:

[0806] The server uses a generative AI model to perform summarization processing to extract important elements from the collected electronic communication data. The communication data collected in step 2 is required as input. Summarized information is generated as output. Here, the AI ​​model prioritizes extracting tasks and important decisions within the emails.

[0807] Step 4:

[0808] The device collects user facial expression data and voice tone to detect the user's emotional state in real time. Sensor data from the camera and microphone are used as input. The output is the user's emotion analysis result, which is sent to the server. Based on this information, the server determines the user's emotional state.

[0809] Step 5:

[0810] The server adjusts the order in which information is presented based on the user's emotional state. If the user is showing signs of stress, the server re-evaluates the information to be presented preferentially and displays the most important content first. Summarized information and the results of the emotional analysis are required as input. The adjusted order of information presentation is determined as output and sent to the terminal.

[0811] Step 6:

[0812] The terminal receives pre-configured information from the server and presents it to the user. The user can then resume their work based on this information. The input is ordered information from the server. The output is a visual display of information on the user's screen. Specifically, information is displayed in a way that suits the user's mood, improving work efficiency.

[0813] Through the steps outlined above, this system aims to prevent users from becoming overwhelmed with information, reduce the burden upon returning to work, and enable efficient information processing.

[0814] (Application Example 2)

[0815] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0816] When returning to work after vacation, individuals may be overwhelmed by the sheer volume of information, hindering their ability to quickly and efficiently resume their duties. Furthermore, when using public services, citizens may lack prior knowledge of waiting times and congestion levels, hindering their efficient use of resources. These situations increase psychological burden and contribute to overall decreased work efficiency. Therefore, there is a need to provide means that enable users to act smoothly and with less burden when using work and daily public services.

[0817] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0818] In this invention, the server includes means for collecting communication information received from a communication terminal during vacation, means for summarizing important information from the communication information using artificial intelligence technology, and means for performing emotion recognition and adjusting the order in which information is presented to reduce psychological burden. This supports users in efficiently returning to work and enables optimization of the timing of users' use of public functions.

[0819] A "communication terminal" is an electronic device capable of sending and receiving information, and is a device used by users to acquire and transmit information.

[0820] "Communication information" refers to information sent and received in digital format, such as emails and messages.

[0821] "Artificial intelligence technology" refers to technologies that enable computer systems to learn, reason, and solve problems in a manner similar to human intelligence.

[0822] "Emotion recognition" is a technology that analyzes data such as a user's facial expressions and voice to determine their emotions and psychological state.

[0823] "Psychological burden" refers to mental stress and pressure, which can be a cause of information overload and hinder efficient action.

[0824] "Adjusting the order in which information is presented" means changing the display order of information so that users can process it effectively.

[0825] A "server" is a computer system that processes and provides information on a network in response to client requests.

[0826] "Public functions" refer to institutions and facilities that provide public services, such as administrative offices and public facilities.

[0827] "Usage data" refers to data that shows the current usage status of facilities and services, such as the level of congestion and the number of users.

[0828] "Recommended information" refers to instructions and advice provided to help users take the best course of action.

[0829] The system implementing this invention primarily consists of a server, communication terminals, and information from public functions. The server summarizes communication information collected from the communication terminals using artificial intelligence technology and performs user emotion recognition. This allows users to obtain timely information suitable for using business and public functions while minimizing psychological burden.

[0830] The server performs these processes over the network. The software used includes a backend application written in Python and Google Cloud's AutoML, and scalable data processing is possible by leveraging Amazon Web Services (AWS) cloud services.

[0831] As a concrete example, consider a scenario where users need to know in advance about waiting times and congestion levels when visiting a government office. Data collected by the user's communication device is uploaded to a server, where it undergoes sentiment analysis, and information regarding congestion levels is prioritized and summarized. As a result, the user is recommended the optimal time to visit via the application.

[0832] An example of a prompt that effectively utilizes a generative AI model is, "Please provide a wait time summary to optimize the timing of your visit, taking into account the current emotional state." This prompt allows the AI ​​model to properly recognize emotions and summarize information, enabling an improved user experience.

[0833] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0834] Step 1:

[0835] The terminal collects communication information. The user grants permission to the terminal, which then collects emails and messages and sends this data to the server. The input data is emails and messages, and the output is the data sent to the server. Specifically, the user launches the terminal application and performs an operation to authorize data collection.

[0836] Step 2:

[0837] The server summarizes communication information using artificial intelligence technology. The server analyzes the received data and uses a generative AI model to summarize important information. The input is data from emails and messages sent from terminals, and the output is summarized information. Specifically, the server runs Python on the cloud and uses Google Cloud's AutoML to summarize the data.

[0838] Step 3:

[0839] The server recognizes the user's emotions. The server receives data from the user's voice and facial expressions sent from the terminal and performs emotion recognition. The input is data from the user's facial expressions and voice, and the output is information about the user's emotional state. Specifically, the server executes an emotion analysis algorithm to evaluate the user's psychological state.

[0840] Step 4:

[0841] The server adjusts the order in which information is presented. Based on the emotion recognition results, the server adjusts the priority of the summarized information and presents it in the optimal order to minimize user stress. The input is information about the emotional state, and the output is the adjusted summarized information. Specifically, the server rearranges the information and prepares it for transmission to the terminal.

[0842] Step 5:

[0843] The terminal provides the user with customized information. The terminal displays the customized information received from the server on the user's screen. The input is the summarized information sent from the server, and the output is the information display provided to the user. Specifically, the terminal displays a notification and prepares an interface for the user to confirm the information.

[0844] This entire process allows users to process information efficiently and stress-free, optimizing their use of public services and other functions.

[0845] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0846] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0847] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0848] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0849] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0850] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0851] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0852] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0853] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0854] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0855] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0856] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0857] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0858] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0859] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0860] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0861] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0862] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0863] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0864] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0865] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0866] The following is further disclosed regarding the embodiments described above.

[0867] (Claim 1)

[0868] A means of collecting emails and messages received from a user's terminal device during their vacation,

[0869] A means for summarizing important information from the aforementioned emails and messages using generative artificial intelligence technology,

[0870] A means for converting audio data and video data into text data, and for analyzing the text data to generate a summary,

[0871] A method of performing sentiment analysis and prioritizing the display of negative information,

[0872] A means of providing the summarized information in an individualized manner based on the user's relevant business activities,

[0873] An information processing system that includes this.

[0874] (Claim 2)

[0875] The information processing system according to claim 1, further comprising means for encrypting the information obtained from the collection of the aforementioned data and implementing access control for secure management.

[0876] (Claim 3)

[0877] The information processing system according to claim 1, which integrates with other business tools and has means for filtering summary information provided to the user based on the user's business project progress.

[0878] "Example 1"

[0879] (Claim 1)

[0880] A means for collecting electronic communication data received from a user's computing device during the holiday period,

[0881] A means for summarizing important information from the aforementioned electronic communication data using generative artificial intelligence technology,

[0882] A means for converting audio and video information into coded data, and for analyzing the coded data to generate a summary,

[0883] A method of performing sentiment analysis and prioritizing the display of negative information,

[0884] A means of providing the summarized information in an individualized manner based on the user's relevant business activities,

[0885] A means of automatically collecting data periodically and protecting it with encryption technology,

[0886] A system that includes this.

[0887] (Claim 2)

[0888] The system according to claim 1, further comprising means for protecting the information obtained from the collection of the aforementioned data using cryptographic technology and implementing access restrictions for secure management.

[0889] (Claim 3)

[0890] The system according to claim 1, which has means for coordinating with other business systems and organizing summary information provided to users based on the progress of the users' business activities.

[0891] "Application Example 1"

[0892] (Claim 1)

[0893] A means of collecting communication data received from the user's device during vacation,

[0894] A means for generating and summarizing important information from the aforementioned communication data using AI technology,

[0895] A means for converting audio data and video data into text information, and for analyzing the text information to generate a summary,

[0896] A method for performing sentiment analysis and prioritizing the display of negative information,

[0897] A means of providing the summarized information in an individualized manner based on the user's relevant business activities,

[0898] Analyzing delivery instructions, customer feedback, and information on new stores in the area accumulated during the holiday period, and providing a means to present important information to delivery personnel,

[0899] A system that includes this.

[0900] (Claim 2)

[0901] The system according to claim 1, further comprising means for encrypting the information obtained from the collection of the aforementioned data and implementing access control for secure management.

[0902] (Claim 3)

[0903] The system according to claim 1, further comprising means for integrating with other business means and filtering summary information provided to the user based on the user's work progress.

[0904] "Example 2 of combining an emotion engine"

[0905] (Claim 1)

[0906] A means of collecting electronic communications and notifications acquired from a user's information device during their vacation,

[0907] A means for summarizing important elements from the aforementioned electronic communications and notifications using generative artificial intelligence methods,

[0908] A means for converting audio and video information into text information, and for analyzing the text information to create a summary,

[0909] A method for conducting emotion analysis and prioritizing the presentation of negative elements,

[0910] A means for detecting the user's emotional state and adjusting the order in which information is presented,

[0911] A means of providing the summary information in an individualized manner based on the user's relevant work content,

[0912] A system that includes this.

[0913] (Claim 2)

[0914] The system according to claim 1, further comprising means for encrypting the information obtained from the acquisition of the aforementioned data and implementing access control for secure management.

[0915] (Claim 3)

[0916] The system according to claim 1, further comprising means for integrating with other work tools and filtering summary information provided to the user based on the user's work project progress.

[0917] "Application example 2 when combining with an emotional engine"

[0918] (Claim 1)

[0919] A means of collecting communication information received from a communication terminal during vacation,

[0920] A means for summarizing important information from the aforementioned communication information using artificial intelligence technology,

[0921] A means for converting audio and video information into text information, and for analyzing the text information to generate a summary,

[0922] A means of adjusting the order in which information is presented in order to reduce psychological burden by performing emotion recognition,

[0923] A means of providing the summarized information in an individualized manner based on the content relevant to the user's work,

[0924] A means of collecting usage data to understand the usage status of public functions in advance and optimize the timing of visits, and providing recommended information,

[0925] A system that includes this.

[0926] (Claim 2)

[0927] The system according to claim 1, further comprising means for encoding the information obtained from the collection of the aforementioned information and implementing access control for secure management.

[0928] (Claim 3)

[0929] The system according to claim 1, which integrates with other processing tools and has means for selecting summary information to be provided to users based on the user's work progress. [Explanation of Symbols]

[0930] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of collecting communication data received from the user's device during vacation, A means for generating and summarizing important information from the aforementioned communication data using AI technology, A means for converting audio data and video data into text information, and for analyzing the text information to generate a summary, A method for performing sentiment analysis and prioritizing the display of negative information, A means of providing the summarized information in an individualized manner based on the user's relevant business activities, Analyzing delivery instructions, customer feedback, and information on new stores in the area accumulated during the holiday period, and providing a means to present important information to delivery personnel, A system that includes this.

2. The system according to claim 1, further comprising means for encrypting the information obtained from the collection of the aforementioned data and implementing access control for secure management.

3. The system according to claim 1, further comprising means for integrating with other business means and filtering summary information provided to the user based on the user's work progress.

Images