system

Abstract

We provide the system. [Solution] A means of acquiring and analyzing video data, audio data, and text data in real time, A method for managing the schedules of multiple users and automatically suggesting the optimal meeting time, A means of monitoring the status of equipment and optimizing lighting and temperature, A means to receive feedback from each user and dynamically update the schedule, Means for transmitting proposals and notifications to the user's device A system that includes this.

Description

Technical Field

[0005] ,

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, the method including: receiving a user utterance; adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot; encoding the prompt; and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In an office environment, the complexity of schedule management and the conscious resource allocation to facility management are causes leading to a decline in work efficiency and comfort experienced by multiple users. Due to such problems, users cannot concentrate on their original core business, and there is a problem of reduced productivity. To solve this problem, it is necessary to unify schedule and facility management and automatically provide an environment in which users can work comfortably.

Means for Solving the Problems

[0005] This invention provides means for acquiring and analyzing video, audio, and text data in real time, enabling automatic management of users' schedules and the suggestion of optimal meeting times. Furthermore, it includes means for optimizing office lighting and temperature by monitoring the status of equipment. In addition, it dynamically updates schedules based on user feedback and sends suggestions and notifications to users' devices, thereby improving work efficiency and providing a comfortable environment.

[0006] "Video data" refers to visual information collected using cameras or other visual devices.

[0007] "Audio data" refers to auditory information collected from microphones and other sound devices.

[0008] "Text data" refers to digital information expressed in character form or in that format.

[0009] "Means of analysis" refers to techniques or processes for analyzing collected data based on appropriate rules or algorithms and extracting meaningful information.

[0010] "Means of managing schedules" refers to techniques or methods for organizing and efficiently coordinating a user's schedule.

[0011] "Means of suggesting meeting times" refers to technologies or methods for indicating the optimal date and time for a meeting based on the user's schedule information.

[0012] "Means of monitoring equipment status" refers to techniques or measures for continuously checking the operation of lighting, temperature, and other office equipment and making adjustments as necessary.

[0013] "Optimization means" refers to techniques or methods for analyzing the current situation or data and automatically determining and implementing the most effective state within the configurable range.

[0014] "Means of receiving feedback" refers to the technology or method of receiving opinions and requests from users and reflecting that content in the system.

[0015] "Means for sending proposals and notifications" refers to the technology or method for communicating the system's derived action plans and important information to users. [Brief explanation of the drawing]

[0016] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of the data processing device and smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Embodiment 2 when combined with an emotion engine. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when combined with an emotion engine.

Mode for Carrying Out the Invention

[0017] Hereinafter, an example of an embodiment of the system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0020] In the following embodiments, a numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0021] In the following embodiments, a numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0022] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0024] [First Embodiment]

[0025] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0026] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0029] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0032] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0036] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0037] This invention is a smart assistant system designed to improve work efficiency and comfort in an office environment. This system includes schedule management, equipment management, and suggestion / notification functions to support the user's work. Its specific operation is described below.

[0038] First, when a user logs into the system, the server retrieves the user's schedule data from an external calendar service API. Based on the retrieved data, the server analyzes the user's schedule and detects duplicates and unprocessed tasks.

[0039] Next, to arrange the meeting, the server adjusts the availability of other participants based on the analysis results and proposes the optimal meeting time. This aims to facilitate smooth scheduling among users.

[0040] Furthermore, regarding facility management, the server works in conjunction with temperature sensors and lighting control systems to monitor the office environment. This ensures that room temperature and lighting are always maintained at optimal levels, creating a comfortable working environment.

[0041] Furthermore, the terminal notifies the user of changes to the proposed schedule and environment settings. If the user provides feedback, the schedule and equipment settings are automatically updated accordingly. This enables quick and flexible work responses.

[0042] For example, if an employee has a meeting scheduled for 2 PM, but one of the participants already has another meeting scheduled, the server will detect this and suggest a meeting at 3 PM instead. This suggestion will be notified to the user via their terminal, and once the user approves, the schedules of all involved parties will be automatically updated.

[0043] Thus, the present invention is a system that streamlines the management of the entire office environment and supports users so that they can concentrate on their core tasks.

[0044] The following describes the processing flow.

[0045] Step 1:

[0046] The user logs into the system and sends a request to begin schedule management. This is done via a smartphone or PC application.

[0047] Step 2:

[0048] Upon receiving login information, the server executes an authentication process and accesses the calendar service API associated with the user's account to retrieve schedule data.

[0049] Step 3:

[0050] The server analyzes the acquired schedule data and applies an analysis algorithm to detect overlapping schedules and unprocessed tasks.

[0051] Step 4:

[0052] Based on the analysis results, the server suggests meeting times by comparing them with the schedule information of other participants. It selects the optimal date and time for the meeting, taking into account the availability of all participants' schedules.

[0053] Step 5:

[0054] The terminal notifies the user of meeting time proposals sent from the server and displays them on the user's screen. The user can then choose to accept or reject the proposal.

[0055] Step 6:

[0056] The user submits feedback on the proposal from their device. If approved, the server updates the schedule and sends a notification to all relevant parties.

[0057] Step 7:

[0058] The server works in conjunction with the facility management system to monitor and optimize office temperature and lighting conditions. Based on sensor data, it automatically implements necessary changes.

[0059] Step 8:

[0060] The terminal notifies the user of any final schedule or setting changes and requests feedback as needed. This ensures continuous interaction between the user and the system.

[0061] (Example 1)

[0062] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0063] Improving work efficiency and maintaining comfort in the office environment are important challenges in many modern workplaces. However, scheduling meetings and optimally managing equipment remain time-consuming, and it is necessary to appropriately and promptly incorporate user feedback. This invention aims to address these challenges and create a more efficient office environment.

[0064] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0065] In this invention, the server includes means for acquiring and analyzing information data in real time, means for managing the activity schedules of multiple users and automatically proposing optimal meeting times, and means for monitoring the operating status of the equipment and optimizing lighting and temperature. This enables efficient work execution through real-time acquisition and analysis of information data, improved user activity efficiency through personalized suggestions, and the maintenance of a comfortable environment through appropriate equipment operation.

[0066] "Information data" refers to information such as users' schedules, equipment status, and user feedback within the office environment.

[0067] "Real-time" refers to a temporal process in which data is processed and analyzed simultaneously with its generation.

[0068] "Analysis" is a method of examining acquired data to identify specific patterns or problems.

[0069] "Planned activities" refer to the activities or events that a user plans to carry out within a specific period of time.

[0070] "Meeting time" refers to the time spent in meetings or discussions where multiple users gather.

[0071] "Equipment operating status" refers to the current operating status and performance of equipment and systems used within the office.

[0072] "Proposal" refers to the optimal solution or option presented to the user based on the analyzed data.

[0073] "Optimizing lighting and temperature" refers to adjusting the visual and thermal conditions of the office environment to be optimal for users.

[0074] This system acquires and analyzes various data in real time to improve work efficiency and comfort in the office environment, and provides optimal suggestions to users.

[0075] The server retrieves user schedule data using an external calendar service API. To ensure security, OAuth 2.0 authentication is used for data retrieval. The retrieved schedule data is then analyzed to detect time overlaps and unprocessed tasks. This analysis is made more accurate by utilizing historical data and patterns stored in the database.

[0076] Furthermore, the server monitors the office environment in conjunction with temperature sensors and lighting control systems. Specifically, it automatically adjusts heating and cooling when the temperature exceeds a certain range. It also has a function that uses motion sensors to turn on the lights only when needed.

[0077] The terminal provides users with suggestions and notifications from the server. This allows users to instantly learn about new meeting proposals and environment settings, enabling quick decision-making.

[0078] When a user provides feedback, the server reviews the entire system's operation based on that feedback. For example, if feedback is received that "the meeting time is proposed too early," the settings will be changed to prioritize afternoon time slots over morning ones for future proposals.

[0079] For example, if a user has scheduled a meeting for 10:00 AM, but analysis reveals that one of the other participants is already scheduled to attend another meeting, the server will detect this and suggest 11:00 AM as the new time slot. This new suggestion will be notified to the user via their device, and the schedule will be automatically updated once the user approves.

[0080] An example of a prompt to input into the generating AI model would be, "Please describe a system that analyzes a user's schedule and suggests the optimal meeting time." This prompt would enable the AI ​​model to generate even more accurate suggestions for improving work efficiency.

[0081] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0082] Step 1:

[0083] The user logs into the system. As input, the user enters their authentication information into the terminal and sends it to the server. The server uses an external authentication service to verify the authentication information and confirm the user's permissions. As output, if authentication is successful, the user's dashboard is displayed on the terminal.

[0084] Step 2:

[0085] The server retrieves the user's schedule data. Using the user ID as input, it sends a request to the calendar service API. The server analyzes the data retrieved from the API to detect duplicate appointments and unprocessed tasks. The analysis results are then displayed on the dashboard.

[0086] Step 3:

[0087] The server generates a meeting proposal. It uses the user's and other participants' schedule information as input. The server uses an analysis algorithm to compare available time slots and calculate the optimal meeting time. The proposed meeting time is then notified to the terminal as output.

[0088] Step 4:

[0089] The server monitors the equipment. It receives data from temperature and lighting sensors as input. The server analyzes this data to determine if the indoor environment is under optimal conditions. As output, it sends instructions to the equipment to adjust lighting and temperature as needed.

[0090] Step 5:

[0091] The terminal notifies the user. It takes input such as suggestions and configuration changes received from the server. The terminal communicates this information to the user through the user interface, using methods such as pop-up notifications and audio alerts. The output is for the user to review the information and approve or modify it.

[0092] Step 6:

[0093] The user provides feedback. As input, they enter their opinions on the proposed schedule and environmental conditions into the device. The device sends this data to the server. The server analyzes the feedback data and updates the AI ​​model to reflect the changes in future proposals. As output, improved proposals and adjusted environmental settings are provided.

[0094] Step 7:

[0095] The server updates the system. It uses user feedback and environmental change data as input. The server uses a generative AI model to create new suggestions for future use. The output is improved overall work efficiency for users.

[0096] (Application Example 1)

[0097] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0098] In factories, if the operation planning and maintenance of equipment and robots are not carried out efficiently, productivity will decline and significant production stoppages will occur due to machine failures. However, with conventional systems, real-time situation analysis and anomaly detection are difficult, and it has been difficult to provide information to personnel quickly. To solve this problem, a system is needed that continuously monitors the activity status of operating equipment and dynamically proposes and updates the optimal schedule.

[0099] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0100] In this invention, the server includes means for analyzing the activity status of operating equipment and optimizing the operating schedule, means for detecting equipment abnormalities and notifying the responsible person, and means for generating personalized suggestions based on the analyzed data. This enables efficient operation of production equipment within the factory and prompt maintenance response.

[0101] "Video data" refers to visual information acquired by a camera or other recording device.

[0102] "Audio data" refers to auditory information obtained by a sound acquisition device such as a microphone.

[0103] "Symbolic data" refers to information that is represented as meaningful symbols, such as letters or numbers.

[0104] "User" refers to an individual or group that uses the system.

[0105] "Activity plan" refers to the schedule of specific actions and events planned by the user.

[0106] "Meeting time" refers to the time set aside for multiple participants to gather and engage in a specific activity.

[0107] "Equipment" is a general term for devices or equipment that have a specific function.

[0108] "Opinions" refer to the content of feedback and evaluations that users provide to the system.

[0109] An "apparatus" is a machine or electronic device used for a specific purpose.

[0110] "Activity status" refers to information about the current operating state of a device or system.

[0111] "Anomaly" is a term that refers to an action or state that is not normally expected.

[0112] A "person in charge" is someone who is assigned to handle a specific task or problem.

[0113] An "information network" is a network used to transmit information using communication technology.

[0114] "Location-based equipment management" refers to the act of monitoring and optimizing the status of equipment located in a specific location.

[0115] A "maintenance plan" refers to a planned maintenance activity carried out to ensure that equipment and systems continue to operate normally.

[0116] In this embodiment of the invention, the system is configured as follows, mainly consisting of a server, a terminal, and a user.

[0117] First, the server acquires real-time activity data from sensors installed on equipment within the factory and analyzes that data. These sensors include vibration sensors and temperature sensors. The data is processed by a Raspberry Pi and transmitted to the server via the network. The server uses programming languages ​​and frameworks such as Python and Django to store the data in a database and perform analysis.

[0118] Next, the server optimizes the operating schedule of each device based on the analyzed data and has a function to notify the person in charge if an anomaly is detected. This notification is made via a device such as a smartphone or smart glasses carried by the person in charge. This ensures that the equipment in the factory is constantly monitored to keep it operating in optimal condition, and necessary maintenance work can be carried out quickly.

[0119] Furthermore, users can receive suggestions from the server via their terminals and submit feedback. This user feedback is analyzed by the server and incorporated into the latest equipment operation schedule. The user interface design utilizes a generative AI model to enhance the user experience and generate natural-sounding prompts.

[0120] For example, if a device detects more vibration than usual, the server recognizes this as an anomaly and sends a notification to the responsible person's terminal stating, "An anomaly has been detected in device A. Please check the details and perform maintenance." This allows the responsible person to immediately check the status and take the necessary action.

[0121] An example of a prompt to the generated AI model is, "Explain scheduling methods that support the efficient operation of factory equipment." This prompt allows the system to generate information to provide appropriate support to the user.

[0122] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0123] Step 1:

[0124] The server receives activity data in real time from sensors attached to equipment within the factory. The input consists of sensor data related to vibration and temperature. The server receives this data from the sensors, converts it to a digital format via a Raspberry Pi, and stores it in a database.

[0125] Step 2:

[0126] The server analyzes stored sensor data and evaluates the operating status of the device. The input is sensor data stored in a database. Based on the data, the server applies an analysis algorithm to identify abnormal patterns and detect unusual vibrations and temperature changes.

[0127] Step 3:

[0128] If an anomaly is detected, the server sends a notification to the responsible person. The input is the anomaly detection information obtained as a result of the analysis. The server generates a warning message regarding the anomaly and sends a notification to the responsible person's smartphone or smart glasses via the network.

[0129] Step 4:

[0130] The user checks notifications sent from the server via their terminal. The input is the notification message sent from the server. Based on the received notification, the user checks the status of the equipment on-site and performs maintenance as needed.

[0131] Step 5:

[0132] Users send feedback to the server about the status of the equipment and maintenance results. The input is the feedback information reported by the user. The server updates the database based on the received feedback and reflects it in the future equipment operation schedule.

[0133] Step 6:

[0134] The server optimizes the operating schedule and maintenance plan based on new feedback and analysis results. Inputs include feedback information and updated analysis data. The server uses this information to apply optimization algorithms and synchronizes the updated schedule with a cloud calendar service.

[0135] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0136] This invention is a smart assistant system that combines an emotion engine to analyze the user's emotional state, aiming to improve work efficiency and comfort in the office environment. This system uses the user's video and audio data to perform real-time emotion analysis, and uses the results to manage schedules and optimize equipment.

[0137] First, when a user logs into the system, the server collects video and audio data through the camera and microphone. This data is sent to the emotion engine, where the user's emotional state is analyzed in real time. The emotion engine determines whether the user is stressed or relaxed.

[0138] Based on the analysis results, the server adjusts schedule management. For example, if it determines that a user is experiencing stress, it adjusts suggested meeting times to be more user-friendly. Furthermore, in terms of optimizing the environment, it changes room temperature and lighting brightness to match the user's emotional state.

[0139] Meanwhile, the device notifies the user of the analysis results obtained from the emotion engine and the suggestions based on those results. Based on this, the user can provide feedback through the device. This feedback is sent to the server and reflected in future suggestions and settings.

[0140] As a concrete example, when a user begins to feel tired during work, the system analyzes data obtained from the camera and microphone using an emotion engine. If "fatigue" is detected through the analysis, the server will suggest postponing the meeting or instruct the room lighting to be changed to a warmer color. This change is notified to the user via the terminal and is implemented only after the user approves it. This invention greatly improves user efficiency and comfort by providing an appropriate work environment that takes emotional states into account in real time.

[0141] The following describes the processing flow.

[0142] Step 1:

[0143] The system starts operating when the user logs in and is ready to begin work. Login is performed using authentication credentials.

[0144] Step 2:

[0145] The server collects video and audio data from the camera and microphone in real time. This data is securely stored and immediately sent to the emotion engine.

[0146] Step 3:

[0147] The server's emotion engine analyzes video and audio data to determine the user's emotional state. This analysis includes the ability to detect states such as "stress," "concentration," and "relaxation" from changes in facial expressions and tone of voice.

[0148] Step 4:

[0149] The server sends feedback to the scheduling function based on the user's emotional state. For example, if the server determines that the user is stressed, it generates suggestions to reschedule existing meetings or tasks.

[0150] Step 5:

[0151] The server optimizes office equipment based on the analysis results. Specifically, it issues instructions such as adjusting the temperature to create a comfortable room temperature or changing the lighting to create an eye-friendly environment.

[0152] Step 6:

[0153] The device notifies the user of suggestions and configuration changes generated by the server. These notifications appear as pop-ups or audio alerts, prompting immediate user response.

[0154] Step 7:

[0155] Users review the suggestions notified on their devices and accept or request changes based on their preferences. This allows user feedback to be reflected in the system.

[0156] Step 8:

[0157] The server dynamically updates schedules and equipment settings based on user feedback. These updates are automatically reflected in stakeholders and systems, allowing for continuous optimization.

[0158] (Example 2)

[0159] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0160] In traditional work environments, the lack of consideration for users' emotional states and stress levels tended to lead to decreased work efficiency. Furthermore, optimizing the environment was difficult due to the difficulty in making adjustments based on individual circumstances, resulting in a reliance on standardized settings. This directly impacted user satisfaction and work efficiency. Additionally, even in remote management environments, there was a challenge in achieving sufficient collaboration among remote locations.

[0161] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0162] In this invention, the server includes a device means for collecting and analyzing video and audio information in real time, an information processing device means for determining the emotional state of individual users, and a device means for monitoring environmental settings and optimizing lighting and temperature based on the user's emotional state. This enables dynamic adjustment of the work environment to reflect the user's emotional state, improving work efficiency and comfort. Furthermore, it enables optimization of remote equipment management even in remote environments.

[0163] "Visual information" refers to all visual data acquired through imaging devices such as cameras.

[0164] "Audio information" refers to all audio data acquired through sound collection devices such as microphones.

[0165] A "real-time data collection and analysis device" refers to a device that has the ability to collect data with minimal time delay and to process and analyze that data immediately.

[0166] "Individual users" refers to each person who uses a service or system, and the term "individual users" means that the processing is tailored to that person.

[0167] An "information processing device for determining emotional state" refers to a device that analyzes data collected from users and has the function of identifying and determining a person's current psychological and emotional state.

[0168] A "device for optimizing environmental settings" refers to a device that has the function of adjusting and optimizing the surrounding physical conditions (e.g., lighting, temperature) according to the user's condition.

[0169] A "device for receiving feedback and incorporating it into future adjustments" refers to a device that receives opinions and reactions from users and uses them to improve and adjust the system's operation and environment settings.

[0170] A "device that transmits suggestions and notifications to a user's terminal" refers to a device that has the function of transmitting information and suggestions generated from a server or system to an electronic device owned by the user to inform them.

[0171] This invention is a smart assistant system designed to improve work efficiency and user comfort in an office environment. This system operates through the coordinated efforts of a server, terminal, and user.

[0172] When a user logs in, the server collects video and audio information in real time via cameras and microphones installed in the office. This data is processed by the server, and an AI-based emotion analysis engine is used to determine the user's emotional state. At this time, deep learning algorithms are used to analyze facial expressions, voice tone, and intonation.

[0173] Based on the analyzed emotional state, the server adjusts the user's schedule management and dynamically determines the optimal meeting time. For example, if stress is detected, the scheduling is flexibly adjusted while considering the importance of the meeting. In addition, as part of optimizing the environment, the room lighting and temperature are automatically adjusted according to the user's emotional state to maintain a comfortable working environment.

[0174] The terminal's role is to send suggestions and notifications from the server to the user. The user receives these suggestions through the terminal and provides approval and feedback. This feedback is sent to the server and used to inform future suggestions and environment adjustments.

[0175] For example, if a user starts feeling fatigued during work, the server's emotion analysis engine will detect "fatigue." Based on this, the server will suggest postponing the meeting and send instructions to the terminal to change the lighting to a warmer color. If the user approves, the system will implement the changes.

[0176] Furthermore, an example of a prompt using the generative AI model is, "If the user is feeling nervous but has an important meeting coming up, please suggest the best course of action." Through this prompt, the AI ​​model generates the optimal response, which is then reflected in the actual system operation on the server.

[0177] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0178] Step 1:

[0179] When a user logs into the system, the server collects video and audio information from the office via the camera and microphone. This input data serves as material for real-time analysis of changes in the user's facial expressions and voice. The server converts the data into an appropriate format and sends it to the emotion analysis engine.

[0180] Step 2:

[0181] The server analyzes the data collected by the emotion analysis engine. Specifically, it uses deep learning algorithms to perform facial expression recognition and voice tone analysis. In this process, the data is classified into emotional states such as stress, fatigue, and relaxation. The analysis results are generated as output, which becomes the basis data for the next step.

[0182] Step 3:

[0183] The server dynamically adjusts the user's schedule based on the sentiment analysis results. It receives the analysis results as input and re-evaluates the importance and priority of meetings within the overall schedule. Even if stress is detected, it flexibly changes meeting times to allow the user to work efficiently.

[0184] Step 4:

[0185] The server optimizes the facilities in response to the user's emotional state. Specifically, based on the output analysis results, it changes the room lighting to an appropriate color tone and adjusts the temperature. As a result, a comfortable working environment is provided as the final output.

[0186] Step 5:

[0187] The terminal notifies the user of suggestions and adjustments from the server. These notifications are provided via email or pop-up messages, and the user reviews and approves them as input for the process. The user's feedback then becomes input for the next step.

[0188] Step 6:

[0189] Users provide feedback on suggestions received through their terminals. This feedback information is sent to the server and used to improve future environment adjustments and scheduling. This allows the system to be continuously optimized, enabling more efficient output.

[0190] (Application Example 2)

[0191] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0192] There is a problem in that it is difficult for customers to have a more comfortable and satisfying shopping experience in physical stores. In particular, it is difficult for staff to understand the emotional state of every customer and respond appropriately. Another challenge is having the flexibility to immediately reflect customer feedback in service.

[0193] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0194] In this invention, the server includes means for acquiring and analyzing video and audio data in real time, means for understanding the situation of multiple customers and automatically proposing the optimal customer service response, and means for receiving feedback from each customer and dynamically updating the service. This makes shopping at physical stores more comfortable for customers and enables staff to quickly respond appropriately to the emotional state of customers.

[0195] "Video data" refers to visual information acquired in real time, and this information is used as the basis for analysis.

[0196] "Audio data" refers to acoustic information collected in real time, which is used as material for emotion analysis.

[0197] "Means of analysis" refers to methods or devices for processing collected data and understanding customer emotions and circumstances.

[0198] "Means for understanding the status of multiple customers" refers to a method or device for understanding the status of each customer and determining the necessary actions based on data obtained from multiple customers.

[0199] "Means for automatically suggesting optimal customer service responses" refers to a method or device for suggesting the most appropriate response to a customer to staff based on analysis results.

[0200] "Means of monitoring the store environment" refers to methods or devices used to observe and optimize environmental elements such as lighting and temperature within a store.

[0201] "Means for dynamically updating services" refers to methods or devices for receiving customer feedback in real time and flexibly changing the content of services based on that information.

[0202] "Means for transmitting suggestions and notifications to staff devices" refers to a method or device for quickly communicating analysis results and suggestions to customer service staff.

[0203] To implement this invention, it is first necessary to configure a system to improve the customer experience in physical stores. This begins with acquiring and analyzing customer video and audio data in real time using smart glasses. The acquired data is transmitted via Wi-Fi to a server in the cloud. On the server, emotion analysis software runs to analyze the customer's emotional state. Software such as Microsoft® Azure® Cognitive Services is used for this analysis.

[0204] Based on the analyzed emotional state of the customer, the server suggests the most appropriate response to the customer service staff wearing smart glasses. This suggestion provides the staff with the information they need to interact with customers in real time. For example, if a customer appears confused in the store, the server might send a suggestion to the staff such as, "You seem to be having trouble. Please let us know if there's anything we can do to help."

[0205] This system allows staff to quickly understand customer needs and respond appropriately. This not only contributes to improved customer satisfaction but also enhances the overall service quality of the store.

[0206] An example of a prompt is, "Please advise how to respond if the customer is confused." This is used in situations where a generative AI model is used to generate suggestions for staff.

[0207] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0208] Step 1:

[0209] The server receives video and audio data from the smart glasses in real time. This input includes information such as the customer's facial expressions and tone of voice. The server receives this data as input and prepares it to be sent to emotion analysis software.

[0210] Step 2:

[0211] The server analyzes the received video and audio data using emotion analysis software. Data processing involves extracting customer facial features from the video data and analyzing emotional patterns from the audio data. This process generates an output that identifies the customer's emotional state.

[0212] Step 3:

[0213] The server uses an AI model based on the analysis results to generate suggestions for the staff. The input is the analysis results from step 2, and the output is the optimal customer service suggestion. This output suggestion is sent to the next step in text format.

[0214] Step 4:

[0215] The server notifies the customer service staff of the generated suggestions via their smart glasses. This allows the staff to respond to the customer's situation in real time. The input is the suggested content output in step 3, and the output is the notification to the staff.

[0216] Step 5:

[0217] Users (customers) provide feedback based on their in-store experience. This feedback, provided via a terminal (smart glasses or other device), is sent to a server. As part of data processing, the feedback data is used to generate suggestions for future visits. The input is user feedback, and the output is data for improved suggestions.

[0218] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0219] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0220] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0221] [Second Embodiment]

[0222] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0223] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0224] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0225] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0226] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0227] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0228] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0229] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0230] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0231] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0232] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0233] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0234] This invention is a smart assistant system designed to improve work efficiency and comfort in an office environment. This system includes schedule management, equipment management, and suggestion / notification functions to support the user's work. Its specific operation is described below.

[0235] First, when a user logs into the system, the server retrieves the user's schedule data from an external calendar service API. Based on the retrieved data, the server analyzes the user's schedule and detects duplicates and unprocessed tasks.

[0236] Next, to arrange the meeting, the server adjusts the availability of other participants based on the analysis results and proposes the optimal meeting time. This aims to facilitate smooth scheduling among users.

[0237] Furthermore, regarding facility management, the server works in conjunction with temperature sensors and lighting control systems to monitor the office environment. This ensures that room temperature and lighting are always maintained at optimal levels, creating a comfortable working environment.

[0238] Furthermore, the terminal notifies the user of changes to the proposed schedule and environment settings. If the user provides feedback, the schedule and equipment settings are automatically updated accordingly. This enables quick and flexible work responses.

[0239] For example, if an employee has a meeting scheduled for 2 PM, but one of the participants already has another meeting scheduled, the server will detect this and suggest a meeting at 3 PM instead. This suggestion will be notified to the user via their terminal, and once the user approves, the schedules of all involved parties will be automatically updated.

[0240] Thus, the present invention is a system that streamlines the management of the entire office environment and supports users so that they can concentrate on their core tasks.

[0241] The following describes the processing flow.

[0242] Step 1:

[0243] The user logs into the system and sends a request to begin schedule management. This is done via a smartphone or PC application.

[0244] Step 2:

[0245] Upon receiving login information, the server executes an authentication process and accesses the calendar service API associated with the user's account to retrieve schedule data.

[0246] Step 3:

[0247] The server analyzes the acquired schedule data and applies an analysis algorithm to detect overlapping schedules and unprocessed tasks.

[0248] Step 4:

[0249] Based on the analysis results, the server suggests meeting times by comparing them with the schedule information of other participants. It selects the optimal date and time for the meeting, taking into account the availability of all participants' schedules.

[0250] Step 5:

[0251] The terminal notifies the user of meeting time proposals sent from the server and displays them on the user's screen. The user can then choose to accept or reject the proposal.

[0252] Step 6:

[0253] The user submits feedback on the proposal from their device. If approved, the server updates the schedule and sends a notification to all relevant parties.

[0254] Step 7:

[0255] The server works in conjunction with the facility management system to monitor and optimize office temperature and lighting conditions. Based on sensor data, it automatically implements necessary changes.

[0256] Step 8:

[0257] The terminal notifies the user of any final schedule or setting changes and requests feedback as needed. This ensures continuous interaction between the user and the system.

[0258] (Example 1)

[0259] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0260] Improving work efficiency and maintaining comfort in the office environment are important challenges in many modern workplaces. However, scheduling meetings and optimally managing equipment remain time-consuming, and it is necessary to appropriately and promptly incorporate user feedback. This invention aims to address these challenges and create a more efficient office environment.

[0261] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0262] In this invention, the server includes means for acquiring and analyzing information data in real time, means for managing the activity schedules of multiple users and automatically proposing optimal meeting times, and means for monitoring the operating status of the equipment and optimizing lighting and temperature. This enables efficient work execution through real-time acquisition and analysis of information data, improved user activity efficiency through personalized suggestions, and the maintenance of a comfortable environment through appropriate equipment operation.

[0263] "Information data" refers to information such as users' schedules, equipment status, and user feedback within the office environment.

[0264] "Real-time" refers to a temporal process in which data is processed and analyzed simultaneously with its generation.

[0265] "Analysis" is a method of examining acquired data to identify specific patterns or problems.

[0266] "Planned activities" refer to the activities or events that a user plans to carry out within a specific period of time.

[0267] "Meeting time" refers to the time spent in meetings or discussions where multiple users gather.

[0268] "Equipment operating status" refers to the current operating status and performance of equipment and systems used within the office.

[0269] "Proposal" refers to the optimal solution or option presented to the user based on the analyzed data.

[0270] "Optimizing lighting and temperature" refers to adjusting the visual and thermal conditions of the office environment to be optimal for users.

[0271] This system acquires and analyzes various data in real time to improve work efficiency and comfort in the office environment, and provides optimal suggestions to users.

[0272] The server retrieves user schedule data using an external calendar service API. To ensure security, OAuth 2.0 authentication is used for data retrieval. The retrieved schedule data is then analyzed to detect time overlaps and unprocessed tasks. This analysis is made more accurate by utilizing historical data and patterns stored in the database.

[0273] Furthermore, the server monitors the office environment in conjunction with temperature sensors and lighting control systems. Specifically, it automatically adjusts heating and cooling when the temperature exceeds a certain range. It also has a function that uses motion sensors to turn on the lights only when needed.

[0274] The terminal provides users with suggestions and notifications from the server. This allows users to instantly learn about new meeting proposals and environment settings, enabling quick decision-making.

[0275] When a user provides feedback, the server reviews the entire system's operation based on that feedback. For example, if feedback is received that "the meeting time is proposed too early," the settings will be changed to prioritize afternoon time slots over morning ones for future proposals.

[0276] For example, if a user has scheduled a meeting for 10:00 AM, but analysis reveals that one of the other participants is already scheduled to attend another meeting, the server will detect this and suggest 11:00 AM as the new time slot. This new suggestion will be notified to the user via their device, and the schedule will be automatically updated once the user approves.

[0277] An example of a prompt to input into the generating AI model would be, "Please describe a system that analyzes a user's schedule and suggests the optimal meeting time." This prompt would enable the AI ​​model to generate even more accurate suggestions for improving work efficiency.

[0278] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0279] Step 1:

[0280] The user logs into the system. As input, the user enters their authentication information into the terminal and sends it to the server. The server uses an external authentication service to verify the authentication information and confirm the user's permissions. As output, if authentication is successful, the user's dashboard is displayed on the terminal.

[0281] Step 2:

[0282] The server retrieves the user's schedule data. Using the user ID as input, it sends a request to the calendar service API. The server analyzes the data retrieved from the API to detect duplicate appointments and unprocessed tasks. The analysis results are then displayed on the dashboard.

[0283] Step 3:

[0284] The server generates a meeting proposal. Using the schedule information of the user and other participants as input, the server compares available time slots using an analysis algorithm and calculates the optimal meeting time. As output, the proposed meeting time is notified to the terminal.

[0285] Step 4:

[0286] The server monitors the facilities. Obtaining data from temperature sensors and lighting sensors as input, the server analyzes this data and determines whether the indoor environment is optimal. As output, instructions to adjust lighting and temperature are sent to the facilities as needed.

[0287] Step 5:

[0288] The terminal notifies the user. Taking in the proposal and information on environmental setting changes received from the server as input, the terminal uses pop-up notifications, voice alerts, etc. through the user interface to convey information to the user. As output, the user checks the information and gives approval or makes changes.

[0289] Step 6:

[0290] The user provides feedback. Inputting opinions on the proposed schedule and environmental conditions to the terminal as input, the terminal sends the data to the server. The server analyzes the feedback data and updates the AI model to be reflected in the next proposal. As output, improved proposals and adjusted environmental settings are made.

[0291] Step 7:

[0292] The server updates the system. Using the feedback from the user and environmental change data as input, the server creates new proposals using a generated AI model for future use. As output, the work efficiency of all users is improved.

[0293] (Application Example 1)

[0294] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0295] In factories, if the operation planning and maintenance of equipment and robots are not carried out efficiently, productivity will decline and significant production stoppages will occur due to machine failures. However, with conventional systems, real-time situation analysis and anomaly detection are difficult, and it has been difficult to provide information to personnel quickly. To solve this problem, a system is needed that continuously monitors the activity status of operating equipment and dynamically proposes and updates the optimal schedule.

[0296] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0297] In this invention, the server includes means for analyzing the activity status of operating equipment and optimizing the operating schedule, means for detecting equipment abnormalities and notifying the responsible person, and means for generating personalized suggestions based on the analyzed data. This enables efficient operation of production equipment within the factory and prompt maintenance response.

[0298] "Video data" refers to visual information acquired by a camera or other recording device.

[0299] "Audio data" refers to auditory information obtained by a sound acquisition device such as a microphone.

[0300] "Symbolic data" refers to information that is represented as meaningful symbols, such as letters or numbers.

[0301] "User" refers to an individual or group that uses the system.

[0302] "Scheduled activities" refer to the schedules of specific actions or events planned by users.

[0303] "Meeting time" refers to the time secured for multiple participants to gather and conduct specific activities.

[0304] "Equipment" refers to the general term for devices and facilities with specific functions.

[0305] "Opinions" refer to the content of feedback and evaluations given by users to the system.

[0306] "Device" refers to a machine or electronic device used for a specific purpose.

[0307] "Activity status" refers to information on the current operating state of devices and systems.

[0308] "Abnormality" is a term referring to actions or states that are not normally expected.

[0309] "Responsible person" refers to a person entrusted with specific tasks or problem handling.

[0310] "Information network" refers to a network for transmitting information using communication technology.

[0311] "Equipment management at a location" refers to the act of monitoring and optimizing the status of equipment at a specific location.

[0312] "Maintenance plan" refers to planned maintenance activities carried out to ensure the continuous normal operation of devices and systems.

[0313] In the embodiments of this invention, the system is configured as follows with the server, terminal, and user as the main components.

[0314] First, the server acquires real-time activity data from sensors installed on equipment within the factory and analyzes that data. These sensors include vibration sensors and temperature sensors. The data is processed by a Raspberry Pi and transmitted to the server via the network. The server uses programming languages ​​and frameworks such as Python and Django to store the data in a database and perform analysis.

[0315] Next, the server optimizes the operating schedule of each device based on the analyzed data and has a function to notify the person in charge if an anomaly is detected. This notification is made via a device such as a smartphone or smart glasses carried by the person in charge. This ensures that the equipment in the factory is constantly monitored to keep it operating in optimal condition, and necessary maintenance work can be carried out quickly.

[0316] Furthermore, users can receive suggestions from the server via their terminals and submit feedback. This user feedback is analyzed by the server and incorporated into the latest equipment operation schedule. The user interface design utilizes a generative AI model to enhance the user experience and generate natural-sounding prompts.

[0317] For example, if a device detects more vibration than usual, the server recognizes this as an anomaly and sends a notification to the responsible person's terminal stating, "An anomaly has been detected in device A. Please check the details and perform maintenance." This allows the responsible person to immediately check the status and take the necessary action.

[0318] An example of a prompt to the generated AI model is, "Explain scheduling methods that support the efficient operation of factory equipment." This prompt allows the system to generate information to provide appropriate support to the user.

[0319] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0320] Step 1:

[0321] The server receives activity data in real time from sensors attached to equipment within the factory. The input consists of sensor data related to vibration and temperature. The server receives this data from the sensors, converts it to a digital format via a Raspberry Pi, and stores it in a database.

[0322] Step 2:

[0323] The server analyzes stored sensor data and evaluates the operating status of the device. The input is sensor data stored in a database. Based on the data, the server applies an analysis algorithm to identify abnormal patterns and detect unusual vibrations and temperature changes.

[0324] Step 3:

[0325] If an anomaly is detected, the server sends a notification to the responsible person. The input is the anomaly detection information obtained as a result of the analysis. The server generates a warning message regarding the anomaly and sends a notification to the responsible person's smartphone or smart glasses via the network.

[0326] Step 4:

[0327] The user checks notifications sent from the server via their terminal. The input is the notification message sent from the server. Based on the received notification, the user checks the status of the equipment on-site and performs maintenance as needed.

[0328] Step 5:

[0329] Users send feedback to the server about the status of the equipment and maintenance results. The input is the feedback information reported by the user. The server updates the database based on the received feedback and reflects it in the future equipment operation schedule.

[0330] Step 6:

[0331] The server optimizes the operating schedule and maintenance plan based on new feedback and analysis results. Inputs include feedback information and updated analysis data. The server uses this information to apply optimization algorithms and synchronizes the updated schedule with a cloud calendar service.

[0332] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0333] This invention is a smart assistant system that combines an emotion engine to analyze the user's emotional state, aiming to improve work efficiency and comfort in the office environment. This system uses the user's video and audio data to perform real-time emotion analysis, and uses the results to manage schedules and optimize equipment.

[0334] First, when a user logs into the system, the server collects video and audio data through the camera and microphone. This data is sent to the emotion engine, where the user's emotional state is analyzed in real time. The emotion engine determines whether the user is stressed or relaxed.

[0335] Based on the analysis results, the server adjusts schedule management. For example, if it determines that a user is experiencing stress, it adjusts suggested meeting times to be more user-friendly. Furthermore, in terms of optimizing the environment, it changes room temperature and lighting brightness to match the user's emotional state.

[0336] Meanwhile, the device notifies the user of the analysis results obtained from the emotion engine and the suggestions based on those results. Based on this, the user can provide feedback through the device. This feedback is sent to the server and reflected in future suggestions and settings.

[0337] As a concrete example, when a user begins to feel tired during work, the system analyzes data obtained from the camera and microphone using an emotion engine. If "fatigue" is detected through the analysis, the server will suggest postponing the meeting or instruct the room lighting to be changed to a warmer color. This change is notified to the user via the terminal and is implemented only after the user approves it. This invention greatly improves user efficiency and comfort by providing an appropriate work environment that takes emotional states into account in real time.

[0338] The following describes the processing flow.

[0339] Step 1:

[0340] The system starts operating when the user logs in and is ready to begin work. Login is performed using authentication credentials.

[0341] Step 2:

[0342] The server collects video and audio data from the camera and microphone in real time. This data is securely stored and immediately sent to the emotion engine.

[0343] Step 3:

[0344] The server's emotion engine analyzes video and audio data to determine the user's emotional state. This analysis includes the ability to detect states such as "stress," "concentration," and "relaxation" from changes in facial expressions and tone of voice.

[0345] Step 4:

[0346] The server sends feedback to the scheduling function based on the user's emotional state. For example, if the server determines that the user is stressed, it generates suggestions to reschedule existing meetings or tasks.

[0347] Step 5:

[0348] The server optimizes office equipment based on the analysis results. Specifically, it issues instructions such as adjusting the temperature to create a comfortable room temperature or changing the lighting to create an eye-friendly environment.

[0349] Step 6:

[0350] The device notifies the user of suggestions and configuration changes generated by the server. These notifications appear as pop-ups or audio alerts, prompting immediate user response.

[0351] Step 7:

[0352] Users review the suggestions notified on their devices and accept or request changes based on their preferences. This allows user feedback to be reflected in the system.

[0353] Step 8:

[0354] The server dynamically updates schedules and equipment settings based on user feedback. These updates are automatically reflected in stakeholders and systems, allowing for continuous optimization.

[0355] (Example 2)

[0356] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0357] In traditional work environments, the lack of consideration for users' emotional states and stress levels tended to lead to decreased work efficiency. Furthermore, optimizing the environment was difficult due to the difficulty in making adjustments based on individual circumstances, resulting in a reliance on standardized settings. This directly impacted user satisfaction and work efficiency. Additionally, even in remote management environments, there was a challenge in achieving sufficient collaboration among remote locations.

[0358] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0359] In this invention, the server includes a device means for collecting and analyzing video and audio information in real time, an information processing device means for determining the emotional state of individual users, and a device means for monitoring environmental settings and optimizing lighting and temperature based on the user's emotional state. This enables dynamic adjustment of the work environment to reflect the user's emotional state, improving work efficiency and comfort. Furthermore, it enables optimization of remote equipment management even in remote environments.

[0360] "Visual information" refers to all visual data acquired through imaging devices such as cameras.

[0361] "Audio information" refers to all audio data acquired through sound collection devices such as microphones.

[0362] A "real-time data collection and analysis device" refers to a device that has the ability to collect data with minimal time delay and to process and analyze that data immediately.

[0363] "Individual users" refers to each person who uses a service or system, and the term "individual users" means that the processing is tailored to that person.

[0364] An "information processing device for determining emotional state" refers to a device that analyzes data collected from users and has the function of identifying and determining a person's current psychological and emotional state.

[0365] A "device for optimizing environmental settings" refers to a device that has the function of adjusting and optimizing the surrounding physical conditions (e.g., lighting, temperature) according to the user's condition.

[0366] A "device for receiving feedback and incorporating it into future adjustments" refers to a device that receives opinions and reactions from users and uses them to improve and adjust the system's operation and environment settings.

[0367] A "device that transmits suggestions and notifications to a user's terminal" refers to a device that has the function of transmitting information and suggestions generated from a server or system to an electronic device owned by the user to inform them.

[0368] This invention is a smart assistant system designed to improve work efficiency and user comfort in an office environment. This system operates through the coordinated efforts of a server, terminal, and user.

[0369] When a user logs in, the server collects video and audio information in real time via cameras and microphones installed in the office. This data is processed by the server, and an AI-based emotion analysis engine is used to determine the user's emotional state. At this time, deep learning algorithms are used to analyze facial expressions, voice tone, and intonation.

[0370] Based on the analyzed emotional state, the server adjusts the user's schedule management and dynamically determines the optimal meeting time. For example, if stress is detected, the scheduling is flexibly adjusted while considering the importance of the meeting. In addition, as part of optimizing the environment, the room lighting and temperature are automatically adjusted according to the user's emotional state to maintain a comfortable working environment.

[0371] The terminal's role is to send suggestions and notifications from the server to the user. The user receives these suggestions through the terminal and provides approval and feedback. This feedback is sent to the server and used to inform future suggestions and environment adjustments.

[0372] For example, if a user starts feeling fatigued during work, the server's emotion analysis engine will detect "fatigue." Based on this, the server will suggest postponing the meeting and send instructions to the terminal to change the lighting to a warmer color. If the user approves, the system will implement the changes.

[0373] Furthermore, an example of a prompt using the generative AI model is, "If the user is feeling nervous but has an important meeting coming up, please suggest the best course of action." Through this prompt, the AI ​​model generates the optimal response, which is then reflected in the actual system operation on the server.

[0374] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0375] Step 1:

[0376] When a user logs into the system, the server collects video and audio information from the office via the camera and microphone. This input data serves as material for real-time analysis of changes in the user's facial expressions and voice. The server converts the data into an appropriate format and sends it to the emotion analysis engine.

[0377] Step 2:

[0378] The server analyzes the data collected by the emotion analysis engine. Specifically, it uses deep learning algorithms to perform facial expression recognition and voice tone analysis. In this process, the data is classified into emotional states such as stress, fatigue, and relaxation. The analysis results are generated as output, which becomes the basis data for the next step.

[0379] Step 3:

[0380] The server dynamically adjusts the user's schedule based on the sentiment analysis results. It receives the analysis results as input and re-evaluates the importance and priority of meetings within the overall schedule. Even if stress is detected, it flexibly changes meeting times to allow the user to work efficiently.

[0381] Step 4:

[0382] The server optimizes the facilities in response to the user's emotional state. Specifically, based on the output analysis results, it changes the room lighting to an appropriate color tone and adjusts the temperature. As a result, a comfortable working environment is provided as the final output.

[0383] Step 5:

[0384] The terminal notifies the user of suggestions and adjustments from the server. These notifications are provided via email or pop-up messages, and the user reviews and approves them as input for the process. The user's feedback then becomes input for the next step.

[0385] Step 6:

[0386] Users provide feedback on suggestions received through their terminals. This feedback information is sent to the server and used to improve future environment adjustments and scheduling. This allows the system to be continuously optimized, enabling more efficient output.

[0387] (Application Example 2)

[0388] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0389] There is a problem in that it is difficult for customers to have a more comfortable and satisfying shopping experience in physical stores. In particular, it is difficult for staff to understand the emotional state of every customer and respond appropriately. Another challenge is having the flexibility to immediately reflect customer feedback in service.

[0390] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0391] In this invention, the server includes means for acquiring and analyzing video and audio data in real time, means for understanding the situation of multiple customers and automatically proposing the optimal customer service response, and means for receiving feedback from each customer and dynamically updating the service. This makes shopping at physical stores more comfortable for customers and enables staff to quickly respond appropriately to the emotional state of customers.

[0392] "Video data" refers to visual information acquired in real time, and this information is used as the basis for analysis.

[0393] "Audio data" refers to acoustic information collected in real time, which is used as material for emotion analysis.

[0394] "Means of analysis" refers to methods or devices for processing collected data and understanding customer emotions and circumstances.

[0395] "Means for understanding the status of multiple customers" refers to a method or device for understanding the status of each customer and determining the necessary actions based on data obtained from multiple customers.

[0396] "Means for automatically suggesting optimal customer service responses" refers to a method or device for suggesting the most appropriate response to a customer to staff based on analysis results.

[0397] "Means of monitoring the store environment" refers to methods or devices used to observe and optimize environmental elements such as lighting and temperature within a store.

[0398] "Means for dynamically updating services" refers to methods or devices for receiving customer feedback in real time and flexibly changing the content of services based on that information.

[0399] "Means for transmitting suggestions and notifications to staff devices" refers to a method or device for quickly communicating analysis results and suggestions to customer service staff.

[0400] To implement this invention, it is first necessary to configure a system to improve the customer experience in physical stores. This begins with acquiring and analyzing customer video and audio data in real time using smart glasses. The acquired data is transmitted via Wi-Fi to a server in the cloud. On the server, emotion analysis software runs to analyze the customer's emotional state. Software such as Microsoft Azure Cognitive Services is used for this analysis.

[0401] Based on the analyzed emotional state of the customer, the server suggests the most appropriate response to the customer service staff wearing smart glasses. This suggestion provides the staff with the information they need to interact with customers in real time. For example, if a customer appears confused in the store, the server might send a suggestion to the staff such as, "You seem to be having trouble. Please let us know if there's anything we can do to help."

[0402] This system allows staff to quickly understand customer needs and respond appropriately. This not only contributes to improved customer satisfaction but also enhances the overall service quality of the store.

[0403] An example of a prompt is, "Please advise how to respond if the customer is confused." This is used in situations where a generative AI model is used to generate suggestions for staff.

[0404] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0405] Step 1:

[0406] The server receives video and audio data from the smart glasses in real time. This input includes information such as the customer's facial expressions and tone of voice. The server receives this data as input and prepares it to be sent to emotion analysis software.

[0407] Step 2:

[0408] The server analyzes the received video and audio data using emotion analysis software. Data processing involves extracting customer facial features from the video data and analyzing emotional patterns from the audio data. This process generates an output that identifies the customer's emotional state.

[0409] Step 3:

[0410] The server uses an AI model based on the analysis results to generate suggestions for the staff. The input is the analysis results from step 2, and the output is the optimal customer service suggestion. This output suggestion is sent to the next step in text format.

[0411] Step 4:

[0412] The server notifies the customer service staff of the generated suggestions via their smart glasses. This allows the staff to respond to the customer's situation in real time. The input is the suggested content output in step 3, and the output is the notification to the staff.

[0413] Step 5:

[0414] Users (customers) provide feedback based on their in-store experience. This feedback, provided via a terminal (smart glasses or other device), is sent to a server. As part of data processing, the feedback data is used to generate suggestions for future visits. The input is user feedback, and the output is data for improved suggestions.

[0415] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0416] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0417] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0418] [Third Embodiment]

[0419] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0420] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0421] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0422] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0423] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0424] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0425] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0426] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0427] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0428] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0429] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0430] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0431] This invention is a smart assistant system designed to improve work efficiency and comfort in an office environment. This system includes schedule management, equipment management, and suggestion / notification functions to support the user's work. Its specific operation is described below.

[0432] First, when a user logs into the system, the server retrieves the user's schedule data from an external calendar service API. Based on the retrieved data, the server analyzes the user's schedule and detects duplicates and unprocessed tasks.

[0433] Next, to arrange the meeting, the server adjusts the availability of other participants based on the analysis results and proposes the optimal meeting time. This aims to facilitate smooth scheduling among users.

[0434] Furthermore, regarding facility management, the server works in conjunction with temperature sensors and lighting control systems to monitor the office environment. This ensures that room temperature and lighting are always maintained at optimal levels, creating a comfortable working environment.

[0435] Furthermore, the terminal notifies the user of changes to the proposed schedule and environment settings. If the user provides feedback, the schedule and equipment settings are automatically updated accordingly. This enables quick and flexible work responses.

[0436] For example, if an employee has a meeting scheduled for 2 PM, but one of the participants already has another meeting scheduled, the server will detect this and suggest a meeting at 3 PM instead. This suggestion will be notified to the user via their terminal, and once the user approves, the schedules of all involved parties will be automatically updated.

[0437] Thus, the present invention is a system that streamlines the management of the entire office environment and supports users so that they can concentrate on their core tasks.

[0438] The following describes the processing flow.

[0439] Step 1:

[0440] The user logs into the system and sends a request to begin schedule management. This is done via a smartphone or PC application.

[0441] Step 2:

[0442] Upon receiving login information, the server executes an authentication process and accesses the calendar service API associated with the user's account to retrieve schedule data.

[0443] Step 3:

[0444] The server analyzes the acquired schedule data and applies an analysis algorithm to detect overlapping schedules and unprocessed tasks.

[0445] Step 4:

[0446] Based on the analysis results, the server suggests meeting times by comparing them with the schedule information of other participants. It selects the optimal date and time for the meeting, taking into account the availability of all participants' schedules.

[0447] Step 5:

[0448] The terminal notifies the user of meeting time proposals sent from the server and displays them on the user's screen. The user can then choose to accept or reject the proposal.

[0449] Step 6:

[0450] The user submits feedback on the proposal from their device. If approved, the server updates the schedule and sends a notification to all relevant parties.

[0451] Step 7:

[0452] The server works in conjunction with the facility management system to monitor and optimize office temperature and lighting conditions. Based on sensor data, it automatically implements necessary changes.

[0453] Step 8:

[0454] The terminal notifies the user of any final schedule or setting changes and requests feedback as needed. This ensures continuous interaction between the user and the system.

[0455] (Example 1)

[0456] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0457] Improving work efficiency and maintaining comfort in the office environment are important challenges in many modern workplaces. However, scheduling meetings and optimally managing equipment remain time-consuming, and it is necessary to appropriately and promptly incorporate user feedback. This invention aims to address these challenges and create a more efficient office environment.

[0458] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0459] In this invention, the server includes means for acquiring and analyzing information data in real time, means for managing the activity schedules of multiple users and automatically proposing optimal meeting times, and means for monitoring the operating status of the equipment and optimizing lighting and temperature. This enables efficient work execution through real-time acquisition and analysis of information data, improved user activity efficiency through personalized suggestions, and the maintenance of a comfortable environment through appropriate equipment operation.

[0460] "Information data" refers to information such as users' schedules, equipment status, and user feedback within the office environment.

[0461] "Real-time" refers to a temporal process in which data is processed and analyzed simultaneously with its generation.

[0462] "Analysis" is a method of examining acquired data to identify specific patterns or problems.

[0463] "Planned activities" refer to the activities or events that a user plans to carry out within a specific period of time.

[0464] "Meeting time" refers to the time spent in meetings or discussions where multiple users gather.

[0465] "Equipment operating status" refers to the current operating status and performance of equipment and systems used within the office.

[0466] "Proposal" refers to the optimal solution or option presented to the user based on the analyzed data.

[0467] "Optimizing lighting and temperature" refers to adjusting the visual and thermal conditions of the office environment to be optimal for users.

[0468] This system acquires and analyzes various data in real time to improve work efficiency and comfort in the office environment, and provides optimal suggestions to users.

[0469] The server retrieves user schedule data using an external calendar service API. To ensure security, OAuth 2.0 authentication is used for data retrieval. The retrieved schedule data is then analyzed to detect time overlaps and unprocessed tasks. This analysis is made more accurate by utilizing historical data and patterns stored in the database.

[0470] Furthermore, the server monitors the office environment in conjunction with temperature sensors and lighting control systems. Specifically, it automatically adjusts heating and cooling when the temperature exceeds a certain range. It also has a function that uses motion sensors to turn on the lights only when needed.

[0471] The terminal provides users with suggestions and notifications from the server. This allows users to instantly learn about new meeting proposals and environment settings, enabling quick decision-making.

[0472] When a user provides feedback, the server reviews the entire system's operation based on that feedback. For example, if feedback is received that "the meeting time is proposed too early," the settings will be changed to prioritize afternoon time slots over morning ones for future proposals.

[0473] For example, if a user has scheduled a meeting for 10:00 AM, but analysis reveals that one of the other participants is already scheduled to attend another meeting, the server will detect this and suggest 11:00 AM as the new time slot. This new suggestion will be notified to the user via their device, and the schedule will be automatically updated once the user approves.

[0474] An example of a prompt to input into the generating AI model would be, "Please describe a system that analyzes a user's schedule and suggests the optimal meeting time." This prompt would enable the AI ​​model to generate even more accurate suggestions for improving work efficiency.

[0475] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0476] Step 1:

[0477] The user logs into the system. As input, the user enters their authentication information into the terminal and sends it to the server. The server uses an external authentication service to verify the authentication information and confirm the user's permissions. As output, if authentication is successful, the user's dashboard is displayed on the terminal.

[0478] Step 2:

[0479] The server retrieves the user's schedule data. Using the user ID as input, it sends a request to the calendar service API. The server analyzes the data retrieved from the API to detect duplicate appointments and unprocessed tasks. The analysis results are then displayed on the dashboard.

[0480] Step 3:

[0481] The server generates a meeting proposal. It uses the user's and other participants' schedule information as input. The server uses an analysis algorithm to compare available time slots and calculate the optimal meeting time. The proposed meeting time is then notified to the terminal as output.

[0482] Step 4:

[0483] The server monitors the equipment. It receives data from temperature and lighting sensors as input. The server analyzes this data to determine if the indoor environment is under optimal conditions. As output, it sends instructions to the equipment to adjust lighting and temperature as needed.

[0484] Step 5:

[0485] The terminal notifies the user. It takes input such as suggestions and configuration changes received from the server. The terminal communicates this information to the user through the user interface, using methods such as pop-up notifications and audio alerts. The output is for the user to review the information and approve or modify it.

[0486] Step 6:

[0487] The user provides feedback. As input, they enter their opinions on the proposed schedule and environmental conditions into the device. The device sends this data to the server. The server analyzes the feedback data and updates the AI ​​model to reflect the changes in future proposals. As output, improved proposals and adjusted environmental settings are provided.

[0488] Step 7:

[0489] The server updates the system. It uses user feedback and environmental change data as input. The server uses a generative AI model to create new suggestions for future use. The output is improved overall work efficiency for users.

[0490] (Application Example 1)

[0491] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0492] In factories, if the operation planning and maintenance of equipment and robots are not carried out efficiently, productivity will decline and significant production stoppages will occur due to machine failures. However, with conventional systems, real-time situation analysis and anomaly detection are difficult, and it has been difficult to provide information to personnel quickly. To solve this problem, a system is needed that continuously monitors the activity status of operating equipment and dynamically proposes and updates the optimal schedule.

[0493] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0494] In this invention, the server includes means for analyzing the activity status of operating equipment and optimizing the operating schedule, means for detecting equipment abnormalities and notifying the responsible person, and means for generating personalized suggestions based on the analyzed data. This enables efficient operation of production equipment within the factory and prompt maintenance response.

[0495] "Video data" refers to visual information acquired by a camera or other recording device.

[0496] "Audio data" refers to auditory information obtained by a sound acquisition device such as a microphone.

[0497] "Symbolic data" refers to information that is represented as meaningful symbols, such as letters or numbers.

[0498] "User" refers to an individual or group that uses the system.

[0499] "Activity plan" refers to the schedule of specific actions and events planned by the user.

[0500] "Meeting time" refers to the time set aside for multiple participants to gather and engage in a specific activity.

[0501] "Equipment" is a general term for devices or equipment that have a specific function.

[0502] "Opinions" refer to the content of feedback and evaluations that users provide to the system.

[0503] An "apparatus" is a machine or electronic device used for a specific purpose.

[0504] "Activity status" refers to information about the current operating state of a device or system.

[0505] "Anomaly" is a term that refers to an action or state that is not normally expected.

[0506] A "person in charge" is someone who is assigned to handle a specific task or problem.

[0507] An "information network" is a network used to transmit information using communication technology.

[0508] "Location-based equipment management" refers to the act of monitoring and optimizing the status of equipment located in a specific location.

[0509] A "maintenance plan" refers to a planned maintenance activity carried out to ensure that equipment and systems continue to operate normally.

[0510] In this embodiment of the invention, the system is configured as follows, mainly consisting of a server, a terminal, and a user.

[0511] First, the server acquires real-time activity data from sensors installed on equipment within the factory and analyzes that data. These sensors include vibration sensors and temperature sensors. The data is processed by a Raspberry Pi and transmitted to the server via the network. The server uses programming languages ​​and frameworks such as Python and Django to store the data in a database and perform analysis.

[0512] Next, the server optimizes the operating schedule of each device based on the analyzed data and has a function to notify the person in charge if an anomaly is detected. This notification is made via a device such as a smartphone or smart glasses carried by the person in charge. This ensures that the equipment in the factory is constantly monitored to keep it operating in optimal condition, and necessary maintenance work can be carried out quickly.

[0513] Furthermore, users can receive suggestions from the server via their terminals and submit feedback. This user feedback is analyzed by the server and incorporated into the latest equipment operation schedule. The user interface design utilizes a generative AI model to enhance the user experience and generate natural-sounding prompts.

[0514] For example, if a device detects more vibration than usual, the server recognizes this as an anomaly and sends a notification to the responsible person's terminal stating, "An anomaly has been detected in device A. Please check the details and perform maintenance." This allows the responsible person to immediately check the status and take the necessary action.

[0515] An example of a prompt to the generated AI model is, "Explain scheduling methods that support the efficient operation of factory equipment." This prompt allows the system to generate information to provide appropriate support to the user.

[0516] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0517] Step 1:

[0518] The server receives activity data in real time from sensors attached to equipment within the factory. The input consists of sensor data related to vibration and temperature. The server receives this data from the sensors, converts it to a digital format via a Raspberry Pi, and stores it in a database.

[0519] Step 2:

[0520] The server analyzes stored sensor data and evaluates the operating status of the device. The input is sensor data stored in a database. Based on the data, the server applies an analysis algorithm to identify abnormal patterns and detect unusual vibrations and temperature changes.

[0521] Step 3:

[0522] If an anomaly is detected, the server sends a notification to the responsible person. The input is the anomaly detection information obtained as a result of the analysis. The server generates a warning message regarding the anomaly and sends a notification to the responsible person's smartphone or smart glasses via the network.

[0523] Step 4:

[0524] The user checks notifications sent from the server via their terminal. The input is the notification message sent from the server. Based on the received notification, the user checks the status of the equipment on-site and performs maintenance as needed.

[0525] Step 5:

[0526] Users send feedback to the server about the status of the equipment and maintenance results. The input is the feedback information reported by the user. The server updates the database based on the received feedback and reflects it in the future equipment operation schedule.

[0527] Step 6:

[0528] The server optimizes the operating schedule and maintenance plan based on new feedback and analysis results. Inputs include feedback information and updated analysis data. The server uses this information to apply optimization algorithms and synchronizes the updated schedule with a cloud calendar service.

[0529] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0530] This invention is a smart assistant system that combines an emotion engine to analyze the user's emotional state, aiming to improve work efficiency and comfort in the office environment. This system uses the user's video and audio data to perform real-time emotion analysis, and uses the results to manage schedules and optimize equipment.

[0531] First, when a user logs into the system, the server collects video and audio data through the camera and microphone. This data is sent to the emotion engine, where the user's emotional state is analyzed in real time. The emotion engine determines whether the user is stressed or relaxed.

[0532] Based on the analysis results, the server adjusts schedule management. For example, if it determines that a user is experiencing stress, it adjusts suggested meeting times to be more user-friendly. Furthermore, in terms of optimizing the environment, it changes room temperature and lighting brightness to match the user's emotional state.

[0533] Meanwhile, the device notifies the user of the analysis results obtained from the emotion engine and the suggestions based on those results. Based on this, the user can provide feedback through the device. This feedback is sent to the server and reflected in future suggestions and settings.

[0534] As a concrete example, when a user begins to feel tired during work, the system analyzes data obtained from the camera and microphone using an emotion engine. If "fatigue" is detected through the analysis, the server will suggest postponing the meeting or instruct the room lighting to be changed to a warmer color. This change is notified to the user via the terminal and is implemented only after the user approves it. This invention greatly improves user efficiency and comfort by providing an appropriate work environment that takes emotional states into account in real time.

[0535] The following describes the processing flow.

[0536] Step 1:

[0537] The system starts operating when the user logs in and is ready to begin work. Login is performed using authentication credentials.

[0538] Step 2:

[0539] The server collects video and audio data from the camera and microphone in real time. This data is securely stored and immediately sent to the emotion engine.

[0540] Step 3:

[0541] The server's emotion engine analyzes video and audio data to determine the user's emotional state. This analysis includes the ability to detect states such as "stress," "concentration," and "relaxation" from changes in facial expressions and tone of voice.

[0542] Step 4:

[0543] The server sends feedback to the scheduling function based on the user's emotional state. For example, if the server determines that the user is stressed, it generates suggestions to reschedule existing meetings or tasks.

[0544] Step 5:

[0545] The server optimizes office equipment based on the analysis results. Specifically, it issues instructions such as adjusting the temperature to create a comfortable room temperature or changing the lighting to create an eye-friendly environment.

[0546] Step 6:

[0547] The device notifies the user of suggestions and configuration changes generated by the server. These notifications appear as pop-ups or audio alerts, prompting immediate user response.

[0548] Step 7:

[0549] Users review the suggestions notified on their devices and accept or request changes based on their preferences. This allows user feedback to be reflected in the system.

[0550] Step 8:

[0551] The server dynamically updates schedules and equipment settings based on user feedback. These updates are automatically reflected in stakeholders and systems, allowing for continuous optimization.

[0552] (Example 2)

[0553] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0554] In traditional work environments, the lack of consideration for users' emotional states and stress levels tended to lead to decreased work efficiency. Furthermore, optimizing the environment was difficult due to the difficulty in making adjustments based on individual circumstances, resulting in a reliance on standardized settings. This directly impacted user satisfaction and work efficiency. Additionally, even in remote management environments, there was a challenge in achieving sufficient collaboration among remote locations.

[0555] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0556] In this invention, the server includes a device means for collecting and analyzing video and audio information in real time, an information processing device means for determining the emotional state of individual users, and a device means for monitoring environmental settings and optimizing lighting and temperature based on the user's emotional state. This enables dynamic adjustment of the work environment to reflect the user's emotional state, improving work efficiency and comfort. Furthermore, it enables optimization of remote equipment management even in remote environments.

[0557] "Visual information" refers to all visual data acquired through imaging devices such as cameras.

[0558] "Audio information" refers to all audio data acquired through sound collection devices such as microphones.

[0559] A "real-time data collection and analysis device" refers to a device that has the ability to collect data with minimal time delay and to process and analyze that data immediately.

[0560] "Individual users" refers to each person who uses a service or system, and the term "individual users" means that the processing is tailored to that person.

[0561] An "information processing device for determining emotional state" refers to a device that analyzes data collected from users and has the function of identifying and determining a person's current psychological and emotional state.

[0562] A "device for optimizing environmental settings" refers to a device that has the function of adjusting and optimizing the surrounding physical conditions (e.g., lighting, temperature) according to the user's condition.

[0563] A "device for receiving feedback and incorporating it into future adjustments" refers to a device that receives opinions and reactions from users and uses them to improve and adjust the system's operation and environment settings.

[0564] A "device that transmits suggestions and notifications to a user's terminal" refers to a device that has the function of transmitting information and suggestions generated from a server or system to an electronic device owned by the user to inform them.

[0565] This invention is a smart assistant system designed to improve work efficiency and user comfort in an office environment. This system operates through the coordinated efforts of a server, terminal, and user.

[0566] When a user logs in, the server collects video and audio information in real time via cameras and microphones installed in the office. This data is processed by the server, and an AI-based emotion analysis engine is used to determine the user's emotional state. At this time, deep learning algorithms are used to analyze facial expressions, voice tone, and intonation.

[0567] Based on the analyzed emotional state, the server adjusts the user's schedule management and dynamically determines the optimal meeting time. For example, if stress is detected, the scheduling is flexibly adjusted while considering the importance of the meeting. In addition, as part of optimizing the environment, the room lighting and temperature are automatically adjusted according to the user's emotional state to maintain a comfortable working environment.

[0568] The terminal's role is to send suggestions and notifications from the server to the user. The user receives these suggestions through the terminal and provides approval and feedback. This feedback is sent to the server and used to inform future suggestions and environment adjustments.

[0569] For example, if a user starts feeling fatigued during work, the server's emotion analysis engine will detect "fatigue." Based on this, the server will suggest postponing the meeting and send instructions to the terminal to change the lighting to a warmer color. If the user approves, the system will implement the changes.

[0570] Furthermore, an example of a prompt using the generative AI model is, "If the user is feeling nervous but has an important meeting coming up, please suggest the best course of action." Through this prompt, the AI ​​model generates the optimal response, which is then reflected in the actual system operation on the server.

[0571] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0572] Step 1:

[0573] When a user logs into the system, the server collects video and audio information from the office via the camera and microphone. This input data serves as material for real-time analysis of changes in the user's facial expressions and voice. The server converts the data into an appropriate format and sends it to the emotion analysis engine.

[0574] Step 2:

[0575] The server analyzes the data collected by the emotion analysis engine. Specifically, it uses deep learning algorithms to perform facial expression recognition and voice tone analysis. In this process, the data is classified into emotional states such as stress, fatigue, and relaxation. The analysis results are generated as output, which becomes the basis data for the next step.

[0576] Step 3:

[0577] The server dynamically adjusts the user's schedule based on the sentiment analysis results. It receives the analysis results as input and re-evaluates the importance and priority of meetings within the overall schedule. Even if stress is detected, it flexibly changes meeting times to allow the user to work efficiently.

[0578] Step 4:

[0579] The server optimizes the facilities in response to the user's emotional state. Specifically, based on the output analysis results, it changes the room lighting to an appropriate color tone and adjusts the temperature. As a result, a comfortable working environment is provided as the final output.

[0580] Step 5:

[0581] The terminal notifies the user of suggestions and adjustments from the server. These notifications are provided via email or pop-up messages, and the user reviews and approves them as input for the process. The user's feedback then becomes input for the next step.

[0582] Step 6:

[0583] Users provide feedback on suggestions received through their terminals. This feedback information is sent to the server and used to improve future environment adjustments and scheduling. This allows the system to be continuously optimized, enabling more efficient output.

[0584] (Application Example 2)

[0585] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0586] There is a problem in that it is difficult for customers to have a more comfortable and satisfying shopping experience in physical stores. In particular, it is difficult for staff to understand the emotional state of every customer and respond appropriately. Another challenge is having the flexibility to immediately reflect customer feedback in service.

[0587] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0588] In this invention, the server includes means for acquiring and analyzing video and audio data in real time, means for understanding the situation of multiple customers and automatically proposing the optimal customer service response, and means for receiving feedback from each customer and dynamically updating the service. This makes shopping at physical stores more comfortable for customers and enables staff to quickly respond appropriately to the emotional state of customers.

[0589] "Video data" refers to visual information acquired in real time, and this information is used as the basis for analysis.

[0590] "Audio data" refers to acoustic information collected in real time, which is used as material for emotion analysis.

[0591] "Means of analysis" refers to methods or devices for processing collected data and understanding customer emotions and circumstances.

[0592] "Means for understanding the status of multiple customers" refers to a method or device for understanding the status of each customer and determining the necessary actions based on data obtained from multiple customers.

[0593] "Means for automatically suggesting optimal customer service responses" refers to a method or device for suggesting the most appropriate response to a customer to staff based on analysis results.

[0594] "Means of monitoring the store environment" refers to methods or devices used to observe and optimize environmental elements such as lighting and temperature within a store.

[0595] "Means for dynamically updating services" refers to methods or devices for receiving customer feedback in real time and flexibly changing the content of services based on that information.

[0596] "Means for transmitting suggestions and notifications to staff devices" refers to a method or device for quickly communicating analysis results and suggestions to customer service staff.

[0597] To implement this invention, it is first necessary to configure a system to improve the customer experience in physical stores. This begins with acquiring and analyzing customer video and audio data in real time using smart glasses. The acquired data is transmitted via Wi-Fi to a server in the cloud. On the server, emotion analysis software runs to analyze the customer's emotional state. Software such as Microsoft Azure Cognitive Services is used for this analysis.

[0598] Based on the analyzed emotional state of the customer, the server suggests the most appropriate response to the customer service staff wearing smart glasses. This suggestion provides the staff with the information they need to interact with customers in real time. For example, if a customer appears confused in the store, the server might send a suggestion to the staff such as, "You seem to be having trouble. Please let us know if there's anything we can do to help."

[0599] This system allows staff to quickly understand customer needs and respond appropriately. This not only contributes to improved customer satisfaction but also enhances the overall service quality of the store.

[0600] An example of a prompt is, "Please advise how to respond if the customer is confused." This is used in situations where a generative AI model is used to generate suggestions for staff.

[0601] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0602] Step 1:

[0603] The server receives video and audio data from the smart glasses in real time. This input includes information such as the customer's facial expressions and tone of voice. The server receives this data as input and prepares it to be sent to emotion analysis software.

[0604] Step 2:

[0605] The server analyzes the received video and audio data using emotion analysis software. Data processing involves extracting customer facial features from the video data and analyzing emotional patterns from the audio data. This process generates an output that identifies the customer's emotional state.

[0606] Step 3:

[0607] The server uses an AI model based on the analysis results to generate suggestions for the staff. The input is the analysis results from step 2, and the output is the optimal customer service suggestion. This output suggestion is sent to the next step in text format.

[0608] Step 4:

[0609] The server notifies the customer service staff of the generated suggestions via their smart glasses. This allows the staff to respond to the customer's situation in real time. The input is the suggested content output in step 3, and the output is the notification to the staff.

[0610] Step 5:

[0611] Users (customers) provide feedback based on their in-store experience. This feedback, provided via a terminal (smart glasses or other device), is sent to a server. As part of data processing, the feedback data is used to generate suggestions for future visits. The input is user feedback, and the output is data for improved suggestions.

[0612] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0613] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0614] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0615] [Fourth Embodiment]

[0616] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0617] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0618] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0619] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0620] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0621] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0622] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0623] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0624] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0625] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0626] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0627] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0628] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0629] This invention is a smart assistant system designed to improve work efficiency and comfort in an office environment. This system includes schedule management, equipment management, and suggestion / notification functions to support the user's work. Its specific operation is described below.

[0630] First, when a user logs into the system, the server retrieves the user's schedule data from an external calendar service API. Based on the retrieved data, the server analyzes the user's schedule and detects duplicates and unprocessed tasks.

[0631] Next, to arrange the meeting, the server adjusts the availability of other participants based on the analysis results and proposes the optimal meeting time. This aims to facilitate smooth scheduling among users.

[0632] Furthermore, regarding facility management, the server works in conjunction with temperature sensors and lighting control systems to monitor the office environment. This ensures that room temperature and lighting are always maintained at optimal levels, creating a comfortable working environment.

[0633] Furthermore, the terminal notifies the user of changes to the proposed schedule and environment settings. If the user provides feedback, the schedule and equipment settings are automatically updated accordingly. This enables quick and flexible work responses.

[0634] For example, if an employee has a meeting scheduled for 2 PM, but one of the participants already has another meeting scheduled, the server will detect this and suggest a meeting at 3 PM instead. This suggestion will be notified to the user via their terminal, and once the user approves, the schedules of all involved parties will be automatically updated.

[0635] Thus, the present invention is a system that streamlines the management of the entire office environment and supports users so that they can concentrate on their core tasks.

[0636] The following describes the processing flow.

[0637] Step 1:

[0638] The user logs into the system and sends a request to begin schedule management. This is done via a smartphone or PC application.

[0639] Step 2:

[0640] Upon receiving login information, the server executes an authentication process and accesses the calendar service API associated with the user's account to retrieve schedule data.

[0641] Step 3:

[0642] The server analyzes the acquired schedule data and applies an analysis algorithm to detect overlapping schedules and unprocessed tasks.

[0643] Step 4:

[0644] Based on the analysis results, the server suggests meeting times by comparing them with the schedule information of other participants. It selects the optimal date and time for the meeting, taking into account the availability of all participants' schedules.

[0645] Step 5:

[0646] The terminal notifies the user of meeting time proposals sent from the server and displays them on the user's screen. The user can then choose to accept or reject the proposal.

[0647] Step 6:

[0648] The user submits feedback on the proposal from their device. If approved, the server updates the schedule and sends a notification to all relevant parties.

[0649] Step 7:

[0650] The server works in conjunction with the facility management system to monitor and optimize office temperature and lighting conditions. Based on sensor data, it automatically implements necessary changes.

[0651] Step 8:

[0652] The terminal notifies the user of any final schedule or setting changes and requests feedback as needed. This ensures continuous interaction between the user and the system.

[0653] (Example 1)

[0654] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0655] Improving work efficiency and maintaining comfort in the office environment are important challenges in many modern workplaces. However, scheduling meetings and optimally managing equipment remain time-consuming, and it is necessary to appropriately and promptly incorporate user feedback. This invention aims to address these challenges and create a more efficient office environment.

[0656] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0657] In this invention, the server includes means for acquiring and analyzing information data in real time, means for managing the activity schedules of multiple users and automatically proposing optimal meeting times, and means for monitoring the operating status of the equipment and optimizing lighting and temperature. This enables efficient work execution through real-time acquisition and analysis of information data, improved user activity efficiency through personalized suggestions, and the maintenance of a comfortable environment through appropriate equipment operation.

[0658] "Information data" refers to information such as users' schedules, equipment status, and user feedback within the office environment.

[0659] "Real-time" refers to a temporal process in which data is processed and analyzed simultaneously with its generation.

[0660] "Analysis" is a method of examining acquired data to identify specific patterns or problems.

[0661] "Planned activities" refer to the activities or events that a user plans to carry out within a specific period of time.

[0662] "Meeting time" refers to the time spent in meetings or discussions where multiple users gather.

[0663] "Equipment operating status" refers to the current operating status and performance of equipment and systems used within the office.

[0664] "Proposal" refers to the optimal solution or option presented to the user based on the analyzed data.

[0665] "Optimizing lighting and temperature" refers to adjusting the visual and thermal conditions of the office environment to be optimal for users.

[0666] This system acquires and analyzes various data in real time to improve work efficiency and comfort in the office environment, and provides optimal suggestions to users.

[0667] The server retrieves user schedule data using an external calendar service API. To ensure security, OAuth 2.0 authentication is used for data retrieval. The retrieved schedule data is then analyzed to detect time overlaps and unprocessed tasks. This analysis is made more accurate by utilizing historical data and patterns stored in the database.

[0668] Furthermore, the server monitors the office environment in conjunction with temperature sensors and lighting control systems. Specifically, it automatically adjusts heating and cooling when the temperature exceeds a certain range. It also has a function that uses motion sensors to turn on the lights only when needed.

[0669] The terminal provides users with suggestions and notifications from the server. This allows users to instantly learn about new meeting proposals and environment settings, enabling quick decision-making.

[0670] When a user provides feedback, the server reviews the entire system's operation based on that feedback. For example, if feedback is received that "the meeting time is proposed too early," the settings will be changed to prioritize afternoon time slots over morning ones for future proposals.

[0671] For example, if a user has scheduled a meeting for 10:00 AM, but analysis reveals that one of the other participants is already scheduled to attend another meeting, the server will detect this and suggest 11:00 AM as the new time slot. This new suggestion will be notified to the user via their device, and the schedule will be automatically updated once the user approves.

[0672] An example of a prompt to input into the generating AI model would be, "Please describe a system that analyzes a user's schedule and suggests the optimal meeting time." This prompt would enable the AI ​​model to generate even more accurate suggestions for improving work efficiency.

[0673] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0674] Step 1:

[0675] The user logs into the system. As input, the user enters their authentication information into the terminal and sends it to the server. The server uses an external authentication service to verify the authentication information and confirm the user's permissions. As output, if authentication is successful, the user's dashboard is displayed on the terminal.

[0676] Step 2:

[0677] The server retrieves the user's schedule data. Using the user ID as input, it sends a request to the calendar service API. The server analyzes the data retrieved from the API to detect duplicate appointments and unprocessed tasks. The analysis results are then displayed on the dashboard.

[0678] Step 3:

[0679] The server generates a meeting proposal. It uses the user's and other participants' schedule information as input. The server uses an analysis algorithm to compare available time slots and calculate the optimal meeting time. The proposed meeting time is then notified to the terminal as output.

[0680] Step 4:

[0681] The server monitors the equipment. It receives data from temperature and lighting sensors as input. The server analyzes this data to determine if the indoor environment is under optimal conditions. As output, it sends instructions to the equipment to adjust lighting and temperature as needed.

[0682] Step 5:

[0683] The terminal notifies the user. It takes input such as suggestions and configuration changes received from the server. The terminal communicates this information to the user through the user interface, using methods such as pop-up notifications and audio alerts. The output is for the user to review the information and approve or modify it.

[0684] Step 6:

[0685] The user provides feedback. As input, they enter their opinions on the proposed schedule and environmental conditions into the device. The device sends this data to the server. The server analyzes the feedback data and updates the AI ​​model to reflect the changes in future proposals. As output, improved proposals and adjusted environmental settings are provided.

[0686] Step 7:

[0687] The server updates the system. It uses user feedback and environmental change data as input. The server uses a generative AI model to create new suggestions for future use. The output is improved overall work efficiency for users.

[0688] (Application Example 1)

[0689] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0690] In factories, if the operation planning and maintenance of equipment and robots are not carried out efficiently, productivity will decline and significant production stoppages will occur due to machine failures. However, with conventional systems, real-time situation analysis and anomaly detection are difficult, and it has been difficult to provide information to personnel quickly. To solve this problem, a system is needed that continuously monitors the activity status of operating equipment and dynamically proposes and updates the optimal schedule.

[0691] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0692] In this invention, the server includes means for analyzing the activity status of operating equipment and optimizing the operating schedule, means for detecting equipment abnormalities and notifying the responsible person, and means for generating personalized suggestions based on the analyzed data. This enables efficient operation of production equipment within the factory and prompt maintenance response.

[0693] "Video data" refers to visual information acquired by a camera or other recording device.

[0694] "Audio data" refers to auditory information obtained by a sound acquisition device such as a microphone.

[0695] "Symbolic data" refers to information that is represented as meaningful symbols, such as letters or numbers.

[0696] "User" refers to an individual or group that uses the system.

[0697] "Activity plan" refers to the schedule of specific actions and events planned by the user.

[0698] "Meeting time" refers to the time set aside for multiple participants to gather and engage in a specific activity.

[0699] "Equipment" is a general term for devices or equipment that have a specific function.

[0700] "Opinions" refer to the content of feedback and evaluations that users provide to the system.

[0701] An "apparatus" is a machine or electronic device used for a specific purpose.

[0702] "Activity status" refers to information about the current operating state of a device or system.

[0703] "Anomaly" is a term that refers to an action or state that is not normally expected.

[0704] A "person in charge" is someone who is assigned to handle a specific task or problem.

[0705] An "information network" is a network used to transmit information using communication technology.

[0706] "Location-based equipment management" refers to the act of monitoring and optimizing the status of equipment located in a specific location.

[0707] A "maintenance plan" refers to a planned maintenance activity carried out to ensure that equipment and systems continue to operate normally.

[0708] In this embodiment of the invention, the system is configured as follows, mainly consisting of a server, a terminal, and a user.

[0709] First, the server acquires real-time activity data from sensors installed on equipment within the factory and analyzes that data. These sensors include vibration sensors and temperature sensors. The data is processed by a Raspberry Pi and transmitted to the server via the network. The server uses programming languages ​​and frameworks such as Python and Django to store the data in a database and perform analysis.

[0710] Next, the server optimizes the operating schedule of each device based on the analyzed data and has a function to notify the person in charge if an anomaly is detected. This notification is made via a device such as a smartphone or smart glasses carried by the person in charge. This ensures that the equipment in the factory is constantly monitored to keep it operating in optimal condition, and necessary maintenance work can be carried out quickly.

[0711] Furthermore, users can receive suggestions from the server via their terminals and submit feedback. This user feedback is analyzed by the server and incorporated into the latest equipment operation schedule. The user interface design utilizes a generative AI model to enhance the user experience and generate natural-sounding prompts.

[0712] For example, if a device detects more vibration than usual, the server recognizes this as an anomaly and sends a notification to the responsible person's terminal stating, "An anomaly has been detected in device A. Please check the details and perform maintenance." This allows the responsible person to immediately check the status and take the necessary action.

[0713] An example of a prompt to the generated AI model is, "Explain scheduling methods that support the efficient operation of factory equipment." This prompt allows the system to generate information to provide appropriate support to the user.

[0714] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0715] Step 1:

[0716] The server receives activity data in real time from sensors attached to equipment within the factory. The input consists of sensor data related to vibration and temperature. The server receives this data from the sensors, converts it to a digital format via a Raspberry Pi, and stores it in a database.

[0717] Step 2:

[0718] The server analyzes stored sensor data and evaluates the operating status of the device. The input is sensor data stored in a database. Based on the data, the server applies an analysis algorithm to identify abnormal patterns and detect unusual vibrations and temperature changes.

[0719] Step 3:

[0720] If an anomaly is detected, the server sends a notification to the responsible person. The input is the anomaly detection information obtained as a result of the analysis. The server generates a warning message regarding the anomaly and sends a notification to the responsible person's smartphone or smart glasses via the network.

[0721] Step 4:

[0722] The user checks notifications sent from the server via their terminal. The input is the notification message sent from the server. Based on the received notification, the user checks the status of the equipment on-site and performs maintenance as needed.

[0723] Step 5:

[0724] Users send feedback to the server about the status of the equipment and maintenance results. The input is the feedback information reported by the user. The server updates the database based on the received feedback and reflects it in the future equipment operation schedule.

[0725] Step 6:

[0726] The server optimizes the operating schedule and maintenance plan based on new feedback and analysis results. Inputs include feedback information and updated analysis data. The server uses this information to apply optimization algorithms and synchronizes the updated schedule with a cloud calendar service.

[0727] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0728] This invention is a smart assistant system that combines an emotion engine to analyze the user's emotional state, aiming to improve work efficiency and comfort in the office environment. This system uses the user's video and audio data to perform real-time emotion analysis, and uses the results to manage schedules and optimize equipment.

[0729] First, when a user logs into the system, the server collects video and audio data through the camera and microphone. This data is sent to the emotion engine, where the user's emotional state is analyzed in real time. The emotion engine determines whether the user is stressed or relaxed.

[0730] Based on the analysis results, the server adjusts schedule management. For example, if it determines that a user is experiencing stress, it adjusts suggested meeting times to be more user-friendly. Furthermore, in terms of optimizing the environment, it changes room temperature and lighting brightness to match the user's emotional state.

[0731] Meanwhile, the device notifies the user of the analysis results obtained from the emotion engine and the suggestions based on those results. Based on this, the user can provide feedback through the device. This feedback is sent to the server and reflected in future suggestions and settings.

[0732] As a concrete example, when a user begins to feel tired during work, the system analyzes data obtained from the camera and microphone using an emotion engine. If "fatigue" is detected through the analysis, the server will suggest postponing the meeting or instruct the room lighting to be changed to a warmer color. This change is notified to the user via the terminal and is implemented only after the user approves it. This invention greatly improves user efficiency and comfort by providing an appropriate work environment that takes emotional states into account in real time.

[0733] The following describes the processing flow.

[0734] Step 1:

[0735] The system starts operating when the user logs in and is ready to begin work. Login is performed using authentication credentials.

[0736] Step 2:

[0737] The server collects video and audio data from the camera and microphone in real time. This data is securely stored and immediately sent to the emotion engine.

[0738] Step 3:

[0739] The server's emotion engine analyzes video and audio data to determine the user's emotional state. This analysis includes the ability to detect states such as "stress," "concentration," and "relaxation" from changes in facial expressions and tone of voice.

[0740] Step 4:

[0741] The server sends feedback to the scheduling function based on the user's emotional state. For example, if the server determines that the user is stressed, it generates suggestions to reschedule existing meetings or tasks.

[0742] Step 5:

[0743] The server optimizes office equipment based on the analysis results. Specifically, it issues instructions such as adjusting the temperature to create a comfortable room temperature or changing the lighting to create an eye-friendly environment.

[0744] Step 6:

[0745] The device notifies the user of suggestions and configuration changes generated by the server. These notifications appear as pop-ups or audio alerts, prompting immediate user response.

[0746] Step 7:

[0747] Users review the suggestions notified on their devices and accept or request changes based on their preferences. This allows user feedback to be reflected in the system.

[0748] Step 8:

[0749] The server dynamically updates schedules and equipment settings based on user feedback. These updates are automatically reflected in stakeholders and systems, allowing for continuous optimization.

[0750] (Example 2)

[0751] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0752] In traditional work environments, the lack of consideration for users' emotional states and stress levels tended to lead to decreased work efficiency. Furthermore, optimizing the environment was difficult due to the difficulty in making adjustments based on individual circumstances, resulting in a reliance on standardized settings. This directly impacted user satisfaction and work efficiency. Additionally, even in remote management environments, there was a challenge in achieving sufficient collaboration among remote locations.

[0753] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0754] In this invention, the server includes a device means for collecting and analyzing video and audio information in real time, an information processing device means for determining the emotional state of individual users, and a device means for monitoring environmental settings and optimizing lighting and temperature based on the user's emotional state. This enables dynamic adjustment of the work environment to reflect the user's emotional state, improving work efficiency and comfort. Furthermore, it enables optimization of remote equipment management even in remote environments.

[0755] "Visual information" refers to all visual data acquired through imaging devices such as cameras.

[0756] "Audio information" refers to all audio data acquired through sound collection devices such as microphones.

[0757] A "real-time data collection and analysis device" refers to a device that has the ability to collect data with minimal time delay and to process and analyze that data immediately.

[0758] "Individual users" refers to each person who uses a service or system, and the term "individual users" means that the processing is tailored to that person.

[0759] An "information processing device for determining emotional state" refers to a device that analyzes data collected from users and has the function of identifying and determining a person's current psychological and emotional state.

[0760] A "device for optimizing environmental settings" refers to a device that has the function of adjusting and optimizing the surrounding physical conditions (e.g., lighting, temperature) according to the user's condition.

[0761] A "device for receiving feedback and incorporating it into future adjustments" refers to a device that receives opinions and reactions from users and uses them to improve and adjust the system's operation and environment settings.

[0762] A "device that transmits suggestions and notifications to a user's terminal" refers to a device that has the function of transmitting information and suggestions generated from a server or system to an electronic device owned by the user to inform them.

[0763] This invention is a smart assistant system designed to improve work efficiency and user comfort in an office environment. This system operates through the coordinated efforts of a server, terminal, and user.

[0764] When a user logs in, the server collects video and audio information in real time via cameras and microphones installed in the office. This data is processed by the server, and an AI-based emotion analysis engine is used to determine the user's emotional state. At this time, deep learning algorithms are used to analyze facial expressions, voice tone, and intonation.

[0765] Based on the analyzed emotional state, the server adjusts the user's schedule management and dynamically determines the optimal meeting time. For example, if stress is detected, the scheduling is flexibly adjusted while considering the importance of the meeting. In addition, as part of optimizing the environment, the room lighting and temperature are automatically adjusted according to the user's emotional state to maintain a comfortable working environment.

[0766] The terminal's role is to send suggestions and notifications from the server to the user. The user receives these suggestions through the terminal and provides approval and feedback. This feedback is sent to the server and used to inform future suggestions and environment adjustments.

[0767] For example, if a user starts feeling fatigued during work, the server's emotion analysis engine will detect "fatigue." Based on this, the server will suggest postponing the meeting and send instructions to the terminal to change the lighting to a warmer color. If the user approves, the system will implement the changes.

[0768] Furthermore, an example of a prompt using the generative AI model is, "If the user is feeling nervous but has an important meeting coming up, please suggest the best course of action." Through this prompt, the AI ​​model generates the optimal response, which is then reflected in the actual system operation on the server.

[0769] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0770] Step 1:

[0771] When a user logs into the system, the server collects video and audio information from the office via the camera and microphone. This input data serves as material for real-time analysis of changes in the user's facial expressions and voice. The server converts the data into an appropriate format and sends it to the emotion analysis engine.

[0772] Step 2:

[0773] The server analyzes the data collected by the emotion analysis engine. Specifically, it uses deep learning algorithms to perform facial expression recognition and voice tone analysis. In this process, the data is classified into emotional states such as stress, fatigue, and relaxation. The analysis results are generated as output, which becomes the basis data for the next step.

[0774] Step 3:

[0775] The server dynamically adjusts the user's schedule based on the sentiment analysis results. It receives the analysis results as input and re-evaluates the importance and priority of meetings within the overall schedule. Even if stress is detected, it flexibly changes meeting times to allow the user to work efficiently.

[0776] Step 4:

[0777] The server optimizes the facilities in response to the user's emotional state. Specifically, based on the output analysis results, it changes the room lighting to an appropriate color tone and adjusts the temperature. As a result, a comfortable working environment is provided as the final output.

[0778] Step 5:

[0779] The terminal notifies the user of suggestions and adjustments from the server. These notifications are provided via email or pop-up messages, and the user reviews and approves them as input for the process. The user's feedback then becomes input for the next step.

[0780] Step 6:

[0781] Users provide feedback on suggestions received through their terminals. This feedback information is sent to the server and used to improve future environment adjustments and scheduling. This allows the system to be continuously optimized, enabling more efficient output.

[0782] (Application Example 2)

[0783] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0784] There is a problem in that it is difficult for customers to have a more comfortable and satisfying shopping experience in physical stores. In particular, it is difficult for staff to understand the emotional state of every customer and respond appropriately. Another challenge is having the flexibility to immediately reflect customer feedback in service.

[0785] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0786] In this invention, the server includes means for acquiring and analyzing video and audio data in real time, means for understanding the situation of multiple customers and automatically proposing the optimal customer service response, and means for receiving feedback from each customer and dynamically updating the service. This makes shopping at physical stores more comfortable for customers and enables staff to quickly respond appropriately to the emotional state of customers.

[0787] "Video data" refers to visual information acquired in real time, and this information is used as the basis for analysis.

[0788] "Audio data" refers to acoustic information collected in real time, which is used as material for emotion analysis.

[0789] "Means of analysis" refers to methods or devices for processing collected data and understanding customer emotions and circumstances.

[0790] "Means for understanding the status of multiple customers" refers to a method or device for understanding the status of each customer and determining the necessary actions based on data obtained from multiple customers.

[0791] "Means for automatically suggesting optimal customer service responses" refers to a method or device for suggesting the most appropriate response to a customer to staff based on analysis results.

[0792] "Means of monitoring the store environment" refers to methods or devices used to observe and optimize environmental elements such as lighting and temperature within a store.

[0793] "Means for dynamically updating services" refers to methods or devices for receiving customer feedback in real time and flexibly changing the content of services based on that information.

[0794] "Means for transmitting suggestions and notifications to staff devices" refers to a method or device for quickly communicating analysis results and suggestions to customer service staff.

[0795] To implement this invention, it is first necessary to configure a system to improve the customer experience in physical stores. This begins with acquiring and analyzing customer video and audio data in real time using smart glasses. The acquired data is transmitted via Wi-Fi to a server in the cloud. On the server, emotion analysis software runs to analyze the customer's emotional state. Software such as Microsoft Azure Cognitive Services is used for this analysis.

[0796] Based on the analyzed emotional state of the customer, the server suggests the most appropriate response to the customer service staff wearing smart glasses. This suggestion provides the staff with the information they need to interact with customers in real time. For example, if a customer appears confused in the store, the server might send a suggestion to the staff such as, "You seem to be having trouble. Please let us know if there's anything we can do to help."

[0797] This system allows staff to quickly understand customer needs and respond appropriately. This not only contributes to improved customer satisfaction but also enhances the overall service quality of the store.

[0798] An example of a prompt is, "Please advise how to respond if the customer is confused." This is used in situations where a generative AI model is used to generate suggestions for staff.

[0799] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0800] Step 1:

[0801] The server receives video and audio data from the smart glasses in real time. This input includes information such as the customer's facial expressions and tone of voice. The server receives this data as input and prepares it to be sent to emotion analysis software.

[0802] Step 2:

[0803] The server analyzes the received video and audio data using emotion analysis software. Data processing involves extracting customer facial features from the video data and analyzing emotional patterns from the audio data. This process generates an output that identifies the customer's emotional state.

[0804] Step 3:

[0805] The server uses an AI model based on the analysis results to generate suggestions for the staff. The input is the analysis results from step 2, and the output is the optimal customer service suggestion. This output suggestion is sent to the next step in text format.

[0806] Step 4:

[0807] The server notifies the customer service staff of the generated suggestions via their smart glasses. This allows the staff to respond to the customer's situation in real time. The input is the suggested content output in step 3, and the output is the notification to the staff.

[0808] Step 5:

[0809] Users (customers) provide feedback based on their in-store experience. This feedback, provided via a terminal (smart glasses or other device), is sent to a server. As part of data processing, the feedback data is used to generate suggestions for future visits. The input is user feedback, and the output is data for improved suggestions.

[0810] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0811] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0812] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0813] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0814] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0815] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0816] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0817] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0818] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0819] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0820] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0821] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0822] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0823] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0824] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0825] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0826] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0827] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0828] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0829] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0830] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0831] The following is further disclosed regarding the embodiments described above.

[0832] (Claim 1)

[0833] A means of acquiring and analyzing video data, audio data, and text data in real time,

[0834] A method for managing the schedules of multiple users and automatically suggesting the optimal meeting time,

[0835] A means of monitoring the status of equipment and optimizing lighting and temperature,

[0836] A means to receive feedback from each user and dynamically update the schedule,

[0837] Means for transmitting proposals and notifications to the user's device

[0838] A system that includes this.

[0839] (Claim 2)

[0840] The system according to claim 1, which enables remote equipment management via a network in order to facilitate effective collaboration with users in a remote environment.

[0841] (Claim 3)

[0842] The system according to claim 1, which generates personalized suggestions to improve the user's work efficiency based on analyzed data.

[0843] "Example 1"

[0844] (Claim 1)

[0845] A means of acquiring and analyzing information data in real time,

[0846] A means to manage the schedules of multiple users and automatically suggest the optimal meeting time,

[0847] A means of monitoring the operating status of the equipment and optimizing lighting and temperature,

[0848] A means of receiving communications from each user and dynamically updating their activity schedule,

[0849] Means for sending proposals and notifications to the user's device,

[0850] A means of securely performing authentication by communicating using an external application program interface,

[0851] A means of dynamically controlling environmental conditions using a human detection device.

[0852] A system that includes this.

[0853] (Claim 2)

[0854] The system according to claim 1, which enables the operation of equipment in a remote location via an information network in order to facilitate effective cooperation with users in a remote environment.

[0855] (Claim 3)

[0856] The system according to claim 1, which generates personalized suggestions to improve the user's activity efficiency based on the analyzed information.

[0857] "Application Example 1"

[0858] (Claim 1)

[0859] A means of acquiring and analyzing video data, audio data, and symbolic data in real time,

[0860] A means to manage the activity schedules of multiple users and automatically suggest the optimal meeting time,

[0861] A means of monitoring the status of equipment and optimizing lighting and temperature,

[0862] A means of receiving feedback from each user and dynamically updating the activity schedule,

[0863] Means for transmitting proposals and notifications to the user's device,

[0864] A means for analyzing the activity status of operating equipment and optimizing the operating schedule,

[0865] A means of detecting equipment malfunctions and notifying the person in charge.

[0866] A system that includes this.

[0867] (Claim 2)

[0868] The system according to claim 1, which enables the management of equipment at a remote location via an information network in order to facilitate effective cooperation with users in a remote environment.

[0869] (Claim 3)

[0870] The system according to claim 1, which generates personalized suggestions to improve the user's work efficiency based on analyzed data and optimizes the maintenance plan for the device.

[0871] "Example 2 of combining an emotion engine"

[0872] (Claim 1)

[0873] A device and means for collecting and analyzing video and audio information in real time,

[0874] Information processing device means for determining the emotional state of individual users,

[0875] A device that manages the user's schedule based on the analysis results and dynamically determines the optimal meeting time,

[0876] A device that monitors environmental settings and optimizes lighting and temperature based on the user's emotional state,

[0877] A device or means for receiving feedback from each user and incorporating it into future adjustments,

[0878] A device that sends proposals and notifications to the user's terminal and accepts the user's approval.

[0879] A system that includes this.

[0880] (Claim 2)

[0881] A system according to claim 1, which enables remote environment management via a network in a remote environment and facilitates effective coordination.

[0882] (Claim 3)

[0883] A system according to claim 1, which provides personalized recommendations to improve the user's work efficiency using generated analysis data.

[0884] "Application example 2 when combining with an emotional engine"

[0885] (Claim 1)

[0886] A means of acquiring and analyzing video and audio data in real time,

[0887] A means to understand the situation of multiple customers and automatically suggest the optimal customer service response,

[0888] A means of monitoring the store environment and optimizing lighting and temperature,

[0889] A means of receiving feedback from each customer and dynamically updating the service,

[0890] Means for transmitting suggestions and notifications to staff devices

[0891] A system that includes this.

[0892] (Claim 2)

[0893] The system according to claim 1, which enables remote environmental management via a network in order to facilitate effective collaboration with customers in the store environment.

[0894] (Claim 3)

[0895] The system according to claim 1, which generates personalized suggestions to improve the customer experience based on analyzed data. [Explanation of symbols]

[0896] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of acquiring and analyzing video data, audio data, and text data in real time, A method for managing the schedules of multiple users and automatically suggesting the optimal meeting time, A means of monitoring the status of equipment and optimizing lighting and temperature, A means to receive feedback from each user and dynamically update the schedule, Means for transmitting proposals and notifications to the user's device A system that includes this.

2. The system according to claim 1, which enables remote equipment management via a network in order to facilitate effective collaboration with users in a remote environment.

3. The system according to claim 1, which generates personalized suggestions to improve the user's work efficiency based on analyzed data.

Images