system

Abstract

We provide the system. [Solution] The present invention relates to a system that centrally stores and manages biometric information such as heart rate, sleep duration, activity level, body temperature, blood pressure, and stress indicators obtained from information terminals and wearable devices, and work information such as working hours, overtime hours, number of tasks, workload, number of meetings, and self-reported stress levels obtained from attendance management systems and task management systems, on a server, and analyzes it multidimensionally and chronologically using generative AI to detect early signs of mental illness with high accuracy, and if an abnormality is found, automatically notifies the individual and their supervisor of specific risk information and recommended countermeasures, thereby realizing early detection of mental illness and prevention of its worsening.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system. Background Art

[0002] Patent Document 1 discloses a persona chatbot control method executed by at least one processor, the method comprising the steps of: receiving a user utterance; adding the user utterance to a prompt including an instruction text associated with a description related to a character of the chatbot; encoding the prompt; and inputting the encoded prompt into a language model to generate a chatbot utterance that responds to the user utterance. Prior Art Documents Patent Documents

[0003] Patent Document 1 Japanese Unexamined Patent Application Publication No. 2022-180282 Summary of the Invention Problems to be Solved by the Invention

[0004] In modern society, mental illnesses such as depression and adjustment disorder are on the rise among working adults who use information terminals and smartphones for their work. These mental illnesses often have few noticeable symptoms in their early stages, making it difficult for the individual and those around them to detect abnormalities early. This leads to serious problems such as worsening of the condition, long-term leave of absence, resignation, and even social losses and decreased productivity. However, conventional methods rely on periodic interviews, self-reporting, and simple stress checks, which make it difficult to objectively and continuously grasp changes in mental state, thus hindering early detection and prompt response. The problem that this invention aims to solve is to collect daily biometric and work information from multiple perspectives and automatically, and through advanced analysis by generative AI, detect early signs of mental illness with high accuracy and in real time. By promptly notifying the individual and their supervisor of specific risk information and countermeasures, the invention aims to achieve early detection of mental illness, prevention of worsening of the condition, and appropriate care. [Means for solving the problem]

[0005] The present invention provides a system comprising a biometric information acquisition unit, a business information acquisition unit, an analysis and notification unit, and a generative AI execution unit. The biometric information acquisition unit periodically and automatically acquires multiple types of biometric information such as heart rate, sleep duration, activity level, body temperature, blood pressure, and stress indicators from information terminals and wearable devices, the business information acquisition unit periodically and automatically acquires multiple types of business-related information such as working hours, overtime hours, number of tasks, work content, workload, break time, and work start and end times from business management systems and attendance management systems, and the generative AI execution unit uses the acquired biometric information and business information history data. This system multidimensionally analyzes the trends in changes in various indicators over a certain period, the frequency of abnormal values, and the correlation between multiple indicators to extract patterns and risk factors that indicate signs of mental illness with high accuracy. Based on the analysis results by the generation AI execution unit, if signs of mental illness are detected, the analysis notification unit automatically sends a detailed warning notification, including the specific indicators where abnormalities were found, their content, the estimated risk level, and recommended countermeasures, to the individual's information terminal and the management screen or email of their supervisor, thereby aiming to achieve early detection of mental illness, prevention of its worsening, and appropriate care. [Brief explanation of the drawing]

[0006] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7]This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Modes for carrying out the invention]

[0007] Hereinafter, an example of an embodiment of the system relating to the technology of this disclosure will be described with reference to the attached drawings.

[0008] First, let's explain the terminology used in the following explanation.

[0009] In the following embodiments, the signed processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Furthermore, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), and APU (Accelerated Processing Unit).

[0010] In the following embodiments, signed RAM (Random Access Memory) is a memory that temporarily stores information and is used as work memory by the processor.

[0011] In the following embodiments, the labeled storage is one or more non-volatile storage devices that store various programs, various parameters, and the like. Examples of non-volatile storage devices include a flash memory (SSD (Solid State Drive)), a magnetic disk (for example, a hard disk), a magnetic tape, or the like.

[0012] In the following embodiments, the labeled communication I / F (Interface) is an interface including a communication processor, an antenna, and the like. The communication I / F manages communication between a plurality of computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), and the like.

[0013] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B". That is, "A and / or B" means that it may be only A, only B, or a combination of A and B. Further, in the present specification, when three or more matters are expressed in association with each other using "and / or", the same concept as that for "A and / or B" is applied.

[0014] [First Embodiment]

[0015] FIG. 1 illustrates an example of the configuration of a data processing system 10 according to the first embodiment.

[0016] As shown in FIG. 1, the data processing system 10 includes a data processing apparatus 12 and a smart device 14. An example of the data processing apparatus 12 is a server.

[0017] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of the "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network), and the like.

[0018] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, camera 42, and communication I / F 44 are also connected to the bus 52.

[0019] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user inputs. The touch panel 38A detects contact of an indicator (for example, a pen, a finger, or the like), thereby receiving a user input from the contact of the indicator. The microphone 38B detects a user's voice, thereby receiving a voice-based user input. A control unit 46A in the processor 46 transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, a specific processing unit 290 acquires the data indicating the user input.

[0020] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0021] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0022] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0023] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0024] Storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the user's emotions using the emotion identification model 59 and perform identification processing using the user's emotions. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotions, including but not limited to these examples. Furthermore, emotion estimation and prediction also include, for example, emotion analysis.

[0025] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output processing program 60 executed on the RAM 48. The smart device 14 also has a data generation model 58 and an emotion identification model 59, similar to the data generation model and emotion identification model 59, and can perform processing similar to that of the specific processing unit 290 using these models. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0026] Furthermore, other devices besides the data processing device 12 may also have the data generation model 58. For example, a server device (e.g., a generation server) may have the data generation model 58. In this case, the data processing device 12 obtains processing results (such as prediction results) using the data generation model 58 by communicating with the server device having the data generation model 58. The data processing device 12 may also be a server device or a terminal device owned by a user (e.g., a mobile phone, robot, home appliance, etc.). Next, an example of processing by the data processing system 10 according to the first embodiment will be described.

[0027] (Example 1)

[0028] The flow of a specific process in Example 1 will now be described. Each part of the system described below is implemented by a data processing device 12 and a smart device 14. The data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0029] (Embodiments for carrying out the present invention)

[0030] Embodiments of the present invention will be described in more specific and detail. The system of the present invention is realized by the cooperative operation of a server and a terminal, and the terminals include smartphones, tablets, laptops, and wearable devices (e.g., smartwatches, activity trackers, sleep trackers, blood pressure monitors, thermometers, pulse oximeters, etc.), and the server consists of a cloud server, an on-premise server, an edge server, etc.

[0031] The biometric information acquisition unit is primarily implemented on the terminal side. For example, an optical heart rate sensor installed in a smartwatch or activity tracker acquires heart rate every minute, and an accelerometer or gyroscope sensor acquires activity data such as steps taken, calories burned, exercise intensity, sitting time, and number of times standing up in real time. Using a sleep tracker or smartphone application, detailed sleep information such as bedtime, wake-up time, sleep duration, proportion of deep sleep, light sleep, and REM sleep, number of awakenings during sleep, snoring detection, and sleep efficiency is acquired daily. Furthermore, biometric information such as body temperature, blood pressure, blood oxygen saturation, respiratory rate, and skin temperature is periodically acquired by linking with thermometers, pulse oximeters, blood pressure monitors, etc. via Bluetooth or Wi-Fi. For example, body temperature can be measured with a thermometer, blood pressure with a blood pressure monitor, and blood oxygen saturation with a pulse oximeter every morning upon waking. Stress indicators can be estimated by heart rate variability analysis, skin electrical response sensors, or voice analysis using a smartphone microphone. Regarding meal content and alcohol intake, users can take pictures of their meals with their smartphone camera, and image recognition AI can estimate the meal content and calories, or record their alcohol intake using a drinking log app. This biometric information is temporarily stored on the device and sent to a server via encrypted communication at regular intervals (e.g., every hour, every day). Before transmission, digital signatures and anonymization can be applied to prevent data tampering and protect privacy.

[0032] The business information acquisition unit is primarily implemented on the server side. For example, it integrates with attendance management systems to automatically acquire daily arrival and departure times, break start and end times, overtime application and approval information, and holiday work information. It integrates with task management and project management systems to acquire the number of assigned tasks, task progress, task priority, task delay status, and task completion rate. It integrates with daily work report systems and self-report stress check systems to acquire work content, self-assessment of workload, self-reported stress levels, job satisfaction, and difficulties encountered in the workplace. It integrates with meeting reservation systems to acquire the number of meetings, meeting duration, meeting type (in-person / online), number of participants, and meeting agenda. It integrates with business travel management systems to acquire business travel frequency, duration, destination, and purpose. It integrates with performance evaluation systems to acquire performance evaluations from superiors and colleagues, feedback content, and award / warning history. This business information is centrally managed on the server as a time-series database for each user. For example, user A's daily records include their arrival time, departure time, overtime hours, number of tasks, number of meetings, and self-reported stress level from June 1st to June 30th, 2024.

[0033] The generative AI execution unit is implemented on the server side. Using the historical data of biometric and business information stored on the server, the generative AI performs multidimensional and time-series analysis. For example, the generative AI calculates the average, standard deviation, maximum, minimum, rate of change, and frequency of outliers for various indicators such as heart rate, sleep duration, activity level, body temperature, blood pressure, stress index, working hours, overtime hours, number of tasks, workload, number of meetings, and self-reported stress level over the past 7, 30, and 90 days. Furthermore, the generative AI automatically models each individual's normal state and past trends. For example, if user B originally tends to have short sleep durations, but their sleep duration has decreased further and their heart rate has increased in the past week, it will be judged as abnormal. The generative AI analyzes the correlation between multiple indicators. For example, if multiple indicators change simultaneously, such as an increase in overtime hours and a decrease in sleep duration, a decrease in activity level and an increase in self-reported stress level, or an increase in the number of meetings and an increase in heart rate, it will be judged as a sign of mental illness. Generative AI can simultaneously detect both short-term, rapid changes (e.g., a 20 percent increase in heart rate and a 2-hour decrease in sleep duration over two days) and long-term, gradual changes (e.g., a gradual increase in overtime hours and a gradual decrease in activity levels over one month). Furthermore, generative AI continuously improves its detection accuracy by learning from past anomaly detection cases and actual cases of mental illness onset as training data. For example, it can learn data patterns from users who have previously experienced mental illness and determine a high risk when similar patterns appear.

[0034] The use of generative AI goes beyond simple thresholding or rule-based anomaly detection; it's characterized by its ability to learn from large amounts of historical data and automatically extract complex patterns and risk factors that indicate signs of mental illness. For example, generative AI can automatically discover complex correlation patterns between multiple indicators, such as combinations of heart rate and activity level, combinations of sleep quality and workload, and increases in the number of tasks and self-reported stress levels, enabling highly accurate detection of signs of mental illness that were previously difficult for humans to notice. Furthermore, generative AI can continuously improve its detection accuracy by incorporating past anomaly detection cases and actual cases of mental illness as training data.

[0035] The analysis notification unit is implemented on the server side. Based on the analysis results from the generation AI execution unit, if signs of mental illness are detected, the analysis notification unit generates a detailed warning notification that includes specific indicators and their contents, estimated risk levels, and recommended countermeasures. For example, it might generate a specific notification such as, "Your average heart rate over the past 7 days has been 15 percent higher than normal, your average sleep time has decreased to 4.5 hours, and your overtime hours have exceeded 10 hours per week, indicating signs of mental illness. We recommend consulting a medical institution or adjusting your workload as soon as possible." The notification content is sent to the individual's information terminal (smartphone, PC, etc.) via push notification or email, and is also automatically sent to the manager's management screen or email. For example, it is possible to provide managers with a dashboard that displays the risk status of multiple subordinates in a list, allowing them to prioritize identifying high-risk employees.

[0036] Furthermore, for example, when an individual receives a notification, they can answer a simple questionnaire or self-declaration form, allowing the generative AI to analyze this self-declared data as well, resulting in a more accurate risk assessment. For instance, if an individual responds with "I've been feeling a lot of stress lately" or "I've been having trouble sleeping," the generative AI will weight this information and incorporate it into the analysis. Moreover, after an individual receives a notification, it's possible to integrate with medical institution reservation systems or counseling services, allowing them to book a consultation with a single click.

[0037] As described above, the system of the present invention primarily implements the biometric information acquisition unit on the terminal side and the business information acquisition unit, generative AI execution unit, and analysis notification unit on the server side through cooperation between the server and the terminal. By performing multidimensional and individually optimized analysis using generative AI, it detects signs of mental illness with high accuracy and in real time, and automatically provides specific and detailed notifications to the individual and their supervisor, thereby aiming to achieve early detection of mental illness, prevention of worsening, and appropriate care. For example, when the individual receives a notification, it is possible to add functions such as requesting adjustments to the workload from their supervisor based on the notification content, automatically displaying advice on stress coping methods, visualizing the current situation by comparing it with past data, and sharing notifications with family and trusted colleagues. Furthermore, it is possible to analyze risk trends across the entire organization and utilize this information to propose business improvements at the organizational level and to plan health management measures. For example, if overtime hours and stress indicators are rising simultaneously in a specific department, it is possible to automatically send alerts to organizational managers recommending a review of work assignments or taking leave. Furthermore, for example, generative AI can use natural language generation capabilities to automatically generate easy-to-understand explanations and advice for the individual and their supervisor, and notify them in simple language that avoids technical jargon. For instance, when the individual receives a notification, it can automatically generate graphs and charts that visualize the current situation by comparing it to their past data, allowing them to visually understand their risk situation.

[0038] Thus, embodiments of the present invention include cooperative operation between servers and terminals, diverse collaboration with various sensors and business systems, multidimensional and individually optimized analysis by generative AI, and various notification and advice functions for the individual, supervisors, and organizational administrators, thereby realizing multifaceted and highly accurate early detection and response to signs of mental illness.

[0039] (System Configuration)

[0040] The system according to this embodiment comprises a biometric information acquisition unit, a business information acquisition unit, a data storage unit, a generation AI execution unit, an analysis notification unit, a user interface unit, and a security management unit. The biometric information acquisition unit automatically acquires various biometric information in real time or periodically using various sensors and external connectivity functions installed in information terminals and wearable devices such as smartphones, tablets, laptops, smartwatches, activity trackers, sleep trackers, blood pressure monitors, thermometers, and pulse oximeters, including heart rate, sleep duration, sleep quality, activity level, steps taken, calories burned, sitting time, number of times standing up, body temperature, blood pressure, blood oxygen saturation, respiratory rate, skin temperature, stress index, dietary content, alcohol intake, and exercise history. For example, this includes specific cases such as acquiring heart rate every minute using the optical heart rate sensor of a smartwatch, measuring steps and exercise intensity using an accelerometer and gyroscope, acquiring the proportion of deep sleep, light sleep, and REM sleep and the number of awakenings during sleep using a sleep tracker, acquiring daily body temperature and blood pressure by connecting with a thermometer and blood pressure monitor via Bluetooth, measuring blood oxygen saturation with a pulse oximeter, taking pictures of meals with a smartphone camera and estimating the contents and calories of the meal with image recognition AI, and recording alcohol intake with an alcohol recording app. The Business Information Acquisition Department integrates with various business-related systems such as attendance management systems, task management systems, project management systems, daily work report systems, self-reported stress check systems, meeting reservation systems, business trip management systems, and performance evaluation systems to automatically acquire diverse business information, including arrival time, departure time, break start and end times, overtime application and approval information, holiday work information, number of assigned tasks, task progress, task priority, task delay status, task completion rate, work content, self-assessment of workload, self-reported stress levels, job satisfaction, difficulties encountered in the work, number of meetings, meeting duration, meeting type, number of meeting participants, meeting agenda, frequency of business trips, duration of business trips, destinations, purpose of business trips, performance evaluations from superiors and colleagues, feedback content, and award / warning history.For example, this includes obtaining daily arrival and departure times and overtime hours from the attendance management system, the number of assigned tasks and their progress from the task management system, self-reported work content and workload from the daily work report system, the number and duration of meetings from the meeting reservation system, the frequency and duration of business trips from the business trip management system, and work evaluations and feedback from the work evaluation system. The data storage unit centrally stores and manages the acquired biometric and work information as a time-series database for each user, date, and indicator. For example, it records user A's heart rate, sleep duration, activity level, arrival time, overtime hours, number of tasks, number of meetings, self-reported stress level, etc., on a daily basis from June 1st to June 30th, 2024, safely and efficiently accumulating data for several months to several years. The generative AI execution unit uses the historical data of biometric and business information stored in the data storage unit to perform multidimensional and time-series analysis using generative AI. For example, it calculates the average value, standard deviation, maximum value, minimum value, rate of change, and frequency of outliers for each indicator such as heart rate, sleep duration, activity level, body temperature, blood pressure, stress index, working hours, overtime hours, number of tasks, workload, number of meetings, and self-reported stress level over the past 7, 30, and 90 days. It automatically models each individual's normal state and past trends, simultaneously detecting correlations between multiple indicators, short-term rapid changes, and long-term gradual changes, and automatically extracts complex patterns and risk factors that indicate signs of mental illness. For example, the system might identify a potential sign of mental illness if a heart rate is more than 10 percent higher than normal for three or more consecutive days, sleep duration is less than 5 hours for two or more consecutive days, and overtime hours exceed an average of 3 hours per week. It might also identify a high risk if a decrease in activity level and an increase in self-reported stress levels occur simultaneously. Furthermore, it might learn from past abnormality detection cases and actual cases of mental illness as training data, and identify a high risk if a similar pattern appears.When activating the generative AI execution unit, prompts such as "Analyze the following user biometric and work history data, extract abnormal patterns and risk factors indicating signs of mental illness, and output the details and recommended countermeasures in natural language if an abnormality is detected," "Analyze time-series data of multiple indicators such as heart rate, sleep duration, activity level, working hours, overtime hours, number of tasks, workload, and stress indicators to detect signs of mental illness with high accuracy, by analyzing short-term rapid changes, long-term gradual changes, and correlations between multiple indicators," and "Use past abnormality detection cases and mental illness onset cases as training data, determine that a similar pattern is high risk, and output the reason and recommended countermeasures" can be given to the generative AI. The analysis notification unit, based on the analysis results from the generation AI execution unit, generates a detailed warning notification if signs of mental illness are detected. This notification includes specific indicators and their content, estimated risk levels, and recommended countermeasures. It is sent to the employee's information terminal via push notification or email, and is also automatically sent to the supervisor's management screen and email. For example, it generates a specific notification such as, "Your average heart rate over the past 7 days is 15 percent higher than normal, your average sleep time has decreased to 4.5 hours, and your overtime hours exceed 10 hours per week. These are signs of mental illness. We recommend consulting a medical institution or adjusting your workload as soon as possible." Supervisors are also provided with a dashboard that displays the risk status of multiple subordinates in a list, allowing them to prioritize identifying high-risk employees. The user interface section provides screens and applications that allow the user and their supervisor to easily view notification content and analysis results. For example, when the user receives a notification, it automatically generates graphs and charts that visualize the current situation by comparing it with their past data, allowing them to visually understand the risk situation. It also provides functions such as requesting adjustments to workload from supervisors based on the notification content, automatically displaying advice on stress management, answering self-report forms, and sharing notifications with family and trusted colleagues.The Security Management Department implements security measures such as data encryption, access control, tamper prevention, anonymization, and digital signatures at each stage of acquisition, storage, analysis, and notification. For example, it uses encrypted communication when transmitting biometric and business information, anonymizes personally identifiable information when storing data on the server, controls access so that only users with access rights can access the data, and prevents tampering with or leakage of notification content. As described above, the system according to this embodiment enables each department to work together to acquire, store, analyze, and notify diverse biometric and business information with high accuracy and security, and to detect early signs of mental illness using generative AI, thereby providing specific and detailed risk information and countermeasures to the individual and their supervisor.

[0041] (Implementation Steps)

[0042] Step 1: Automatic acquisition of biometric information Using information terminals and wearable devices such as smartphones, smartwatches, activity trackers, sleep trackers, blood pressure monitors, thermometers, and pulse oximeters, a variety of biometric information such as heart rate, sleep duration, sleep quality, activity level, steps, calories burned, sitting time, number of times standing up, body temperature, blood pressure, blood oxygen saturation, respiratory rate, skin temperature, stress indicators, diet, alcohol intake, and exercise history can be automatically acquired in real time or on a regular basis. For example, this could involve acquiring heart rate every minute using the optical heart rate sensor of a smartwatch, measuring steps and exercise intensity with an accelerometer, acquiring the ratio of deep sleep, light sleep, and REM sleep and the number of awakenings during sleep with a sleep tracker, acquiring daily body temperature and blood pressure by connecting with a thermometer or blood pressure monitor via Bluetooth, measuring blood oxygen saturation with a pulse oximeter, taking pictures of meals with a smartphone camera and estimating the meal content and calories with image recognition AI, and recording alcohol intake with a drinking record app.

[0043] Step 2: Automatic acquisition of business information The system integrates with various business-related systems such as attendance management systems, task management systems, project management systems, daily work report systems, self-reported stress check systems, meeting reservation systems, business trip management systems, and performance evaluation systems to automatically acquire diverse business information such as arrival time, departure time, break start and end times, overtime application and approval information, holiday work information, number of assigned tasks, task progress, task priority, task delay status, task completion rate, work content, self-assessment of workload, self-reported stress level, job satisfaction, work difficulties, number of meetings, meeting duration, meeting type, number of participants, meeting agenda, frequency of business trips, duration of business trips, destinations, purpose of business trips, performance evaluations from superiors and colleagues, feedback content, and award / warning history. For example, this includes specific cases such as obtaining daily arrival and departure times and overtime hours from an attendance management system, obtaining the number of assigned tasks and their progress from a task management system, obtaining self-reported work content and workload from a daily work report system, obtaining the number and duration of meetings from a meeting reservation system, obtaining the frequency and duration of business trips from a business trip management system, and obtaining work evaluations and feedback from a performance evaluation system.

[0044] Step 3: Data storage and time-series management The acquired biometric and work information is centrally stored and managed on the server as a time-series database, organized by user, date, and indicator. For example, user A's heart rate, sleep duration, activity level, arrival time, overtime hours, number of tasks, number of meetings, and self-reported stress level from June 1st to June 30th, 2024, are recorded on a daily basis, ensuring the secure and efficient accumulation of data spanning several months to several years. Data transmission is encrypted and anonymized to ensure privacy and security.

[0045] Step 4: Multidimensional analysis using generative AI Using the historical data of biometric and work information stored in the data storage unit, the generative AI performs multidimensional and time-series analysis. For example, for each indicator such as heart rate, sleep duration, activity level, body temperature, blood pressure, stress indicators, working hours, overtime hours, number of tasks, workload, number of meetings, and self-reported stress levels, the AI ​​calculates the average value, standard deviation, maximum value, minimum value, rate of change, and frequency of outliers over the past 7, 30, and 90 days. It automatically models each individual's normal state and past trends, simultaneously detecting correlations between multiple indicators, short-term rapid changes, and long-term gradual changes, and automatically extracts complex patterns and risk factors that indicate signs of mental illness. Specific examples of prompts to be fed into the generating AI include: "Analyze the following user biometric and work history data, extract abnormal patterns and risk factors indicating signs of mental illness, and output the details and recommended countermeasures in natural language if an abnormality is detected," "Analyze time-series data of multiple indicators such as heart rate, sleep duration, activity level, working hours, overtime hours, number of tasks, workload, and stress indicators to detect signs of mental illness with high accuracy by analyzing short-term rapid changes, long-term gradual changes, and correlations between multiple indicators," and "Use past abnormality detection cases and mental illness onset cases as training data, determine that a similar pattern is high risk when it appears, and output the reason and recommended countermeasures."

[0046] Step 5: Notification of analysis results and feedback Based on the analysis results from the generative AI execution unit, the analysis notification unit generates a detailed warning notification that includes specific indicators where anomalies were found, their content, estimated risk levels, and recommended countermeasures. This notification is sent to the individual's information terminal via push notification or email, and is also automatically sent to the supervisor's management screen and email. For example, it might generate a specific notification such as, "Your average heart rate over the past 7 days has been 15 percent higher than normal, your average sleep time has decreased to 4.5 hours, and your overtime hours have exceeded 10 hours per week, indicating a potential mental illness. We recommend seeking medical attention or adjusting your workload as soon as possible." When the individual receives the notification, they can answer a self-declaration form, which allows the generative AI to analyze the self-declared data as well, enabling a more accurate risk assessment.

[0047] Step 6: Visualization and action support through user interface The user interface section provides screens and applications that allow the user and their supervisor to easily view notification content and analysis results. For example, when the user receives a notification, it automatically generates graphs and charts that visualize the current situation by comparing it with their past data, allowing them to visually understand the risk situation. It also provides functions such as requesting adjustments to workload from supervisors based on the notification content, automatically displaying advice on stress management, answering self-report forms, and sharing notifications with family and trusted colleagues.

[0048] Step 7: Security Management and Privacy Protection The Security Management Department implements security measures such as data encryption, access control, tamper prevention, anonymization, and digital signatures at each stage of acquisition, storage, analysis, and notification. For example, it uses encrypted communication when transmitting biometric information and business information, anonymizes personally identifiable information when storing data on the server, controls access so that only users with access rights can access the data, and prevents tampering with or leakage of notification content.

[0049] (Specific use cases)

[0050] For example, consider applying the system of the present invention to working adults who routinely use information terminals and wearable devices for their daily work. The biometric information acquisition unit acquires heart rate every minute using the heart rate sensor of a smartwatch, measures steps and activity levels using an accelerometer, and acquires daily data such as bedtime, wake-up time, sleep duration, the proportion of deep sleep, light sleep, and REM sleep, and the number of awakenings during sleep using a sleep tracker. Furthermore, it acquires body temperature and blood pressure every morning in conjunction with a thermometer and blood pressure monitor, measures blood oxygen saturation with a pulse oximeter, takes pictures of meals with a smartphone camera to estimate meal contents and calories using image recognition AI, and records alcohol intake with a drinking record app. The Business Information Acquisition Department acquires attendance time, departure time, overtime application / approval information, and holiday work information from the attendance management system; the number of assigned tasks, task progress, task priority, task delay status, and task completion rate from the task management system; work content, self-assessment of workload, self-reported stress levels, job satisfaction, and work-related difficulties from the daily work report system; the number of meetings, meeting duration, meeting type, number of participants, and meeting agenda from the meeting reservation system; the frequency of business trips, duration, destination, and purpose of business trips from the business trip management system; and work evaluations from superiors and colleagues, feedback content, and award / warning history from the work evaluation system. The Data Storage Department centrally stores and manages this biometric and business information on a server as a time-series database, organized by user, date, and indicator.

[0051] The generative AI execution unit calculates the average, standard deviation, maximum, minimum, rate of change, and frequency of outliers for various indicators such as heart rate, sleep duration, activity level, body temperature, blood pressure, stress index, working hours, overtime hours, number of tasks, workload, number of meetings, and self-reported stress level over the past 7, 30, and 90 days, and automatically models each individual's normal state and past trends. For example, it determines that a person may be at risk of mental illness if their heart rate is 10 percent or more above normal for three consecutive days, their sleep duration is less than 5 hours for two consecutive days, and their overtime hours exceed an average of 3 hours per week. Furthermore, it determines a high risk if a decrease in activity level and an increase in self-reported stress level occur simultaneously. The generative AI simultaneously analyzes short-term rapid changes, long-term gradual changes, and correlations between multiple indicators, and automatically extracts complex patterns and risk factors that indicate signs of mental illness. For example, it learns from past anomaly detection cases and actual cases of mental illness as training data, and determines a high risk when similar patterns appear. Specific examples of prompts to be fed into a generative AI include: "Analyze the following user biometric and work history data, extract abnormal patterns and risk factors indicating signs of mental illness, and output the details and recommended countermeasures in natural language if an abnormality is detected," "Analyze time-series data of multiple indicators such as heart rate, sleep duration, activity level, working hours, overtime hours, number of tasks, workload, and stress indicators to detect signs of mental illness with high accuracy by analyzing short-term rapid changes, long-term gradual changes, and correlations between multiple indicators," and "Use past abnormality detection cases and mental illness onset cases as training data, determine that a similar pattern is high risk when it appears, and output the reason and recommended countermeasures."

[0052] The analysis notification unit generates a detailed warning notification based on the analysis results from the generation AI execution unit. This notification includes specific indicators where anomalies were found, their content, estimated risk levels, and recommended countermeasures. This notification is sent to the user's information terminal via push notification or email, and is also automatically sent to the supervisor's management screen and email. For example, it might generate a specific notification such as, "Your average heart rate over the past 7 days has been 15 percent higher than normal, your average sleep time has decreased to 4.5 hours, and your overtime hours have exceeded 10 hours per week. These are signs of a potential mental illness. We recommend seeking medical attention or adjusting your workload as soon as possible." When the user receives the notification, they can answer a self-declaration form, which allows the generation AI to analyze the self-declared data as well, enabling a more accurate risk assessment. The User Interface Department provides screens and applications that allow users and their supervisors to easily view notification content and analysis results. For example, when a user receives a notification, it automatically generates graphs and charts that visualize the current situation by comparing it with their past data, allowing them to visually grasp the risk situation. It also provides functions such as requesting adjustments to workload from supervisors based on the notification content, automatically displaying advice on stress management methods, responding to self-report forms, and sharing notifications with family and trusted colleagues. The Security Management Department implements security measures such as data encryption, access control, tamper prevention, anonymization, and electronic signatures at each stage of acquisition, storage, analysis, and notification. For example, it uses encrypted communication when transmitting biometric information and business information, anonymizes personally identifiable information when storing data on the server, controls access so that only users with access rights can access the data, and prevents tampering with or leakage of notification content. Thus, the system of the present invention integrates the acquisition and storage of daily biometric and business information from multiple perspectives, multidimensional and time-series analysis by generative AI, rapid and detailed notification when anomalies are detected, visualization and action support through a user interface, and privacy protection through security management. This enables early detection of signs of mental illness and provides specific and effective risk information and countermeasures to the individual and their supervisor.

[0053] (Application Example 1) The flow of a specific process in Application Example 1 will be explained below. Each part of the system described below is implemented by a data processing device 12 and a smart device 14. The data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0054] (Embodiments for carrying out the present invention)

[0055] Embodiments of the present invention will be described in more specific and detail. The system of the present invention can be configured to include a care recipient information acquisition unit, a care worker information acquisition unit, a data storage unit, a generation-based AI analysis and notification unit, a user interface unit, a security management unit, an external linkage unit, and an action support unit.

[0056] The care recipient information acquisition unit works in conjunction with wearable devices worn by the care recipient, various sensors installed in the care facility, smartphones, tablets, cameras, microphones, medication management devices, excretion sensors, bathroom sensors, door sensors, GPS terminals, meal recording apps, etc., to collect information such as heart rate, blood pressure, body temperature, respiratory rate, blood oxygen saturation, sleep duration, sleep quality (proportion of deep sleep, light sleep, and REM sleep, number of awakenings), activity level (steps, exercise intensity, sitting time, number of times standing up), food intake (estimated by AI from photos and records of meals), meal content (nutritional balance and calories), fluid intake, medication status (from medication management devices and recording apps), and fall detection. The system automatically acquires a variety of biometric and lifestyle information in real time or on a regular basis, including data from acceleration sensors and floor sensors, facial expression analysis (AI determines smiles, blank expressions, anxiety, anger, etc. from camera images), voice emotion analysis (AI estimates calmness, sadness, irritation, etc. from conversations and monologues), excretion status (excretion sensors and recording apps), bathing status (bathroom sensors and recordings), frequency of going out (GPS and door sensors), frequency of social interaction (visitation records and conversation records), skin condition (detection of pressure ulcers and rashes through image analysis), weight, BMI, walking speed, grip strength, chewing frequency, swallowing status, abnormalities in cough and breathing sounds, speech volume, detection of monologue and wandering, and self-reporting via an emotion diary app. For example, the system can automatically record daily body temperature, blood pressure, and blood oxygen saturation measurements; track sleep using a sleep tracker at night; take photos during meals and use AI to estimate meal content; automatically record fluid intake; automatically record medication intake via button presses or medication management devices; automatically trigger an alarm in case of falls; monitor emotions during the day using a facial camera and voice microphone; record excretion using an excretion sensor; record bathing using a bathroom sensor; detect going out using GPS and door sensors; understand the frequency of social interaction through visitor and conversation records; detect pressure ulcers and rashes through AI analysis of skin images; measure muscle strength using scales and grip strength meters; understand eating habits using chewing and swallowing sensors; detect respiratory abnormalities through AI analysis of coughs and breath sounds; automatically detect speech volume, talking to oneself, and wandering; and self-reporting through an emotion diary app.

[0057] The care worker information acquisition unit works in conjunction with the shift management system, task management system, care record system, self-reporting app, wearable devices, break management system, survey system, performance evaluation system, feedback record system, etc., to automatically acquire work information and stress information such as working hours (shift management system and time clock devices), number of night shifts, number of tasks (care record system and task management app), delay in care record input (timing of input in the record system), stress indicators (wearable devices and self-reporting app), break acquisition status (break time clocking and sensors), self-reported fatigue level (app input), mental state (self-report and voice analysis), job satisfaction (survey), self-reported stress level (periodic survey), performance evaluation (evaluation from supervisors and colleagues), feedback content (free-form text and evaluation comments), intention to leave, sleep quality, heart rate variability, reports of poor health, frequency of workplace communication, distance traveled during work, amount of speech during work, facial expression analysis during work, and accident / near-miss reports during work. For example, specific examples include obtaining work records from shift management systems, obtaining care records from task management apps, obtaining heart rate variability and stress levels from wearable devices, detecting input delays in recording systems, inputting "fatigue" and "decreased motivation" in self-reporting apps, understanding break times through break clocking and sensors, regularly obtaining job satisfaction and stress levels through questionnaires, collecting work evaluations and feedback from supervisors and colleagues, obtaining questionnaires on intention to leave, understanding sleep quality with sleep trackers, estimating stress through heart rate variability analysis, managing health with health reporting apps, understanding conversation frequency with workplace communication apps, automatically measuring distance traveled and speech volume during work, understanding emotional changes through facial expression analysis during work, and collecting risk information with accident and near-miss reporting systems.

[0058] The data storage unit centrally stores and manages all information obtained from care recipients and caregivers as a time-series database on the server, organized by user, date, and indicator. For example, it stores data for care recipient A from June 1st to June 30th, 2024, including heart rate, blood pressure, body temperature, respiratory rate, blood oxygen saturation, sleep duration, sleep quality, activity level, food intake, food content, fluid intake, medication status, fall records, facial expression analysis results, voice emotion analysis results, excretion status, bathing status, frequency of going out, frequency of social interaction, skin condition, weight, BMI, walking speed, grip strength, chewing frequency, swallowing status, abnormalities in cough and breath sounds, speech volume, self-talk / wandering records, and emotions. The diary app records self-reported information on a daily basis, including caregiver B's working hours, number of night shifts, number of tasks, delays in care record entry, stress indicators, break status, self-reported fatigue level, mental state, job satisfaction, job evaluation, feedback content, intention to leave, sleep quality, heart rate variability, reports of poor health, frequency of workplace communication, distance traveled during work, amount of speech during work, facial expression analysis during work, and accident / near-miss reports. Data transmission is encrypted and anonymized to ensure privacy and security.

[0059] The generational AI analysis and notification unit uses the historical data of the care recipient and caregiver stored in the data storage unit to analyze and notify the following: heart rate, blood pressure, body temperature, respiratory rate, blood oxygen saturation, sleep, activity level, food intake, fluid intake, medication status, falls, facial expressions, vocal emotions, excretion, bathing, going out, social interaction, skin condition, weight, BMI, walking speed, grip strength, chewing frequency, swallowing status, cough and breathing sounds, speech volume, talking to oneself / wandering, emotional diary, working hours, number of night shifts, number of tasks, delay in care record entry, stress indicators, and rest. For various indicators such as acquisition status, self-reported fatigue level, mental state, job satisfaction, job evaluation, feedback content, intention to leave, sleep quality, heart rate variability, reports of poor health, frequency of workplace communication, distance traveled during work, amount of speech during work, facial expression analysis during work, and accident / near-miss reports, the system calculates the average value, standard deviation, maximum value, minimum value, rate of change, and frequency of outliers over the past 7, 30, and 90 days, automatically modeling each individual's normal state and past trends. For example, if a care recipient's heart rate, blood pressure, body temperature, respiratory rate, and blood oxygen saturation rise above normal levels, they experience fewer than 5 hours of sleep for several consecutive days, their activity level decreases, their food and fluid intake decreases, they forget to take their medication multiple times, falls are detected, anxiety and irritability are frequently detected from their facial expressions and voice, they experience bowel and bladder abnormalities, their bathing frequency decreases, their outings and social interactions decrease, their skin condition worsens, they lose weight, their walking speed decreases, their grip strength decreases, they have difficulty chewing and swallowing, their cough and breathing sounds are abnormal, their speech decreases, their self-talk and wandering increase, and they frequently report negative things in their emotional diary, these are judged to be signs of serious physical and mental risks such as dehydration, infection, progression of dementia, depression, risk of falls, aspiration pneumonia, malnutrition, and pressure ulcer risk. For care workers, if they experience consecutive days of working more than 12 hours, frequent night shifts, a large number of tasks, high stress indicators, multiple delays in care record keeping, insufficient breaks, worsening self-reported fatigue and mental state, decreased job satisfaction, negative trends in job evaluations and feedback, increased intention to leave, decreased sleep quality, decreased heart rate variability, increased reports of poor health, decreased frequency of workplace communication, decreased distance traveled and volume of speech during work, increased negative emotions in facial expression analysis during work, and increased accident and near-miss reports, they are judged to be at high risk of burnout, care errors, and turnover.Generative AI learns from short-term rapid changes, long-term gradual changes, correlations between multiple indicators, past anomaly detection cases, and actual cases of health deterioration and caregiving accidents as training data, and can determine a high risk when similar patterns appear.

[0060] The generation-based AI analysis and notification unit automatically sends warning notifications to the individual, family, and administrator if multiple abnormalities or risks are detected, indicating a serious care risk or a premonition of mental or physical abnormalities. Notification content may include: "Continued abnormal vital signs of the person being cared for," "Continued sleep deprivation," "Significantly reduced activity level," "Frequent days of insufficient food or fluid intake," "Multiple instances of missed medication," "Detected falls," "Continued emotional abnormalities," "Continued excretory abnormalities," "Decreased frequency of bathing," "Decreased outings and social interaction," "Worsening skin condition," "Weight loss," "Decreased walking speed," "Decreased grip strength," "Difficulty chewing or swallowing," "Cough or abnormal breathing sounds," "Decreased speech," "Increased self-talk and wandering," "Increased negative reporting in emotional diaries," "Continued long working hours for caregivers," "Frequent night shifts," "Continued high stress levels," "Multiple instances of delays in care record keeping," "Insufficient rest," and "Self-reported fatigue." The system includes specific risk items such as "deterioration of physical and mental state," "decreased job satisfaction," "negative trend in work evaluations and feedback," "increased intention to leave," "decreased sleep quality," "decreased heart rate variability," "increased reports of poor health," "decreased frequency of workplace communication," "decreased distance traveled and amount of speech during work," "increased negative emotions in facial expression analysis during work," and "increased accident and near-miss reports," along with recommended countermeasures (such as visiting a medical institution, reviewing care plans, recommending rest for caregivers, contacting family members, consulting with professionals, nutritional guidance, rehabilitation, environmental adjustments, promoting workplace communication, reviewing work assignments, mental health care, and measures to prevent employee turnover). For example, it can generate specific notifications such as, "Your average heart rate over the past 7 days has been 15 percent higher than normal, your average sleep time has decreased to 4.5 hours, two falls have been detected, and your food and fluid intake has been insufficient for 3 days, increasing your risk of dehydration, infection, and falls. We recommend that you visit a medical institution and review your care plan as soon as possible."

[0061] The user interface provides screens and applications that allow the individual, family, and administrator to easily view notification content and analysis results. For example, when the individual or family receives a notification, it automatically generates graphs and charts that visualize the current situation by comparing it with past data, allowing them to visually understand the risk situation. It also includes functions to request adjustments to the care plan based on the notification content, functions to automatically display stress coping methods and lifestyle improvement advice, functions to answer self-declaration forms, functions to share notifications with family and trusted stakeholders, and a dashboard for administrators that displays the risk status of multiple care recipients and caregivers in a list, allowing them to prioritize identifying and responding to high-risk individuals.

[0062] The Security Management Department implements security measures such as data encryption, access control, tamper prevention, anonymization, and digital signatures at each stage of acquisition, storage, analysis, and notification. For example, it uses encrypted communication when transmitting biometric information and business information, anonymizes personally identifiable information when storing data on the server, controls access so that only users with access rights can access the data, and prevents tampering with or leakage of notification content.

[0063] The External Collaboration Department can connect with medical institutions' electronic medical record systems, rehabilitation systems, nutrition guidance systems, family communication systems, and administrative service collaboration systems via APIs, etc., and can automate information sharing with external experts and families, collaboration with medical and welfare services, and reporting to the government as needed.

[0064] Based on the notification content, the Action Support Department can perform actions such as automatically proposing care plans, automatically generating rehabilitation menus, automatically presenting nutritional guidance, automatically sending medication reminders, scheduling consultations with family and professionals, proposing events to promote workplace communication, automatically providing information on mental health care programs, and automatically proposing measures to prevent employee turnover.

[0065] Thus, embodiments of the present invention acquire, store, analyze, and notify diverse biometric, lifestyle, and work information of care recipients and caregivers in an extremely multifaceted and highly accurate manner. They utilize generative AI to detect early signs of mental illness, physical illness, and serious care risks, and provide specific and detailed risk information and countermeasures to the individual, family, administrators, and external experts. This significantly improves safety and the quality of care in care settings, while comprehensively realizing collaboration and action support with various stakeholders such as medical professionals, welfare workers, government officials, and family members.

[0066] (System Configuration)

[0067] The system according to this embodiment includes a care recipient information acquisition unit, a care worker information acquisition unit, a data storage unit, a generation-based AI analysis and notification unit, a user interface unit, a security management unit, an external linkage unit, and an action support unit. The care recipient information acquisition unit automatically acquires diverse biometric and lifestyle information in real time or periodically by linking with wearable devices worn by the care recipient, various sensors installed in the care facility, smartphones, tablets, cameras, microphones, medication management devices, excretion sensors, bathroom sensors, door sensors, GPS terminals, meal recording apps, etc., including heart rate, blood pressure, body temperature, respiratory rate, blood oxygen saturation, sleep duration, sleep quality, activity level, steps taken, exercise intensity, sitting time, number of times standing up, food intake, food content, fluid intake, medication status, fall detection, facial expression analysis, voice emotion analysis, excretion status, bathing status, frequency of going out, frequency of social interaction, skin condition, weight, BMI, walking speed, grip strength, number of chews, swallowing status, abnormalities in cough and breath sounds, speech volume, self-talk / wandering detection, and self-reporting via an emotion diary app. For example, the system automatically records daily body temperature, blood pressure, and blood oxygen saturation; tracks sleep using a sleep tracker at night; takes photos during meals and uses AI to estimate meal content; automatically records fluid intake; automatically records medication intake via button presses or medication management devices; automatically triggers an alarm in case of falls; monitors emotions during the day using a facial camera and voice microphone; records excretion using an excretion sensor; records bathing using a bathroom sensor; detects going out using GPS and door sensors; tracks the frequency of social interaction through visitor and conversation records; detects pressure ulcers and rashes through AI analysis of skin images; measures muscle strength using scales and grip strength meters; monitors eating habits using chewing and swallowing sensors; detects respiratory abnormalities through AI analysis of coughs and breath sounds; automatically detects speech volume, talking to oneself, and wandering; and allows self-reporting through an emotion diary app.The care worker information acquisition unit works in conjunction with shift management systems, task management systems, care record systems, self-reporting apps, wearable devices, break management systems, survey systems, performance evaluation systems, feedback record systems, etc., to automatically acquire work information and stress information such as working hours, number of night shifts, number of tasks, delays in care record input, stress indicators, break acquisition status, self-reported fatigue level, mental state, job satisfaction, self-reported stress level, performance evaluation, feedback content, intention to leave, sleep quality, heart rate variability, reports of poor health, frequency of workplace communication, distance traveled during work, amount of speech during work, facial expression analysis during work, and accident / near-miss reports. For example, specific examples include obtaining work records from shift management systems, obtaining care records from task management apps, obtaining heart rate variability and stress levels from wearable devices, detecting input delays in recording systems, inputting "fatigue" and "decreased motivation" in self-reporting apps, understanding break times through break clocking and sensors, regularly obtaining job satisfaction and stress levels through questionnaires, collecting work evaluations and feedback from supervisors and colleagues, obtaining questionnaires on intention to leave, understanding sleep quality with sleep trackers, estimating stress through heart rate variability analysis, managing health with health reporting apps, understanding conversation frequency with workplace communication apps, automatically measuring distance traveled and speech volume during work, understanding emotional changes through facial expression analysis during work, and collecting risk information with accident and near-miss reporting systems.The data storage unit centrally stores and manages all information obtained from care recipients and caregivers as a time-series database on the server, organized by user, date, and indicator. For example, it stores data for care recipient A from June 1st to June 30th, 2024, including heart rate, blood pressure, body temperature, respiratory rate, blood oxygen saturation, sleep duration, sleep quality, activity level, food intake, food content, fluid intake, medication status, fall records, facial expression analysis results, voice emotion analysis results, excretion status, bathing status, frequency of going out, frequency of social interaction, skin condition, weight, BMI, and walking distance. The system records daily data such as speed, grip strength, chewing frequency, swallowing status, abnormalities in cough and breathing sounds, speech volume, self-talk / wandering records, and self-reported emotional diary entries. It also records daily data such as caregiver B's working hours, number of night shifts, number of tasks, delays in care record entry, stress indicators, break status, self-reported fatigue level, mental state, job satisfaction, job evaluation, feedback content, intention to leave, sleep quality, heart rate variability, reports of poor health, frequency of workplace communication, distance traveled during work, speech volume during work, facial expression analysis during work, and accident / near-miss reports. Data transmission is encrypted and anonymized to ensure privacy and security. The generational AI analysis and notification unit uses the historical data of care recipients and caregivers stored in the data storage unit to analyze heart rate, blood pressure, body temperature, respiratory rate, blood oxygen saturation, sleep, activity level, food intake, fluid intake, medication status, falls, facial expressions, vocal emotions, excretion, bathing, going out, social interaction, skin condition, weight, BMI, walking speed, grip strength, chewing frequency, swallowing status, cough and breathing sounds, speech volume, talking to oneself / wandering, emotional diary, working hours, number of night shifts, number of tasks, delay in care record entry, stress indicators, break acquisition status, self-reported fatigue level, mental state, job satisfaction, job evaluation, feedback content, intention to leave, and sleep. For each indicator, such as quality, heart rate variability, reports of poor health, frequency of workplace communication, distance traveled during work, amount of speech during work, facial expression analysis during work, and accident / near-miss reports, the system calculates the average value, standard deviation, maximum value, minimum value, rate of change, and frequency of outliers over the past 7, 30, and 90 days. It automatically models each individual's normal state and past trends, learning from short-term rapid changes, long-term gradual changes, correlations between multiple indicators, past anomaly detection cases, and actual cases of health deterioration / caregiving accidents as training data. When similar patterns appear, it determines a high risk.For example, if a care recipient's heart rate, blood pressure, body temperature, respiratory rate, and blood oxygen saturation exceed the standard values, they experience consecutive days of sleep durations of less than 5 hours, their activity level decreases, their food and fluid intake decreases, they forget to take their medication multiple times, falls are detected, anxiety and irritability are frequently detected from their facial expressions and voice, they experience bowel and bladder abnormalities, reduced frequency of bathing, decreased outings and social interaction, worsening skin condition, weight loss, decreased walking speed, decreased grip strength, chewing and swallowing difficulties, abnormal coughs and breathing sounds, decreased speech volume, increased talking to themselves and wandering, and many negative reports in their emotional diary, these are judged to be signs of serious physical and mental risks such as dehydration, infection, progression of dementia, depression, risk of falls, aspiration pneumonia, malnutrition, and pressure ulcer risk. Specific examples of prompts to be fed into the generating AI include: "Analyze the following historical data of biometric information, lifestyle information, and work information of the care recipient and caregiver, extract signs of physical and mental risks and serious care risks, and output the details and recommended countermeasures in natural language if an abnormality is detected," "Analyze time-series data such as heart rate, blood pressure, body temperature, respiratory rate, blood oxygen saturation, sleep, activity level, food intake, fluid intake, medication status, falls, facial expressions, voice emotion, working hours, number of tasks, stress indicators, and recording delays to detect serious risks with high accuracy by analyzing short-term rapid changes, long-term gradual changes, and correlations between multiple indicators," and "Use past care accidents and health deterioration cases as training data, determine that a similar pattern is high risk when it appears, and output the reason and recommended countermeasures." The user interface provides screens and applications that allow the individual, family, and administrator to easily view notification content and analysis results. For example, when the individual or family receives a notification, it automatically generates graphs and charts that visualize the current situation by comparing it with past data, allowing them to visually understand the risk situation. It also includes functions to request adjustments to the care plan based on the notification content, functions to automatically display stress coping methods and lifestyle improvement advice, functions to answer self-declaration forms, functions to share notifications with family and trusted stakeholders, and a dashboard for administrators that displays the risk status of multiple care recipients and caregivers in a list, allowing them to prioritize identifying and responding to high-risk individuals.The Security Management Department implements security measures such as data encryption, access control, tamper prevention, anonymization, and digital signatures at each stage of acquisition, storage, analysis, and notification. For example, it uses encrypted communication when transmitting biometric and business information, anonymizes personally identifiable information when storing data on the server, controls access so that only users with access rights can access the data, and prevents tampering with or leakage of notification content. The External Collaboration Department collaborates with medical institutions' electronic medical record systems, rehabilitation systems, nutrition guidance systems, family communication systems, and administrative service collaboration systems via APIs, etc., and can automate information sharing with external experts and families, collaboration with medical and welfare services, and reporting to the government as needed. The Action Support Department can perform actions such as automatically proposing care plans, automatically generating rehabilitation menus, automatically presenting nutrition guidance, automatically sending medication reminders, scheduling consultations with family and professionals, proposing events to promote workplace communication, automatically guiding users through mental health care programs, and automatically proposing measures to prevent employee turnover, based on the content of notifications. Thus, the system according to this embodiment works in cooperation with each component to acquire, store, analyze, and notify diverse biometric and operational information with high accuracy and security. It utilizes generative AI to detect early signs of mental illness, physical illness, and serious care risks, and provides specific and detailed risk information and countermeasures to the individual, family, administrator, and external experts. This significantly improves safety and the quality of care in care settings, and comprehensively realizes collaboration and action support with various stakeholders such as medical professionals, welfare workers, government officials, and family members.

[0068] (Implementation Steps)

[0069] Step 1: Multifaceted automatic acquisition of care recipient information The care recipient information acquisition unit works in conjunction with wearable devices, sensors within care facilities, smartphones, tablets, cameras, microphones, medication management devices, excretion sensors, bathroom sensors, door sensors, GPS terminals, meal logging apps, etc., to automatically acquire diverse biometric and lifestyle information in real time or periodically, such as heart rate, blood pressure, body temperature, respiratory rate, blood oxygen saturation, sleep duration, sleep quality, activity level, steps taken, exercise intensity, sitting time, number of times standing up, food intake, food content, fluid intake, medication status, fall detection, facial expression analysis, voice emotion analysis, excretion status, bathing status, frequency of going out, frequency of social interaction, skin condition, weight, BMI, walking speed, grip strength, number of chews, swallowing status, abnormalities in cough and breath sounds, speech volume, self-talk / wandering detection, and self-reporting via an emotion diary app. For example, this includes daily measurement of body temperature, blood pressure, and blood oxygen saturation; nighttime sleep tracking using a sleep tracker; photo capture during meals and AI-based estimation of meal content; automatic recording of fluid intake; automatic recording of medication adherence using a medication management device; automatic alarm activation in case of falls; emotional monitoring using a facial camera and voice microphone; excretion recording using an excretion sensor; bathing recording using a bathroom sensor; detection of going out using GPS and door sensors; understanding the frequency of social interaction through visitor and conversation records; detection of pressure ulcers and rashes through AI analysis of skin images; muscle strength measurement using a weighing scale and grip strength meter; understanding of eating habits using chewing and swallowing sensors; detection of respiratory abnormalities through AI analysis of coughs and breath sounds; automatic detection of speech volume, talking to oneself, and wandering; and self-reporting through an emotion diary app.

[0070] Step 2: Multifaceted automated acquisition of caregiver information The care worker information acquisition unit integrates with shift management systems, task management systems, care record systems, self-reporting apps, wearable devices, break management systems, survey systems, performance evaluation systems, feedback record systems, etc., to automatically acquire work information and stress information such as working hours, number of night shifts, number of tasks, delays in care record input, stress indicators, break acquisition status, self-reported fatigue level, mental state, job satisfaction, self-reported stress level, performance evaluation, feedback content, intention to leave, sleep quality, heart rate variability, reports of poor health, frequency of workplace communication, distance traveled during work, amount of speech during work, facial expression analysis during work, and accident / near-miss reports. For example, this includes obtaining work records from shift management systems, obtaining care records from task management apps, obtaining heart rate variability and stress levels from wearable devices, detecting input delays in recording systems, inputting fatigue and decreased motivation from self-reporting apps, understanding break times through break clocking and sensors, regularly obtaining job satisfaction and stress levels through questionnaires, collecting work evaluations and feedback from supervisors and colleagues, obtaining questionnaires on intention to leave the company, understanding sleep quality through sleep trackers, estimating stress through heart rate variability analysis, managing health through health reporting apps, understanding conversation frequency through workplace communication apps, automatically measuring distance traveled and speech volume during work, understanding emotional changes through facial expression analysis during work, and collecting risk information through accident and near-miss reporting systems.

[0071] Step 3: Time-series storage and management of multidimensional data The data storage unit centrally stores and manages all information obtained from care recipients and caregivers as a time-series database on the server, organized by user, date, and indicator. For example, it stores data for care recipient A from June 1st to June 30th, 2024, including heart rate, blood pressure, body temperature, respiratory rate, blood oxygen saturation, sleep duration, sleep quality, activity level, food intake, food content, fluid intake, medication status, fall records, facial expression analysis results, voice emotion analysis results, excretion status, bathing status, frequency of going out, frequency of social interaction, skin condition, weight, BMI, walking speed, grip strength, chewing frequency, swallowing status, abnormalities in cough and breath sounds, speech volume, self-talk / wandering records, and emotions. The diary app records self-reported information on a daily basis, including caregiver B's working hours, number of night shifts, number of tasks, delays in care record entry, stress indicators, break status, self-reported fatigue level, mental state, job satisfaction, job evaluation, feedback content, intention to leave, sleep quality, heart rate variability, reports of poor health, frequency of workplace communication, distance traveled during work, amount of speech during work, facial expression analysis during work, and accident / near-miss reports. Data transmission is encrypted and anonymized to ensure privacy and security.

[0072] Step 4: Multidimensional analysis and risk extraction using generative AI The generational AI analysis and notification unit uses the historical data of care recipients and caregivers stored in the data storage unit to analyze heart rate, blood pressure, body temperature, respiratory rate, blood oxygen saturation, sleep, activity level, food intake, fluid intake, medication status, falls, facial expressions, vocal emotions, excretion, bathing, going out, social interaction, skin condition, weight, BMI, walking speed, grip strength, chewing frequency, swallowing status, cough and breathing sounds, speech volume, talking to oneself / wandering, emotional diary, working hours, number of night shifts, number of tasks, delay in care record entry, stress indicators, break acquisition status, self-reported fatigue level, mental state, job satisfaction, job evaluation, feedback content, intention to leave, and sleep. For each indicator, such as quality, heart rate variability, reports of poor health, frequency of workplace communication, distance traveled during work, amount of speech during work, facial expression analysis during work, and accident / near-miss reports, the system calculates the average value, standard deviation, maximum value, minimum value, rate of change, and frequency of outliers over the past 7, 30, and 90 days. It automatically models each individual's normal state and past trends, learning from short-term rapid changes, long-term gradual changes, correlations between multiple indicators, past anomaly detection cases, and actual cases of health deterioration / caregiving accidents as training data. When similar patterns appear, it determines a high risk. Specific examples of prompts to be fed into the generating AI include: "Analyze the following historical data of biometric information, lifestyle information, and work information of the care recipient and caregiver, extract signs of physical and mental risks and serious care risks, and output the details and recommended countermeasures in natural language if an abnormality is detected," "Analyze time-series data such as heart rate, blood pressure, body temperature, respiratory rate, blood oxygen saturation, sleep, activity level, food intake, fluid intake, medication status, falls, facial expressions, voice emotion, working hours, number of tasks, stress indicators, and recording delays to detect serious risks with high accuracy by analyzing short-term rapid changes, long-term gradual changes, and correlations between multiple indicators," and "Use past care accidents and health deterioration cases as training data, determine that a similar pattern is high risk when it appears, and output the reason and recommended countermeasures."

[0073] Step 5: Risk notification, action support, and visualization. The generation-based AI analysis and notification unit automatically sends a warning notification to the individual, family, and administrator if multiple abnormalities or risks are detected, indicating a serious care risk or a precursor to mental or physical abnormalities. The notification content may include: "Continued abnormal vital signs of the person being cared for," "Continued sleep deprivation," "Significantly reduced activity level," "Frequent days of insufficient food or fluid intake," "Multiple instances of missed medication," "Detected falls," "Continued emotional abnormalities," "Continued excretory abnormalities," "Decreased frequency of bathing," "Decreased outings and social interaction," "Worsening skin condition," "Weight loss," "Decreased walking speed," "Decreased grip strength," "Difficulty chewing and swallowing," "Coughing or abnormal breathing sounds," "Decreased speech," "Increased talking to oneself or wandering," "Increased negative reporting in emotional diaries," "Continued long working hours for caregivers," "Frequent night shifts," "Continued high stress levels," "Multiple instances of delays in care record keeping," "Insufficient rest," and "Self-reported fatigue." The report includes specific risk items such as "deterioration of physical and mental state," "decreased job satisfaction," "negative trend in work evaluations and feedback," "increased intention to leave the company," "decreased sleep quality," "decreased heart rate variability," "increased reports of poor physical condition," "decreased frequency of workplace communication," "decreased distance traveled and amount of speech during work," "increased negative emotions in facial expression analysis during work," and "increased accident and near-miss reports," along with recommended countermeasures (such as visiting a medical institution, reviewing care plans, recommending rest for care workers, contacting family members, consulting with professionals, nutritional guidance, rehabilitation, environmental adjustments, promotion of workplace communication, review of work assignments, mental health care, and measures to prevent employee turnover). The user interface provides screens and applications that allow the individual, family, and administrator to easily view notification content and analysis results. For example, when the individual or family receives a notification, it automatically generates graphs and charts that visualize the current situation by comparing it with past data, allowing them to visually understand the risk situation. It also includes functions to request adjustments to the care plan based on the notification content, functions to automatically display stress coping methods and lifestyle improvement advice, functions to answer self-declaration forms, functions to share notifications with family and trusted stakeholders, and a dashboard for administrators that displays the risk status of multiple care recipients and caregivers in a list, allowing them to prioritize identifying and responding to high-risk individuals.

[0074] (Specific use cases)

[0075] For example, when the system of the present invention is applied in a nursing care facility, the care recipient information acquisition unit works in conjunction with wearable devices, facility sensors, smartphones, tablets, cameras, microphones, medication management devices, excretion sensors, bathroom sensors, door sensors, GPS terminals, meal recording apps, etc., to automatically acquire a variety of biometric and lifestyle information in real time or periodically, such as heart rate, blood pressure, body temperature, respiratory rate, blood oxygen saturation, sleep duration, sleep quality, activity level, steps taken, exercise intensity, sitting time, number of times standing up, food intake, food content, fluid intake, medication status, fall detection, facial expression analysis, voice emotion analysis, excretion status, bathing status, frequency of going out, frequency of social interaction, skin condition, weight, BMI, walking speed, grip strength, number of chews, swallowing status, abnormalities in cough and respiratory sounds, speech volume, self-talk / wandering detection, and self-reporting via an emotion diary app. For example, it includes automatic recording of daily body temperature, blood pressure, and blood oxygen saturation measurements; sleep tracking using a sleep tracker at night; photo capture during meals and AI-based estimation of meal content; automatic recording of fluid intake; automatic recording of medication adherence using a medication management device; automatic alarm activation in case of falls; emotional monitoring using a facial camera and voice microphone; excretion recording using an excretion sensor; bathing recording using a bathroom sensor; detection of going out using GPS and door sensors; understanding the frequency of social interaction through visitor and conversation records; detection of pressure ulcers and rashes through AI analysis of skin images; muscle strength measurement using a weighing scale and grip strength meter; understanding of eating habits using chewing and swallowing sensors; detection of respiratory abnormalities through AI analysis of coughs and breath sounds; automatic detection of speech volume, talking to oneself, and wandering; and self-reporting through an emotion diary app.

[0076] Furthermore, the care worker information acquisition unit integrates with shift management systems, task management systems, care record systems, self-reporting apps, wearable devices, break management systems, survey systems, performance evaluation systems, feedback record systems, etc., to automatically acquire work information and stress information such as working hours, number of night shifts, number of tasks, delays in care record input, stress indicators, break acquisition status, self-reported fatigue level, mental state, job satisfaction, self-reported stress level, performance evaluation, feedback content, intention to leave, sleep quality, heart rate variability, reports of poor health, frequency of workplace communication, distance traveled during work, amount of speech during work, facial expression analysis during work, and accident / near-miss reports. For example, this includes obtaining work records from shift management systems, obtaining care records from task management apps, obtaining heart rate variability and stress levels from wearable devices, detecting input delays in recording systems, inputting fatigue and decreased motivation from self-reporting apps, understanding break times through break clocking and sensors, regularly obtaining job satisfaction and stress levels through questionnaires, collecting work evaluations and feedback from supervisors and colleagues, obtaining questionnaires on intention to leave, understanding sleep quality with sleep trackers, estimating stress through heart rate variability analysis, managing health through health reporting apps, understanding conversation frequency through workplace communication apps, automatically measuring distance traveled and speech volume during work, understanding emotional changes through facial expression analysis during work, and collecting risk information through accident and near-miss reporting systems.

[0077] The data storage unit centrally stores and manages all information obtained from care recipients and caregivers as a time-series database on the server, organized by user, date, and indicator. The generational AI analysis and notification unit uses this historical data to analyze and notify the following: heart rate, blood pressure, body temperature, respiratory rate, blood oxygen saturation, sleep, activity level, food intake, fluid intake, medication status, falls, facial expressions, vocal emotions, excretion, bathing, going out, social interaction, skin condition, weight, BMI, walking speed, grip strength, chewing count, swallowing status, cough and breathing sounds, speech volume, talking to oneself / wandering, emotional diary, working hours, number of night shifts, number of tasks, delay in care record entry, stress indicators, break acquisition status, self-reported fatigue level, mental state, etc. For various indicators such as job satisfaction, job evaluation, feedback content, intention to leave, sleep quality, heart rate variability, reports of poor health, frequency of workplace communication, distance traveled during work, amount of speech during work, facial expression analysis during work, and accident / near-miss reports, the system calculates the average value, standard deviation, maximum value, minimum value, rate of change, and frequency of outlier occurrences over the past 7, 30, and 90 days. It automatically models each individual's normal state and past trends, learning from short-term rapid changes, long-term gradual changes, correlations between multiple indicators, past anomaly detection cases, and actual cases of health deterioration / caregiving accidents as training data. When similar patterns appear, it determines a high risk. For example, if a care recipient's heart rate, blood pressure, body temperature, respiratory rate, and blood oxygen saturation rise above normal levels, they experience fewer than 5 hours of sleep for several consecutive days, their activity level decreases, their food and fluid intake decreases, they forget to take their medication multiple times, falls are detected, anxiety and irritability are frequently detected from their facial expressions and voice, they experience bowel and bladder abnormalities, their bathing frequency decreases, their outings and social interactions decrease, their skin condition worsens, they lose weight, their walking speed decreases, their grip strength decreases, they have difficulty chewing and swallowing, their cough and breathing sounds are abnormal, their speech decreases, their self-talk and wandering increase, and they frequently report negative things in their emotional diary, these are judged to be signs of serious physical and mental risks such as dehydration, infection, progression of dementia, depression, risk of falls, aspiration pneumonia, malnutrition, and pressure ulcer risk.For care workers, if they experience consecutive days of working more than 12 hours, frequent night shifts, a large number of tasks, high stress indicators, multiple delays in care record keeping, insufficient breaks, worsening self-reported fatigue and mental state, decreased job satisfaction, negative trends in job evaluations and feedback, increased intention to leave, decreased sleep quality, decreased heart rate variability, increased reports of poor health, decreased frequency of workplace communication, decreased distance traveled and volume of speech during work, increased negative emotions in facial expression analysis during work, and increased accident and near-miss reports, they are judged to be at high risk of burnout, care errors, and turnover.

[0078] Specific examples of prompts to be fed into the generating AI include: "Analyze the following historical data of biometric information, lifestyle information, and work information of the care recipient and caregiver, extract signs of physical and mental risks and serious care risks, and output the details and recommended countermeasures in natural language if an abnormality is detected," "Analyze time-series data such as heart rate, blood pressure, body temperature, respiratory rate, blood oxygen saturation, sleep, activity level, food intake, fluid intake, medication status, falls, facial expressions, voice emotion, working hours, number of tasks, stress indicators, and recording delays to detect serious risks with high accuracy by analyzing short-term rapid changes, long-term gradual changes, and correlations between multiple indicators," and "Use past care accidents and health deterioration cases as training data, determine that a similar pattern is high risk when it appears, and output the reason and recommended countermeasures."

[0079] The generation-based AI analysis and notification unit automatically sends a warning notification to the individual, family, and administrator if multiple abnormalities or risks are detected, indicating a serious care risk or a precursor to mental or physical abnormalities. The notification content may include: "Continued abnormal vital signs of the person being cared for," "Continued sleep deprivation," "Significantly reduced activity level," "Frequent days of insufficient food or fluid intake," "Multiple instances of missed medication," "Detected falls," "Continued emotional abnormalities," "Continued excretory abnormalities," "Decreased frequency of bathing," "Decreased outings and social interaction," "Worsening skin condition," "Weight loss," "Decreased walking speed," "Decreased grip strength," "Difficulty chewing and swallowing," "Coughing or abnormal breathing sounds," "Decreased speech," "Increased talking to oneself or wandering," "Increased negative reporting in emotional diaries," "Continued long working hours for caregivers," "Frequent night shifts," "Continued high stress levels," "Multiple instances of delays in care record keeping," "Insufficient rest," and "Self-reported fatigue." The report includes specific risk items such as "deterioration of physical and mental state," "decreased job satisfaction," "negative trend in work evaluations and feedback," "increased intention to leave the company," "decreased sleep quality," "decreased heart rate variability," "increased reports of poor physical condition," "decreased frequency of workplace communication," "decreased distance traveled and amount of speech during work," "increased negative emotions in facial expression analysis during work," and "increased accident and near-miss reports," along with recommended countermeasures (such as visiting a medical institution, reviewing care plans, recommending rest for care workers, contacting family members, consulting with professionals, nutritional guidance, rehabilitation, environmental adjustments, promotion of workplace communication, review of work assignments, mental health care, and measures to prevent employee turnover). The user interface provides screens and applications that allow the individual, their family, and administrators to easily view notification content and analysis results. For example, when the individual or their family receives a notification, it automatically generates graphs and charts that visualize the current situation by comparing it with past data, allowing them to visually grasp the risk situation. It also includes functions to request adjustments to the care plan based on the notification content, to automatically display stress coping methods and lifestyle improvement advice, to answer self-declaration forms, to share notifications with family and trusted stakeholders, and to provide administrators with a dashboard that displays the risk status of multiple care recipients and caregivers in a list, enabling them to prioritize identifying and responding to high-risk individuals. In this way, the system of the present invention realizes multifaceted risk management and improvement of the quality of care in the care setting.

[0080] The specific processing unit 290 transmits the result of the specific processing obtained in the embodiment 1 or application example 1 described above to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0081] Data generation model 58 is a form of so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> Examples of generative AI include text generation AI, image generation AI, and multimodal generation AI. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images (e.g., still image data or video data). The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference result in one or more data formats from audio data, text data, and image data. The data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specific processing unit 290 performs the specific processing described above using the data generation model 58. The data generation model 58 may be a fine-tuned model that outputs inference results from prompts that do not contain instructions, in which case the data generation model 58 can output inference results from prompts that do not contain instructions. In the data processing device 12, etc., there are multiple types of data generation models 58, and the data generation model 58 includes AI other than generative AI. AI other than generative AI includes, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVMs), k-means clustering, convolutional neural networks (CNNs), recurrent neural networks (RNNs), generative adversarial networks (GANs), or naive Bayes, and can perform various processes, but is not limited to these examples. Also, the AI ​​may be an AI agent. Furthermore, when the processing of each of the above parts is performed by the AI, the processing may be performed by the AI ​​in part or in whole, but is not limited to this example.Furthermore, processing performed by AI, including generative AI, may be replaced with rule-based processing, and rule-based processing may be replaced with processing performed by AI, including generative AI.

[0082] Furthermore, the processing performed by the data processing system 10 described above is carried out by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart device 14, but it may also be carried out by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart device 14. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information necessary for processing from the smart device 14 or an external device, and the smart device 14 acquires or collects information necessary for processing from the data processing device 12 or an external device.

[0083] For example, the data collection unit is implemented by the control unit 46A of the smart device 14 or the specific processing unit 290 of the data processing device 12. For example, the acquisition unit acquires step count data using the camera 42 and communication I / F 44 of the smart device 14 and processes it with the specific processing unit 290 of the data processing device 12. For example, the analysis unit is implemented by the specific processing unit 290 of the data processing device 12 and analyzes the data from the data collection unit and the acquisition unit. For example, the generation unit is implemented by the specific processing unit 290 of the data processing device 12 and generates a cooking menu using generation AI. For example, the serving unit is implemented by the output device 40 of the smart device 14 or the specific processing unit 290 of the data processing device 12 and provides the generated cooking menu to the user. The correspondence between each unit and the devices and control units is not limited to the examples described above and can be modified in various ways.

[0084] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0085] [Second Embodiment]

[0086] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0087] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0088] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0089] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and communication interface 44 are also connected to the bus 52.

[0090] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0091] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0092] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0093] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0094] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0095] Storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the user's emotions using the emotion identification model 59 and perform identification processing using the user's emotions. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotions, including but not limited to these examples. Furthermore, emotion estimation and prediction also include, for example, emotion analysis.

[0096] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48. The smart glasses 214 also has a data generation model 58 and an emotion identification model 59, similar to the data generation model and emotion identification model 59, and can perform processing similar to that of the identification processing unit 290 using these models.

[0097] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. Each part of the system described below is realized by the data processing device 12 and the smart glasses 214. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0098] (Example 1)

[0099] Since the process flow is the same as in Example 1 described in the First Embodiment above, the explanation will be omitted.

[0100] (Application Example 1) Since the process flow is the same as in Application Example 1 described in the first embodiment above, the explanation will be omitted.

[0101] The specific processing unit 290 transmits the result of the specific processing obtained in the embodiment 1 or application example 1 described above to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0102] Furthermore, the processing by the data processing system 10 described above is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart glasses 214, but it may also be performed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart glasses 214. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information necessary for processing from the smart glasses 214 or an external device, and the smart glasses 214 acquires or collects information necessary for processing from the data processing device 12 or an external device.

[0103] For example, the data collection unit is implemented by the control unit 46A of the smart glasses 214 or the specific processing unit 290 of the data processing device 12. For example, the acquisition unit acquires step count data using the camera 42 and communication I / F 44 of the smart glasses 214 and processes it using the specific processing unit 290 of the data processing device 12. For example, the analysis unit is implemented by the specific processing unit 290 of the data processing device 12 and analyzes the data from the data collection unit and the acquisition unit. For example, the generation unit is implemented by the specific processing unit 290 of the data processing device 12 and generates a cooking menu using generation AI. For example, the serving unit is implemented by the speaker 240 of the smart glasses 214 or the specific processing unit 290 of the data processing device 12 and provides the generated cooking menu to the user. The correspondence between each unit and the device or control unit is not limited to the examples described above and can be modified in various ways.

[0104] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0105] [Third Embodiment]

[0106] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0107] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0108] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0109] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, display 343, and communication interface 44 are also connected to the bus 52.

[0110] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0111] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0112] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0113] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0114] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0115] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0116] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0117] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. Each part of the system described below is implemented by the data processing device 12 and the headset terminal 314. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0118] (Example 1) Since the process flow is the same as in Example 1 described in the First Embodiment above, the explanation will be omitted.

[0119] (Application Example 1) Since the process flow is the same as in Application Example 1 described in the first embodiment above, the explanation will be omitted.

[0120] The specific processing unit 290 transmits the result of the specific processing obtained in the embodiment 1 or application example 1 described above to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0121] Furthermore, the processing performed by the data processing system 10 described above is carried out by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the headset terminal 314, but it may also be carried out by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the headset terminal 314. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information necessary for processing from the headset terminal 314 or an external device, and the headset terminal 314 acquires or collects information necessary for processing from the data processing device 12 or an external device.

[0122] For example, the data collection unit is implemented by the control unit 46A of the headset terminal 314 or the specific processing unit 290 of the data processing device 12. For example, the acquisition unit acquires step count data using the camera 42 and communication I / F 44 of the headset terminal 314 and processes it using the specific processing unit 290 of the data processing device 12. For example, the analysis unit is implemented by the specific processing unit 290 of the data processing device 12 and analyzes the data from the data collection unit and the acquisition unit. For example, the generation unit is implemented by the specific processing unit 290 of the data processing device 12 and generates a cooking menu using generation AI. For example, the serving unit is implemented by the speaker 240 and display 343 of the headset terminal 314 or the specific processing unit 290 of the data processing device 12 and provides the generated cooking menu to the user. The correspondence between each unit and the device or control unit is not limited to the examples described above and can be modified in various ways.

[0123] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0124] [Fourth Embodiment]

[0125] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0126] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0127] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0128] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, controlled object 443, and communication interface 44 are also connected to the bus 52.

[0129] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0130] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0131] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0132] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0133] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0134] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0135] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0136] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0137] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. Each part of the system described below is implemented by the data processing device 12 and the robot 414. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0138] (Example 1) Since the process flow is the same as in Example 1 described in the First Embodiment above, the explanation will be omitted.

[0139] (Application Example 1) Since the process flow is the same as in Application Example 1 described in the first embodiment above, the explanation will be omitted.

[0140] The specific processing unit 290 transmits the result of the specific processing obtained in the embodiment 1 or application example 1 described above to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0141] Furthermore, the processing performed by the data processing system 10 described above is carried out by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the robot 414, but it may also be carried out by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the robot 414. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information necessary for processing from the robot 414 or external devices, and the robot 414 acquires or collects information necessary for processing from the data processing device 12 or external devices.

[0142] For example, the data collection unit is implemented by the control unit 46A of the robot 414 or the specific processing unit 290 of the data processing device 12. For example, the acquisition unit acquires step count data using the camera 42 and communication I / F 44 of the robot 414 and processes it using the specific processing unit 290 of the data processing device 12. For example, the analysis unit is implemented by the specific processing unit 290 of the data processing device 12 and analyzes the data from the data collection unit and the acquisition unit. For example, the generation unit is implemented by the specific processing unit 290 of the data processing device 12 and generates a cooking menu using a generation AI. For example, the serving unit is implemented by the speaker 240 and controlled object 443 of the robot 414 and the specific processing unit 290 of the data processing device 12 and provides the generated cooking menu to the user. The correspondence between each unit and the device or control unit is not limited to the examples described above and can be modified in various ways.

[0143] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0144] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0145] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. In the upper and lower directions of the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. Also, the upper side of the concentric circles is where "pleasant" emotions are located, and the lower side is where "unpleasant" emotions are located. In this way, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0146] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0147] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0148] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0149] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0150] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0151] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0152] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0153] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0154] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0155] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0156] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0157] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0158] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0159] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0160] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0161] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0162] Furthermore, the following additional information is disclosed regarding the above explanation.

[0163] (Claim 1)

[0164] The system comprises a care recipient information acquisition unit, a care worker information acquisition unit, and a generative AI analysis and notification unit. The care recipient information acquisition unit automatically acquires various biometric and lifestyle information in real time or periodically from smartphones, wearable devices, sensors in care facilities, cameras, microphones, medication management devices, etc., such as heart rate, blood pressure, body temperature, sleep duration, sleep quality, activity level, steps, exercise intensity, sitting time, number of times standing up, food intake, food content, medication status, fall detection, facial expression analysis, and voice emotion analysis. The care worker information acquisition unit acquires working hours, number of night shifts, number of tasks, delay in care record input, stress indicators, break acquisition status, self-reported fatigue level, and mental state from shift management systems, task management systems, care record systems, self-reporting apps, wearable devices, etc. This system automatically acquires business information and stress information, and the generative AI analysis notification unit uses time-series data acquired from the care recipient information acquisition unit and the care worker information acquisition unit to analyze multidimensional and time-series changes and correlations between multiple indicators such as heart rate, blood pressure, body temperature, sleep, activity level, food intake, medication status, falls, facial expressions, voice emotion, working hours, number of tasks, stress indicators, and recording delays using generative AI. It accurately detects signs of mental and physical risks and serious care risks such as signs of dementia progression, depression, dehydration / infection risk, fall risk, and care worker burnout risk, and if an abnormality is detected, it automatically sends a detailed warning notification to the individual, family, and manager, including specific risk details and recommended countermeasures. (Claim 2)

[0165] The system according to claim 1, wherein the system acquires biometric and lifestyle information from the care recipient information acquisition unit, including heart rate, blood pressure, body temperature, sleep duration, sleep quality, activity level, steps taken, exercise intensity, sitting time, number of times standing up, food intake, food content, medication status, fall detection, facial expression analysis, voice emotion analysis, excretion status, bathing status, frequency of going out, frequency of social interaction, etc., and acquires work information and stress information from the care worker information acquisition unit, including working hours, number of night shifts, number of tasks, delay in care record input, stress indicators, break acquisition status, self-reported fatigue level, mental state, job satisfaction, self-reported stress level, job evaluation, feedback content, etc., and the generative AI analysis notification unit uses this diverse data to automatically learn and detect short-term rapid changes, long-term gradual changes, and complex correlation patterns between multiple indicators, and generates and notifies different notification content and care suggestions according to the type and degree of risk. (Claim 3)

[0166] The system according to claim 1, wherein the generative AI analysis notification unit automatically models the normal state and past trends for each person receiving care and caregiver, dynamically optimizes individual reference values ​​and abnormal thresholds, detects short-term rapid changes and long-term gradual changes, and correlation patterns between multiple indicators with high accuracy, and when an abnormality is detected, automatically sends a detailed notification to the person, family, and administrator including the type of abnormality detected, the degree of change, comparison with the past, estimated cause, recommended medical consultation or care plan revision, recommendation for caregiver rest, contact with family, and consultation with professionals. [Explanation of Symbols]

[0167] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots

Claims

1. A system comprising a biometric information acquisition unit, a business information acquisition unit, an analysis notification unit, and a generative AI execution unit, wherein the biometric information acquisition unit periodically and automatically acquires multiple types of biometric information such as heart rate, sleep duration, activity level, body temperature, blood pressure, and stress indicators from information terminals and wearable devices, etc.; the business information acquisition unit periodically and automatically acquires multiple types of business-related information such as working hours, overtime hours, number of tasks, work content, workload, break time, and start and end times of work from business management systems and attendance management systems, etc.; the generative AI execution unit uses the acquired biometric information and business information history data to multidimensionally analyze the trends in changes of various indicators over a certain period in the past, the frequency of occurrence of abnormal values, and the correlation between multiple indicators, etc., and extracts patterns and risk factors that indicate signs of mental illness with high accuracy; and the analysis notification unit, based on the analysis results by the generative AI execution unit, automatically sends a detailed warning notification, including the specific indicators in which abnormalities were found and their content, the estimated risk level, and recommended countermeasures, to the user's information terminal and the management screen or email of the supervisor, etc., when signs of mental illness are detected.

2. The system according to claim 1, wherein the biometric information acquisition unit acquires biometric information including heart rate, sleep duration, activity level, body temperature, blood pressure, stress index, respiratory rate, skin temperature, blood oxygen saturation, dietary content, alcohol intake, exercise history, etc., and the work information acquisition unit acquires work information including working hours, overtime hours, number of tasks, work content, workload, break time, start and end times of work, number of meetings, frequency of business trips, work evaluation, self-reported stress level, etc., and the generation AI execution unit analyzes this diverse data chronologically and statistically, automatically learns and detects multiple abnormal patterns indicating signs of mental illness, and the analysis notification unit generates and notifies different notification content and recommended countermeasures for each detected abnormal pattern.

3. The system according to claim 1, wherein the generation AI execution unit automatically models each individual's normal state and past trends using acquired biometric information and historical data of work information, dynamically optimizes each individual's reference values ​​and abnormal thresholds, detects both short-term rapid changes and long-term gradual changes with high accuracy, and when an abnormality is detected, the analysis notification unit automatically sends a detailed notification to the individual and their supervisor, including the type of abnormality detected, the degree of change, a comparison with the past, the estimated cause, and specific countermeasures such as recommended medical consultations and work adjustments.

Images