System

Abstract

A system is provided.SOLUTION: A system comprising: means for recording interactions between a parent and a child of a Mental Disability; means for pre-processing the recorded interactions to train a generative AI; means for the generative AI to communicate with the child after the parent dies; means for the generative AI to monitor the child for anomalous behavior; and means for providing an emergency notification if an anomaly is detected.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Parents of children with intellectual disabilities take care of their children on a daily basis, but there is a problem that they are no longer able to do so due to their own poor health or death. When a parent is no longer present, children with intellectual disabilities are often isolated and anxious about their daily lives and how to respond in emergencies. This increases the burden on families, and social support is currently insufficient. It is necessary to solve these problems and provide an environment where children with intellectual disabilities can live safely even after their parents have passed away. [Means for solving the problem]

[0005] The present invention provides a means for recording interactions between parents and their intellectually disabled children. It also includes a means for preprocessing the recorded data and training a generating AI. After the parents pass away, the generating AI communicates with the child and provides daily support. This generating AI also has a means for monitoring abnormal behavior in the child and sends an emergency notification if an abnormality is detected. The emergency notification is sent to pre-registered family members and welfare organizations, allowing for a prompt response. This provides a system that maintains a safe and secure living environment for intellectually disabled children after the parents pass away.

[0006] "Means for recording interactions between parents and their children with intellectual disabilities" refers to the functionality of devices or software that record communication between parents and their children with intellectual disabilities in audio, text, video, or other formats.

[0007] "Means for preprocessing recorded data and training the generative AI" refers to a function that is part of a system that preprocesses collected data, such as converting it to text and removing noise, and then inputs it into a generative AI model for training.

[0008] "Means for the generative AI to communicate with children after the parents have passed away" refers to a function that allows the generative AI to continue to interact with and provide guidance to intellectually disabled children even after the parents have passed away.

[0009] "Means for the generative AI to monitor children's abnormal behavior" refers to a function that allows the generative AI to constantly monitor the daily behavior of children with intellectual disabilities and automatically detect abnormal behavioral patterns or situations.

[0010] "Means for sending emergency notifications when abnormalities are detected" refers to communication methods or systems that will promptly notify pre-set contacts when the generating AI detects abnormal behavior in a child.

[0011] "Means for storing in a database and updating the generative AI model" refers to the function for storing preprocessed data in a database and periodically updating the generative AI learning model based on that data.

[0012] "Means for sending emergency notifications to pre-registered family members or welfare agencies" refers to a communication function that allows for rapid notification to pre-registered family members or welfare agency contacts when abnormal behavior is detected. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0014] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0017] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0018] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0019] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0021] [First embodiment]

[0022] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0023] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0026] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0029] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0033] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0034] This invention is a system that aims to provide an environment where parents of intellectually disabled children can live with peace of mind. This system records interactions between parents and their intellectually disabled children and trains a generative AI to learn from this data, allowing the generative AI to continue communicating with the child even after the parent has passed away. It also includes a function to monitor abnormal behavior and notify pre-registered family members and welfare organizations in the event of an emergency.

[0035] System Overview

[0036] Data collection and learning phase

[0037] 1. User initiates interaction

[0038] Devices: Family members' smartphones and tablets record daily interactions between parents and children in audio, text, and video formats.

[0039] Server: Receives data sent from the terminal and performs preprocessing.

[0040] 2. Data Preprocessing

[0041] Server: Converts received data into text, removes noise and unnecessary information, and structures the data by adding metadata such as time, location, and emotion tags.

[0042] 3. Generative AI Training

[0043] Server: The preprocessed data is input into the generative AI's learning model to learn the interaction patterns between parents and children, allowing the generative AI to mimic the support provided by parents.

[0044] Communication phase using generative AI

[0045] 1. Daily communication

[0046] User (child): Asks questions such as "What should we do today?" to a device such as a tablet.

[0047] Terminal: Converts this question into text data and sends it to the server.

[0048] Server: Inputs text data into the AI ​​generator and generates an appropriate response. This response is then sent back to the device and spoken back to the child.

[0049] Examples:

[0050] User (child): "What shall we do today?"

[0051] Generative AI: "How about going to the park today?"

[0052] Emergency Notification Phase

[0053] 1. Monitoring for Abnormal Behavior

[0054] Generative AI: Constantly monitors the child's behavior and condition, and if it detects behavior that deviates from the norm, it determines that it is abnormal.

[0055] Trigger emergency notifications if an anomaly is detected.

[0056] 2. Sending emergency notifications

[0057] Server: Generates emergency notification messages and sends them to pre-defined contacts (family members and welfare agencies).

[0058] Examples:

[0059] Generative AI: Detects when a child loses consciousness.

[0060] Server: Send emergency notification to family: "Your child has lost consciousness. Please check the scene immediately."

[0061] Example

[0062] In a concrete example, the process works as follows:

[0063] 1. User (child): Communicate regularly with the generated AI in daily life.

[0064] 2. Generative AI: Acts as a parent, providing appropriate advice and instructions.

[0065] 3. In case of abnormality: The generated AI detects abnormal behavior of the child and the server sends an emergency notification.

[0066] 4. Who to notify: Families and welfare agencies that receive the notification will respond promptly.

[0067] In this way, this invention is a system that provides an environment where children with intellectual disabilities can live safely even after their parents pass away. The data processing and generative AI functions of each phase make continuous support possible.

[0068] The processing flow will be explained below.

[0069] Data collection and learning phase

[0070] Step 1:

[0071] Users use devices such as smartphones and tablets to record and record conversations and actions between parents and children.

[0072] Step 2:

[0073] The device sends the recorded data to the server in real time.

[0074] Step 3:

[0075] The server preprocesses the received data, converting the audio data to text, removing noise, trimming unnecessary parts, and extracting important information.

[0076] Step 4:

[0077] The server adds metadata (time, location, emotion tags, etc.) to the preprocessed data and stores it in a database.

[0078] Step 5:

[0079] The server inputs the stored data into the Generative AI's learning algorithm, which learns characteristics from parent-child interactions and builds a model to generate future responses.

[0080] Communication phase using generative AI

[0081] Step 6:

[0082] In the course of daily interactions, the user (child) asks questions such as, "What shall we do today?"

[0083] Step 7:

[0084] The device records the user's voice and converts it into text data (voice recognition function).

[0085] Step 8:

[0086] The terminal transmits the converted text data to the server.

[0087] Step 9:

[0088] The server inputs the received text data into the generation AI, which then generates an appropriate response.

[0089] Step 10:

[0090] The server transmits the generated response to the terminal as text data.

[0091] Step 11:

[0092] The terminal converts the text data into speech and plays it back to the user.

[0093] Emergency Notification Phase

[0094] Step 12:

[0095] The generative AI monitors a child's daily behavior and detects any unusual behavior (e.g., suddenly collapsing).

[0096] Step 13:

[0097] When the generative AI detects abnormal behavior, it sends a signal to the server that triggers an emergency notification.

[0098] Step 14:

[0099] The server generates the content of the emergency notification and sends it to pre-registered family members and welfare agencies.

[0100] Step 15:

[0101] The user (family or welfare agency) receives a notification and promptly goes to the scene or takes the necessary action (e.g., calls 119).

[0102] The system's program records detailed daily interactions between parents and their intellectually disabled children, and trains the AI ​​to learn from these interactions, providing ongoing support for the child even after the parents have passed away. It also has a built-in notification system for rapid response in emergencies.

[0103] Example 1

[0104] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0105] There is a lack of support for intellectually disabled children to live safely after their parents pass away. There is no way to continue communicating with their children, monitoring their behavior, or responding in the event of an emergency, as parents did while they were alive, which raises the issue of the possibility that the safety and quality of life of the children may decline.

[0106] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0107] In this invention, the server includes a device that records interactions between parents and their intellectually disabled children, a device that preprocesses the recorded data and trains a generative AI model, a device that allows the generative AI model to communicate with the child after the parents pass away, a device that allows the generative AI model to monitor abnormal behavior in the child, and a device that issues an emergency notification if an abnormality is detected. This enables intellectually disabled children to continue living in a safe and stable environment even after the parents pass away.

[0108] "Devices that record interactions between parents and children with intellectual disabilities" means devices that collect and retain audio, text, and video communications between parents and children with intellectual disabilities.

[0109] The "device that preprocesses recorded data and trains a generative AI model" is a system that performs preprocessing such as noise removal, text conversion, and metadata addition based on recorded interaction data, and then uses that data to train a generative AI model.

[0110] The "device that allows a generative AI model to communicate with children after their parents have passed away" is a system that allows a generative AI model to continuously interact with and support children with intellectual disabilities based on data learned, even when their parents are no longer present.

[0111] The "device in which a generative AI model monitors children's abnormal behavior" is a system in which a generative AI model constantly monitors children's behavior and judges it as abnormal if it deviates from normal behavior patterns.

[0112] The "device that sends emergency notifications when an abnormality is detected" is a system that sends emergency notifications to pre-registered contacts when the AI ​​model detects abnormal behavior in a child.

[0113] "Means for storing preprocessed data in an information storage device and updating the configuration of the generative AI model" refers to a technology for restoring preprocessed data to a database or storage device and using that data to update the algorithms and parameters of the generative AI model.

[0114] "Means for sending emergency notifications to pre-registered family members and welfare agencies" refers to a system that sends emergency messages to pre-designated family members and welfare agencies when a child exhibits abnormal behavior.

[0115] This invention is a system that provides a safe and secure environment for parents of intellectually disabled children. The system records interactions between parents and their children and trains a generative AI model to learn from this data, enabling the model to continue communicating with the child after the parent's death. The system also includes a function in which the generative AI model monitors for abnormal behavior and notifies pre-registered family members and welfare agencies in the event of an emergency.

[0116] Data collection and learning phase

[0117] First, users (parents and children) use devices such as smartphones and tablets to carry out everyday interactions. These interactions are recorded in audio, text, and video formats. For example, a parent might say, "Good morning, how was your day?", and a child might respond, "I drew a picture at school today."

[0118] The device then sends the recorded data to a server, which converts the received data into text using voice recognition technology (e.g., Google Speech-to-Text API), removes noise, filters out unnecessary information, and extracts only the important information. The server also structures the data by adding metadata such as time, location, and emotion tags.

[0119] The server then feeds the pre-processed data into a generative AI model to learn parent-child interaction patterns, allowing the model to mimic the support provided by the parent.

[0120] Communication phase using generative AI

[0121] In everyday communication, a user (child) will ask a device such as a tablet, "What shall we do today?" This question is converted into text data using speech recognition technology and sent to a server. The server inputs the text data into a generative AI model to generate an appropriate response. This response is sent back to the device and is returned to the child as a voice using speech synthesis technology (e.g., Google Text-to-Speech API).

[0122] For example, if a child asks, "What shall we do today?", the generative AI model might suggest, "How about going to the park today?"

[0123] Emergency Notification Phase

[0124] The system's generative AI model constantly monitors a child's behavior and state, and if it detects behavior that deviates from the norm, it will determine that something is abnormal, such as if the child loses consciousness or exhibits unusual behavioral patterns.

[0125] If an abnormality is detected, the server generates an emergency notification message and sends it to pre-defined contacts (family members or welfare agencies). For example, it could send a message via SMS or email saying, "Your child has lost consciousness. Please check the scene immediately."

[0126] In this way, the present invention is a system that realizes functions such as recording interactions with parents, learning using a generative AI model, communication, monitoring abnormal behavior, and emergency notifications, in order to provide an environment where intellectually disabled children can live safely even after their parents have passed away.

[0127] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0128] Step 1:

[0129] User initiates interaction

[0130] Input: Daily interactions between users (parents and children) (voice, text, video)

[0131] Action: Parent and child engage in everyday interactions, such as parent saying, "Good morning, how was your day?" and child responding, "I drew a picture at school today."

[0132] Output: Interaction data is recorded on a smartphone or tablet

[0133] Step 2:

[0134] Sending data

[0135] Input: Recorded interaction data (audio, text, video)

[0136] How it works: The device sends the recorded data to a server. For example, audio data recorded using a tablet's microphone is uploaded to a cloud server via Wi-Fi.

[0137] Output: The server receives the data

[0138] Step 3:

[0139] Data Preprocessing

[0140] Input: Data of the interaction received by the server (audio, text, video)

[0141] How it works: The server uses speech recognition technology (e.g., Google Speech-to-Text API) to convert voice data into text, remove noise, and filter out unnecessary information. It also structures the data by adding metadata such as time, location, and emotion tags. For example, voice data such as "Good morning, how was your day?" is converted to text like "Good morning, how was your day?", and metadata such as October 10, 2023, and the living room are added.

[0142] Output: Preprocessed text data

[0143] Step 4:

[0144] Training generative AI models

[0145] Input: Preprocessed text data

[0146] How it works: The server feeds the preprocessed data into a generative AI model that learns parent-child interaction patterns, such as what response a parent gets when they say "good morning."

[0147] Output: A trained generative AI model

[0148] Step 5:

[0149] Daily communication

[0150] Input: Questions (voice) asked by the user (child) to the device

[0151] How it works: The user (child) asks a tablet or other device, "What shall we do today?" This question is converted into text data using speech recognition technology and sent to a server. The server inputs the text data into a generative AI model and generates an appropriate response. This response is sent back to the device, and is replies to the child as audio using speech synthesis technology (e.g., Google Text-to-Speech API).

[0152] How it works: When a child asks, "What shall we do today?", the generative AI model suggests, "How about going to the park today?"

[0153] Output: Voice response

[0154] Step 6:

[0155] Abnormal behavior monitoring

[0156] Input: User (child) behavior data

[0157] How it works: The generative AI model constantly monitors the child's behavior and state, and if it detects any behavior that deviates from the norm, it will judge it to be abnormal. For example, if the child suddenly stops moving or exhibits an unusual behavior pattern, it will detect an abnormality.

[0158] Output: Anomaly detection alert

[0159] Step 7:

[0160] Sending emergency notifications

[0161] Input: Anomaly detection alert, pre-registered contact information

[0162] What it does: The server generates an emergency notification message and sends it to pre-defined contacts (family or welfare agencies), for example, an SMS or email message saying "Your child has lost consciousness. Please check the scene immediately."

[0163] Output: Send emergency notification message

[0164] (Application example 1)

[0165] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0166] There is a need to reduce the anxiety that users who particularly need supervision, such as people with intellectual disabilities and the elderly, experience in their daily lives. It is important to provide an environment where users can live safely even when their parents or caregivers are not always nearby. There is also a need for a system that can respond quickly in the event of abnormal behavior or an emergency. Furthermore, there is a need to ensure that users do not feel lonely even after their parents pass away, and that they continue to receive the necessary support.

[0167] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0168] In this invention, the server includes a means for recording interactions between parents and persons with intellectual disabilities, a means for preprocessing the recorded data and having the generation AI learn from it, a means for the generation AI to communicate with the user after the parent has passed away, a means for the generation AI to monitor the user's abnormal behavior, a means for sending an emergency notification when an abnormality is detected, a means for monitoring the user's behavior and status using the camera and microphone of the smart device, a means for the generation AI to communicate with the user in real time, and a means for sending a notification to pre-set contacts when abnormal behavior is detected. This provides an environment where users can live safely and enables rapid response in emergencies.

[0169] A "means for recording interactions between a parent and a child with an intellectual disability" is a device or method for recording, by audio, text, video, or other means, the conversations or activities between a parent and a child with an intellectual disability.

[0170] "Means for preprocessing recorded data and training a generative AI" refers to a device or method for converting recorded data into text, performing preprocessing such as noise removal and adding metadata, and then training the generative AI model.

[0171] "Means for the generative AI to communicate with the user after the parents have passed away" refers to a device or method that allows the generative AI to interact with the user on behalf of the parents and provide necessary support and advice.

[0172] "Means for monitoring abnormal user behavior by a generation AI" refers to a device or method by which a generation AI monitors a user's daily behavior and detects abnormal behavior.

[0173] The "means for sending an emergency notification when an abnormality is detected" refers to a device or method for sending an emergency notification to pre-defined contacts when abnormal user behavior is detected.

[0174] "Means for monitoring a user's behavior and status using a camera or microphone on a smart device" refers to a device or method for recording and monitoring a user's behavior and status using a camera and microphone on a device such as a smartphone or head-mounted display.

[0175] "Means for the generation AI to communicate with the user in real time" refers to a device or method that enables the generation AI to respond immediately to the user's questions or requests and realize real-time dialogue.

[0176] "Means for sending notifications to pre-defined contacts when abnormal behavior is detected" refers to a device or method for sending notifications to pre-defined emergency contacts (such as family members or welfare agencies) when abnormal behavior of a user is detected.

[0177] The embodiments for implementing the invention are carried out in the following methods and steps.

[0178] Hardware and Software Use

[0179] 1. Hardware

[0180] Mobile devices such as smartphones and tablets

[0181] head-mounted display

[0182] Cloud server (e.g. AWS, Google Cloud)

[0183] Camera and microphone

[0184] 2. Software

[0185] Data collection applications (e.g., applications developed with Flutter)

[0186] Data preprocessing script (e.g., Python)

[0187] Natural language processing libraries (e.g. NLTK)

[0188] Generative AI models (e.g., OpenAI GPT-4)

[0189] Emergency notification API (e.g. Twilio API)

[0190] Anomaly detection algorithms (e.g., TensorFlow)

[0191] Data collection

[0192] The device (smartphone or tablet) records the interactions between parents and their children with intellectual disabilities in audio, text, and video formats, and the recorded data is sent to a cloud server where it is stored.

[0193] Data Preprocessing

[0194] The server converts the transmitted data into text, removes noise, and structures the data by adding metadata such as time, location, and emotion tags. This preprocessed data is then stored in a database.

[0195] Generative AI training

[0196] The server inputs the preprocessed data into the generative AI's learning model, learning the interaction patterns between parents and children, allowing the generative AI to provide appropriate support as a substitute parent.

[0197] Communication with users

[0198] The user (child) asks everyday questions or requests to the device, such as "What shall we do today?"

[0199] Examples:

[0200] User: "What should I do today?"

[0201] Generative AI: "How about going to the park today?"

[0202] The server converts the user's question into text data, inputs it into a generation AI, and generates an appropriate response. This response is then sent back to the device and replies to the user via voice.

[0203] Abnormal behavior monitoring and emergency notification

[0204] The AI ​​constantly monitors the user's behavior and status, and if it detects any behavior that deviates from the norm, it judges it to be an anomaly. If an anomaly is detected, the server will send an emergency notification to pre-defined contacts.

[0205] Examples:

[0206] Generative AI: Detects when the user has lost consciousness.

[0207] Server: Send emergency notification to family: "Your child has lost consciousness. Please check the scene immediately."

[0208] Prompt Sentence Examples

[0209] An example of a prompt sentence might be:

[0210] "Design a system that constantly monitors the user's mood and state and sends emergency notifications at the appropriate time."

[0211] "How can I automatically notify family members or designated contacts when abnormal behavior is detected?"

[0212] This system allows users to live with peace of mind and enables quick response in emergencies.

[0213] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0214] Step 1:

[0215] Data collection

[0216] Subject: Terminal

[0217] Input: Interactions between parents and their intellectually disabled children (audio, text, video)

[0218] Specific actions: Daily conversations and activities between parents and children are recorded using a data collection application on a smartphone or tablet.

[0219] Data processing and calculation: Recorded audio data and captured video data are sent to a cloud server.

[0220] Output: Recorded audio and video data is saved on a cloud server.

[0221] Step 2:

[0222] Data Preprocessing

[0223] Subject: Server

[0224] Input: Recorded audio and video data

[0225] Specific operations: Convert the data into text (speech recognition), remove noise, and add appropriate metadata (date, time, location, emotion tags, etc.).

[0226] Data processing and computation: Convert the audio data to text using a natural language processing library (e.g., NLTK) and update the database to add metadata.

[0227] Output: Preprocessed text data and structured metadata

[0228] Step 3:

[0229] Generative AI training

[0230] Subject: Server

[0231] Input: Preprocessed text data, structured metadata

[0232] Specific operation: The preprocessed data is input into a generative AI model (e.g., OpenAI GPT-4) to learn the interaction patterns between parents and children.

[0233] Data processing and calculation: Preprocessing data is fed into the learning model in large quantities, allowing the generative AI to recognize and learn patterns, enabling the AI ​​model to provide appropriate support in place of a parent.

[0234] Output: Trained generative AI model

[0235] Step 4:

[0236] Communication with users

[0237] Subject: Terminal

[0238] Input: User question or request (voice, text)

[0239] Specific behavior: The user asks a question or speaks to the device. Example: "What shall we do today?"

[0240] Data processing and calculation: The voice recognition function converts the user's question into text data, and this text data is sent to the cloud server.

[0241] Output: The user's question text data is sent to the generation AI.

[0242] Step 5:

[0243] Response generation by generative AI

[0244] Subject: Server

[0245] Input: User question text data

[0246] Specific operation: The generation AI generates an appropriate response based on the user's question text data.

[0247] Data processing and calculation: Based on the patterns learned by the generative AI, appropriate answers are generated to the user's questions.

[0248] Output: Generated response text data

[0249] Step 6:

[0250] Providing a response

[0251] Subject: Terminal

[0252] Input: Generated response text data

[0253] Specific operation: The generated response text data is converted into speech and output to the user. Example: "How about going to the park today?"

[0254] Data processing and calculation: Convert text into audio data using a speech conversion engine (e.g., Google Text-to-Speech).

[0255] Output: A spoken response to the user

[0256] Step 7:

[0257] Abnormal behavior monitoring

[0258] Subject: Generation AI

[0259] Input: User behavior and state data (camera, microphone)

[0260] Specific behavior: The generative AI monitors the user's real-time behavior and state and detects behavior that deviates from the norm.

[0261] Data processing and calculation: Identify abnormal behavior using an abnormal behavior detection algorithm (e.g., TensorFlow).

[0262] Output: Anomaly notification data when abnormal behavior is detected

[0263] Step 8:

[0264] Sending emergency notifications

[0265] Subject: Server

[0266] Input: Error notification data

[0267] Specific behavior: When an abnormality is detected, an emergency notification will be sent to pre-defined contacts. For example, "Your child has lost consciousness. Please check the scene immediately."

[0268] Data processing and calculation: Use emergency notification APIs (e.g., Twilio API) to send notifications about abnormal behavior to pre-registered contacts.

[0269] Output: Emergency notification message

[0270] This processing flow allows users to live with peace of mind and provides a system that can respond quickly in emergencies.

[0271] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0272] This invention is a system that provides a safe and secure environment for parents of intellectually disabled children. This system records interactions between parents and their intellectually disabled children, and trains a generation AI to learn from them, enabling the generation AI to continue communicating with the children even after the parents have passed away. The generation AI also monitors the child's abnormal behavior and notifies pre-registered family members and welfare agencies in the event of an emergency. Furthermore, this invention incorporates an emotion engine that recognizes the user's emotions, allowing the generation AI to understand the user's emotional state and provide appropriate responses.

[0273] System Overview

[0274] Data collection and learning phase

[0275] 1. User initiates interaction

[0276] Device: Records conversations and actions between parents and children in audio, text, and video formats via smartphones and tablets used by family members.

[0277] Server: Receives data sent from the terminal in real time and performs preprocessing.

[0278] 2. Data Preprocessing

[0279] Server: Converts received data into text, removes noise, trims unnecessary parts, and extracts important information.

[0280] Server: Adds metadata (time, location, emotion tags, etc.) to the preprocessed data and stores it in a database.

[0281] 3. Generative AI Training

[0282] Server: The preprocessed data is input into the Generative AI's learning algorithm, which learns the interaction patterns between parents and children. This allows the Generative AI to imitate the support provided by parents.

[0283] Communication phase using generative AI

[0284] 1. Daily communication

[0285] User (child): In everyday life, the user asks questions such as "What should I do today?" to a device such as a tablet.

[0286] Terminal: Records the user's voice and converts it into text data (voice recognition function).

[0287] Terminal: Sends the converted text data to the server.

[0288] Server: The received text data is input into the generation AI, which generates an appropriate response. This response is sent back to the device and replied to the user by voice.

[0289] Examples:

[0290] User (child): "What shall we do today?"

[0291] Generative AI: "How about going to the park today?"

[0292] Emergency Notification Phase

[0293] 1. Monitoring for Abnormal Behavior

[0294] Generative AI: Monitors children's daily behavior and detects unusual behavior.

[0295] Generative AI: When it detects abnormal behavior, it sends a signal to the server that triggers an emergency notification.

[0296] 2. Sending emergency notifications

[0297] Server: Generates a message for emergency notification and sends it to pre-registered contacts (family members and welfare agencies).

[0298] Examples:

[0299] Generative AI: Detects when a child loses consciousness.

[0300] Server: Send emergency notification to family: "Your child has lost consciousness. Please check the scene immediately."

[0301] Use of emotion engine

[0302] sentiment analysis

[0303] Device: Analyzes emotions from the user's voice and text.

[0304] Server: The emotion engine detects the user's emotional state from their speaking style and tone of voice, and reflects this in the generative AI.

[0305] Generative AI: Based on detected emotional information, it generates more appropriate responses that take into account the user's psychological state.

[0306] Examples:

[0307] User (child): "My friend said something bad about me at school today."

[0308] Emotion Engine: Detects sadness from the tone and content of the user's voice.

[0309] Generating AI: "That's tough, but you're a wonderful person," it encourages.

[0310] Emotional abnormality monitoring

[0311] Generative AI: Continuously monitors the user's emotional state and sends an emergency notification if an abnormal state is detected.

[0312] Examples:

[0313] The user (child) frequently makes statements that indicate sadness or despair.

[0314] Emotion Engine: Detects emotional anomalies that exceed the normal range.

[0315] Generative AI: Detects an abnormality and the server sends an emergency notification: "There is something abnormal in the child's emotional state. Please check immediately."

[0316] The system's program records detailed interactions between parents and their intellectually disabled children, and by training the generative AI, it provides ongoing support to children even after the parents have passed away. Furthermore, by incorporating an emotion engine, it can recognize the user's emotional state and provide more appropriate support. This allows for a rapid response in emergencies, providing a safe and secure environment for children.

[0317] The processing flow will be explained below.

[0318] Data collection and learning phase

[0319] Step 1:

[0320] Users use devices such as smartphones and tablets to record and record conversations and actions between parents and children.

[0321] Step 2:

[0322] The device sends the recorded data to the server in real time.

[0323] Step 3:

[0324] The server preprocesses the received data, converting the audio data to text, removing noise, trimming unnecessary parts, and extracting important information.

[0325] Step 4:

[0326] The server adds metadata (time, location, emotion tags, etc.) to the preprocessed data and stores it in a database.

[0327] Step 5:

[0328] The server inputs the stored data into the Generative AI's learning algorithm, which learns characteristics from parent-child interactions and builds a model to generate future responses.

[0329] Communication phase using generative AI

[0330] Step 6:

[0331] In the course of daily interactions, the user (child) asks questions such as, "What shall we do today?"

[0332] Step 7:

[0333] The device records the user's voice and converts it into text data (voice recognition function).

[0334] Step 8:

[0335] The terminal transmits the converted text data to the server.

[0336] Step 9:

[0337] The server inputs the received text data into the generation AI, which then generates an appropriate response.

[0338] Step 10:

[0339] The server transmits the generated response to the terminal as text data.

[0340] Step 11:

[0341] The terminal converts the text data into speech and plays it back to the user.

[0342] Emergency Notification Phase

[0343] Step 12:

[0344] The generative AI monitors a child's daily behavior and detects any unusual behavior (e.g., suddenly collapsing).

[0345] Step 13:

[0346] When the generative AI detects abnormal behavior, it sends a signal to the server that triggers an emergency notification.

[0347] Step 14:

[0348] The server generates the content of the emergency notification and sends it to pre-registered family members and welfare agencies.

[0349] Step 15:

[0350] The user (family or welfare agency) receives a notification and promptly goes to the scene or takes the necessary action (e.g., calls 119).

[0351] Use of emotion engine

[0352] sentiment analysis

[0353] Step 16:

[0354] The device collects the user's voice and text data and sends it to the emotion engine.

[0355] Step 17:

[0356] The server uses an emotion engine to analyze the user's emotional state from the tone of voice and text, for example, identifying emotions such as joy, sadness, and anger from the tone of voice.

[0357] Step 18:

[0358] The generative AI generates an appropriate response based on the emotional information obtained from the emotion engine.

[0359] Examples:

[0360] User (child): "My friend said something bad about me at school today."

[0361] Emotion Engine: Detects sadness from the tone and content of the user's voice.

[0362] Generating AI: "That's tough, but you're a wonderful person," it encourages.

[0363] Emotional abnormality monitoring

[0364] Step 19:

[0365] The generative AI monitors the user's emotional state in real time, and if negative emotions are detected repeatedly, it will be deemed an abnormality.

[0366] Step 20:

[0367] When the generative AI detects an emotional anomaly, it sends a signal to the server that triggers an emergency notification.

[0368] Step 21:

[0369] The server generates emergency notifications about emotional abnormalities and sends them to pre-registered family members and welfare agencies.

[0370] Examples:

[0371] The user (child) frequently makes statements that express sadness and despair.

[0372] Emotion Engine: Detects abnormal emotional states that exceed the normal range.

[0373] Generative AI: Detects an abnormality and the server sends an emergency notification: "There is something abnormal in the child's emotional state. Please check immediately."

[0374] This system records detailed daily interactions between parents and their intellectually disabled children, and uses this information to train a generative AI, providing ongoing support for the child even after the parents have passed away. It also incorporates an emotion engine to recognize the user's emotional state and provide more appropriate support. It can respond quickly in emergencies, providing a safe and secure environment for children.

[0375] Example 2

[0376] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0377] There is a need to provide an environment where children with intellectual disabilities can live safely even after their parents pass away. However, conventional systems have difficulty completely replacing the role of parents, and are particularly limited in terms of communication between parents and emergency response. Furthermore, they lack the ability to appropriately understand and respond to children's emotional states, making it difficult to maintain a sense of psychological security for children.

[0378] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0379] In this invention, the server includes a means for recording interactions between a parent and their intellectually disabled child, a means for preprocessing the recorded data and training the generation AI, a means for the generation AI to communicate with the child after the parent's death, a means for the generation AI to monitor the child's behavior and detect abnormalities, a means for sending an emergency notification to contacts when an abnormality is detected, a means for analyzing the user's emotions, and a means for the generation AI to generate an appropriate response based on the analyzed emotional information. This allows the child to receive continuous support even after parental support is no longer available, and allows for rapid response in emergencies. Furthermore, the emotion engine can be used to understand the child's emotional state and provide appropriate psychological support.

[0380] "Recording means" means any device or software that collects and stores in audio, text, and video form communications between a parent and their child with an intellectual disability.

[0381] "Preprocessing means" refers to a device or program that removes noise from recorded data, converts it into text, and otherwise converts it into a format suitable for learning by generative AI.

[0382] "Generative AI" refers to an artificial intelligence model that can learn from input data and generate natural, human-like responses.

[0383] "Means of communication" refers to the device or software that enables the generative AI to generate appropriate responses to questions posed by the child and return them to the child in voice or text format.

[0384] "Behavioral monitoring means" refers to a device or program that continuously records a child's daily behavior and detects deviations from normal patterns.

[0385] "Means for detecting anomalies" refers to devices or software that identify anomalous patterns of behavior from monitored behavioral data and issue an alert if such patterns are detected.

[0386] "Means for sending emergency notifications" refers to a device or program that sends emergency notifications via SMS, email, etc. to pre-registered contacts (family members or welfare agencies) when an abnormality is detected.

[0387] "Means for analyzing emotions" refers to a device or software that analyzes the emotional state of a user from their voice or text and reflects the results in the generative AI's response.

[0388] "Means for generating an appropriate response" refers to a device or program that enables the generation AI to create a response that takes into account the user's psychological state based on analyzed emotional information and user input.

[0389] This invention is a system that records interactions between parents and intellectually disabled children and trains a generative AI to provide ongoing support for children even after their parents pass away. It also has the ability to monitor abnormal behavior and send prompt notifications in the event of an emergency. It can also analyze the user's emotions and provide appropriate responses based on that information.

[0390] System configuration

[0391] Data collection and learning phase

[0392] First, the user uses a device such as a smartphone or tablet to record conversations and behaviors between parents and their children in audio, text, and video formats. Recordings are made using the default recording or camera app. The recorded data is then sent in real time to a cloud server (e.g., AWS S3).

[0393] The server then preprocesses the received data. Audio data is converted to text using the Google Cloud Speech-to-Text API, and noise is removed using NLTK (Natural Language Toolkit). Specific frames of the video data are extracted using FFmpeg and saved as images. Metadata (e.g., recording date and time, recording location, emotion tags) is added to the preprocessed data, and it is then stored in a NoSQL database such as MongoDB.

[0394] The stored data is then fed into a generative AI (such as OpenAI's GPT-3) to learn the interaction patterns between parents and children. In the process, the generative AI will be able to mimic the support provided by parents.

[0395] Communication phase using generative AI

[0396] Users (children) can ask a device such as a tablet, "What shall we do today?" in their daily lives. The question is recorded on the device and converted into text data using the Google Cloud Speech-to-Text API. The converted text data is sent to a server and input into the generation AI.

[0397] The server generates an appropriate response using the AI ​​and sends it back to the device. The user is then told a voice response, using Amazon Polly for speech synthesis.

[0398] Examples:

[0399] User (child): "What shall we do today?"

[0400] Generative AI: "How about going to the park today?"

[0401] Emergency Notification Phase

[0402] The AI ​​monitors the child's daily activities and detects any unusual behavior. For example, if a child stays in the same position for a long time, it is detected as abnormal behavior. When abnormal behavior is detected, the AI ​​sends a signal to the server that triggers an emergency notification.

[0403] The server sends emergency notifications via SMS or email to pre-registered contacts (family members or welfare agencies) using services such as AWS SNS (Simple Notification Service).

[0404] Examples:

[0405] Generative AI: Detects when a child loses consciousness.

[0406] Server: Send emergency notification to family: "Your child has lost consciousness. Please check the scene immediately."

[0407] Use of emotion engine

[0408] The user's voice and text are analyzed for emotional state using IBM Watson Tone Analyzer. The server uses an emotion engine to detect emotional information from the user's speaking style and tone of writing, and reflects that information in the generation AI. Based on the detected emotional information, the generation AI generates a more appropriate response that takes the user's psychological state into consideration.

[0409] Examples:

[0410] User (child): "My friend said something bad about me at school today."

[0411] Emotion Engine: Detects sadness from the tone and content of the user's voice.

[0412] Generating AI: "That's tough, but you're a wonderful person," it encourages.

[0413] The emotion engine also continuously monitors the user's emotional state and provides emergency notifications if it detects an abnormal state.

[0414] Examples:

[0415] The user (child) frequently makes statements that indicate sadness or despair.

[0416] Emotion Engine: Detects emotional anomalies that exceed the normal range.

[0417] Generative AI: An abnormality is detected and the server sends an emergency notification: "There is something abnormal in the child's emotional state. Please check immediately."

[0418] As described above, this system records detailed interactions between parents and their intellectually disabled children and trains the generative AI to provide ongoing support for the child even after the parents have passed away. Furthermore, by combining it with an emotion engine, it can recognize the user's emotional state and provide more appropriate support. This allows for a rapid response in emergencies, providing an environment where children can live in peace of mind.

[0419] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0420] Program processing flow

[0421] Data collection and learning phase

[0422] Step 1: User initiates interaction

[0423] Device: A smartphone or tablet is used to record conversations and actions between parents and children in audio, text, and video formats. The input is the conversation or action, and the output is an audio file, text file, or video file. Specifically, audio recording is done using a recording app, and video recording is done using a camera app.

[0424] Step 2: Preprocessing the data

[0425] Server: Receives data sent from the device in real time and performs preprocessing. The inputs are audio files, text files, and video files, and the output is processed text data and image data. Specifically, the Google Cloud Speech-to-Text API is used to convert audio data into text, and NLTK is used to remove noise. FFmpeg is used to extract specific frames from video data and save them as image files.

[0426] Step 3: Training the generative AI

[0427] Server: The preprocessed data is given metadata (recording date and time, recording location, emotion tag) and stored in a database (e.g., MongoDB). The input is the preprocessed data, and the output is a database entry with the metadata. The stored data is fed into a generative AI (e.g., OpenAI's GPT-3), which learns the interaction patterns between parents and children. Through this learning, the generative AI is able to imitate the support provided by the parent.

[0428] Communication phase using generative AI

[0429] Step 4: Daily Communication

[0430] User (child): Asks a device such as a tablet, "What shall we do today?" The input is the user's voice, and the output is a recorded audio file.

[0431] Terminal: Convert the recorded audio into text data using the Google Cloud Speech-to-Text API. The input is an audio file and the output is text data.

[0432] Terminal: Transmits converted text data to the server. The input is text data, and the output is data transmission to the server.

[0433] Server: Inputs the received text data into the generation AI and generates an appropriate response. The input is text data, and the output is the generated response text.

[0434] Server: Sends the generated response to the device and replies to the user via speech synthesis (using Amazon Polly). The input is the generated response text and the output is the audio response.

[0435] Examples:

[0436] User (child): "What shall we do today?"

[0437] Generative AI: "How about going to the park today?"

[0438] Emergency Notification Phase

[0439] Step 5: Monitor for abnormal behavior

[0440] Generative AI: Monitors a child's daily behavior and detects any unusual behavior. The input is daily behavior data, and the output is an alert for abnormal behavior detection. For example, it monitors long periods of inactivity or abnormal behavior patterns.

[0441] Step 6: Send emergency notifications

[0442] Server: When abnormal behavior is detected, it generates an emergency notification and sends it to pre-registered contacts (family members or welfare agencies). The input is the abnormal behavior detection alert, and the output is an emergency notification message to the contacts. The notification is sent using AWS SNS (Simple Notification Service).

[0443] Examples:

[0444] Generative AI: Detects when a child loses consciousness.

[0445] Server: Send emergency notification to family: "Your child has lost consciousness. Please check the scene immediately."

[0446] Use of emotion engine

[0447] Step 7: Sentiment analysis

[0448] Terminal: Analyzes emotions from user voice and text using IBM Watson Tone Analyzer. The input is an audio file or text data, and the output is analyzed emotional information.

[0449] Server: Using the emotion engine, the server detects the user's emotional state from their speaking style and tone of voice, and then feeds that information into the generative AI. The input is analyzed emotional information, and the output is text data with emotion tags.

[0450] Step 8: Generate an appropriate response

[0451] Generative AI: Generates more appropriate responses based on detected emotional information. The input is emotion-tagged text data, and the output is an emotion-sensitive response.

[0452] Examples:

[0453] User (child): "My friend said something bad about me at school today."

[0454] Emotion Engine: Detects sadness from the tone and content of the user's voice.

[0455] Generating AI: "That's tough, but you're a wonderful person," it encourages.

[0456] Step 9: Monitor for emotional abnormalities

[0457] Generative AI: Continuously monitors the user's emotional state and sends an emergency notification if an abnormality is detected. The input is emotional state data, and the output is an alert for anomaly detection. A specific example is when a user frequently makes statements that express sadness or despair.

[0458] Server: Detects anomalies and sends emergency notifications. The input is an anomaly detection alert, and the output is an emergency notification message.

[0459] Examples:

[0460] Emotion Engine: Detects emotional anomalies that exceed the normal range.

[0461] Generative AI: An abnormality is detected and the server sends an emergency notification: "There is something abnormal in the child's emotional state. Please check immediately."

[0462] (Application example 2)

[0463] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0464] Conventional monitoring systems are not specialized for people with intellectual disabilities or the elderly, and may not respond appropriately in emergencies. Furthermore, there is a lack of systems that can detect changes in a user's emotional state in real time and provide appropriate responses. Therefore, new technology is needed to balance user safety and mental care.

[0465] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0466] In this invention, the server includes: means for recording interactions between a parent and their intellectually disabled child; means for preprocessing the recorded data and training a generation AI; means for the generation AI to communicate with the child after the parent has passed away; means for the generation AI to monitor the child's abnormal behavior; means for issuing an emergency notification if an abnormality is detected; means for recognizing the user's voice and converting it into text; means for analyzing the emotions of the converted text data using an emotion engine; means for generating an appropriate response based on the analyzed emotional data; means for replying to the user with the generated response by voice; and means for detecting an abnormal emotional state through emotion analysis and issuing an emergency notification. This enables both daily safety management and psychological care for the user.

[0467] An "interaction recording device" is any device or method that records, in audio, text, or video format, the conversations or activities between a parent and their child with an intellectual disability.

[0468] "Preprocessing" refers to the process of converting recorded data into text, removing noise, and extracting important information.

[0469] "Generative AI" is an artificial intelligence that uses machine learning models to mimic the interactions between parents and their intellectually disabled children, enabling continuous communication.

[0470] "Means of communication" refers to the ability to use generative AI to converse and interact with children.

[0471] The "means of monitoring abnormal behavior" is a function that allows the generating AI to monitor a child's daily behavior and detect any behavior that is out of the ordinary.

[0472] The "means for sending emergency notifications" is a function that sends an emergency message to pre-registered family members and welfare organizations when an abnormality is detected.

[0473] "Means for recognizing voice and converting it into text" refers to technology that analyzes the voice spoken by the user and converts it into text data.

[0474] An "emotion engine" is an artificial intelligence model that analyzes emotions from a user's voice and text.

[0475] "Means for generating an appropriate response" refers to the function by which the generative AI creates an appropriate response or reply based on the analyzed emotional data.

[0476] "Means for replying by voice" refers to a technology that uses voice synthesis technology to convey the generated response to the user by voice.

[0477] "Means for detecting emotional state" refers to the function by which the emotion engine detects changes in the user's emotions in real time and provides that information to the generation AI.

[0478] The present invention is a system that records interactions between parents and their intellectually disabled children, formats the data through preprocessing, trains a generating AI, and then enables the generating AI to communicate with the child even after the parents have passed away, monitors abnormal behavior or emotional abnormalities, and sends emergency notifications. Detailed embodiments of the system are described below.

[0479] In this invention, the terminal can be a smartphone, tablet, intercom, etc., which can be used to communicate with the user and record voice. The server is installed on the cloud, where the recorded data is stored, preprocessed, and the generative AI model is trained.

[0480] Hardware and software used

[0481] Hardware: Smartphone, tablet, intercom, computer with microphone

[0482] Software: Python, SpeechRecognition, Transformers (Huggingface), Twilio API

[0483] Program Overview and Processing

[0484] 1. Recording interactions

[0485] The device records conversations and behaviors between parents and their children with intellectual disabilities in audio, text, or video format, and the recorded data is sent to a server in real time.

[0486] 2. Data Preprocessing

[0487] The server converts the received data into text, removes noise, and extracts important information. The preprocessed data is then annotated with metadata (such as time, location, and emotion tags) and stored in a database.

[0488] 3. Generative AI Training

[0489] The server inputs the preprocessed data into the generative AI's learning algorithm, which learns the interaction patterns between parents and children. The generative AI is then able to mimic the support provided by the parent and generate appropriate responses.

[0490] 4. Daily communication

[0491] The user (child) asks questions to the device in their daily life. The device records the user's voice and converts it into text data. This data is sent to the server, where the AI ​​generates an appropriate response and sends it to the device, which then replies to the user by voice.

[0492] Example: User (child): "What shall we do today?"

[0493] Generative AI: "How about going to the park today?"

[0494] 5. Emotion Recognition and Analysis

[0495] The device analyzes the user's emotions from their voice and text, and the server uses an emotion engine to detect the user's emotional state and reflects this in the generating AI.

[0496] Example: User (child): "My friend said something bad about me at school today."

[0497] Emotion Engine: Detects sadness from the tone and content of the user's voice.

[0498] Generator: "That's tough, but you're a wonderful being."

[0499] 6. Monitoring for abnormal behavior and emotional abnormalities

[0500] The AI ​​monitors the child's daily behavior and detects abnormal behavior. It also continuously monitors the user's emotional state and sends an emergency notification if it detects an abnormal state.

[0501] Example: Generative AI detects when a child loses consciousness.

[0502] Server: Send emergency notification to family: "Your child has lost consciousness. Please check the scene immediately."

[0503] Example prompt: "I'm not feeling too great today." Please assess the emotion in this text. Please advise how to respond on behalf of the parent.

[0504] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0505] Step 1:

[0506] The user speaks to the device, which records the user's voice.

[0507] Input: User's voice

[0508] Output: Recorded audio data

[0509] What it does: Uses the device's microphone to record what the user says.

[0510] Step 2:

[0511] The device converts recorded audio into text.

[0512] Input: Recorded audio data

[0513] Output: Text data

[0514] What it does: Uses speech recognition software (e.g., Google Speech Recognition API) to convert voice data into text.

[0515] Step 3:

[0516] The terminal transmits the text data to the server.

[0517] Input: Text data

[0518] Output: Text data sent to the server

[0519] Operation: Uses the device's communication function to send text data to a server in the cloud.

[0520] Step 4:

[0521] The server preprocesses the received text data, removing noise and extracting important information.

[0522] Input: Text data sent to the server

[0523] Output: Preprocessed text data

[0524] Operation: The server performs noise removal and important information extraction on the text data, and then adds metadata.

[0525] Step 5:

[0526] The server inputs the preprocessed text data into the generation AI, which generates an appropriate response.

[0527] Input: Preprocessed text data

[0528] Output: Response by the generative AI

[0529] How it works: The server inputs text data into the generative AI model, which generates appropriate responses based on the parent's response patterns learned by the model.

[0530] Step 6:

[0531] The server generates a response and sends it to the terminal.

[0532] Input: Response by generative AI

[0533] Output: The response sent to the device

[0534] Operation: The server sends the response generated by the generation AI to the terminal via its communication function.

[0535] Step 7:

[0536] The terminal returns the response received from the server to the user by voice.

[0537] Input: The response sent to the terminal

[0538] Output: A spoken response to the user

[0539] How it works: The device uses speech synthesis software to convert the text response received from the server into audio and play it back to the user.

[0540] Step 8:

[0541] The terminal transmits the text data to an emotion engine to analyze the user's emotions.

[0542] Input: Text data

[0543] Output: Emotion analysis results

[0544] How it works: The device uses an emotion engine (e.g., a Transformers emotion analysis model) to analyze the emotional state of text data.

[0545] Step 9:

[0546] The server detects emotional abnormalities and sends an emergency notification if any abnormalities are detected.

[0547] Input: Sentiment analysis results

[0548] Output: Send emergency notification

[0549] How it works: The server analyzes the emotion analysis results, and if it detects an abnormal emotional state, it sends an emergency notification to pre-registered contacts.

[0550] Step 10:

[0551] Emergency notifications reach registered contacts.

[0552] Input: Send emergency notification

[0553] Output: Contact notification

[0554] What it does: Your server uses a communication service, such as the Twilio API, to send an emergency message to registered contacts.

[0555] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0556] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0557] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0558] [Second embodiment]

[0559] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0560] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0561] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0562] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0563] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0564] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0565] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0566] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0567] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0568] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0569] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0570] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0571] This invention is a system that aims to provide an environment where parents of intellectually disabled children can live with peace of mind. This system records interactions between parents and their intellectually disabled children and trains a generative AI to learn from this data, allowing the generative AI to continue communicating with the child even after the parent has passed away. It also includes a function to monitor abnormal behavior and notify pre-registered family members and welfare organizations in the event of an emergency.

[0572] System Overview

[0573] Data collection and learning phase

[0574] 1. User initiates interaction

[0575] Devices: Family members' smartphones and tablets record daily interactions between parents and children in audio, text, and video formats.

[0576] Server: Receives data sent from the terminal and performs preprocessing.

[0577] 2. Data Preprocessing

[0578] Server: Converts received data into text, removes noise and unnecessary information, and structures the data by adding metadata such as time, location, and emotion tags.

[0579] 3. Generative AI Training

[0580] Server: The preprocessed data is input into the generative AI's learning model to learn the interaction patterns between parents and children, allowing the generative AI to mimic the support provided by parents.

[0581] Communication phase using generative AI

[0582] 1. Daily communication

[0583] User (child): Asks questions such as "What should we do today?" to a device such as a tablet.

[0584] Terminal: Converts this question into text data and sends it to the server.

[0585] Server: Inputs text data into the AI ​​generator and generates an appropriate response. This response is then sent back to the device and spoken back to the child.

[0586] Examples:

[0587] User (child): "What shall we do today?"

[0588] Generative AI: "How about going to the park today?"

[0589] Emergency Notification Phase

[0590] 1. Monitoring for Abnormal Behavior

[0591] Generative AI: Constantly monitors the child's behavior and condition, and if it detects behavior that deviates from the norm, it determines that it is abnormal.

[0592] Trigger emergency notifications if an anomaly is detected.

[0593] 2. Sending emergency notifications

[0594] Server: Generates emergency notification messages and sends them to pre-defined contacts (family members and welfare agencies).

[0595] Examples:

[0596] Generative AI: Detects when a child loses consciousness.

[0597] Server: Send emergency notification to family: "Your child has lost consciousness. Please check the scene immediately."

[0598] Example

[0599] In a concrete example, the process works as follows:

[0600] 1. User (child): Communicate regularly with the generated AI in daily life.

[0601] 2. Generative AI: Acts as a parent, providing appropriate advice and instructions.

[0602] 3. In case of abnormality: The generated AI detects abnormal behavior of the child and the server sends an emergency notification.

[0603] 4. Who to notify: Families and welfare agencies that receive the notification will respond promptly.

[0604] In this way, this invention is a system that provides an environment where children with intellectual disabilities can live safely even after their parents pass away. The data processing and generative AI functions of each phase make continuous support possible.

[0605] The processing flow will be explained below.

[0606] Data collection and learning phase

[0607] Step 1:

[0608] Users use devices such as smartphones and tablets to record and record conversations and actions between parents and children.

[0609] Step 2:

[0610] The device sends the recorded data to the server in real time.

[0611] Step 3:

[0612] The server preprocesses the received data, converting the audio data to text, removing noise, trimming unnecessary parts, and extracting important information.

[0613] Step 4:

[0614] The server adds metadata (time, location, emotion tags, etc.) to the preprocessed data and stores it in a database.

[0615] Step 5:

[0616] The server inputs the stored data into the Generative AI's learning algorithm, which learns characteristics from parent-child interactions and builds a model to generate future responses.

[0617] Communication phase using generative AI

[0618] Step 6:

[0619] In the course of daily interactions, the user (child) asks questions such as, "What shall we do today?"

[0620] Step 7:

[0621] The device records the user's voice and converts it into text data (voice recognition function).

[0622] Step 8:

[0623] The terminal transmits the converted text data to the server.

[0624] Step 9:

[0625] The server inputs the received text data into the generation AI, which then generates an appropriate response.

[0626] Step 10:

[0627] The server transmits the generated response to the terminal as text data.

[0628] Step 11:

[0629] The terminal converts the text data into speech and plays it back to the user.

[0630] Emergency Notification Phase

[0631] Step 12:

[0632] The generative AI monitors a child's daily behavior and detects any unusual behavior (e.g., suddenly collapsing).

[0633] Step 13:

[0634] When the generative AI detects abnormal behavior, it sends a signal to the server that triggers an emergency notification.

[0635] Step 14:

[0636] The server generates the content of the emergency notification and sends it to pre-registered family members and welfare agencies.

[0637] Step 15:

[0638] The user (family or welfare agency) receives a notification and promptly goes to the scene or takes the necessary action (e.g., calls 119).

[0639] The system's program records detailed daily interactions between parents and their intellectually disabled children, and trains the AI ​​to learn from these interactions, providing ongoing support for the child even after the parents have passed away. It also has a built-in notification system for rapid response in emergencies.

[0640] Example 1

[0641] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0642] There is a lack of support for intellectually disabled children to live safely after their parents pass away. There is no way to continue communicating with their children, monitoring their behavior, or responding in the event of an emergency, as parents did while they were alive, which raises the issue of the possibility that the safety and quality of life of the children may decline.

[0643] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0644] In this invention, the server includes a device that records interactions between parents and their intellectually disabled children, a device that preprocesses the recorded data and trains a generative AI model, a device that allows the generative AI model to communicate with the child after the parents pass away, a device that allows the generative AI model to monitor abnormal behavior in the child, and a device that issues an emergency notification if an abnormality is detected. This enables intellectually disabled children to continue living in a safe and stable environment even after the parents pass away.

[0645] "Devices that record interactions between parents and children with intellectual disabilities" means devices that collect and retain audio, text, and video communications between parents and children with intellectual disabilities.

[0646] The "device that preprocesses recorded data and trains a generative AI model" is a system that performs preprocessing such as noise removal, text conversion, and metadata addition based on recorded interaction data, and then uses that data to train a generative AI model.

[0647] The "device that allows a generative AI model to communicate with children after their parents have passed away" is a system that allows a generative AI model to continuously interact with and support children with intellectual disabilities based on data learned, even when their parents are no longer present.

[0648] The "device in which a generative AI model monitors children's abnormal behavior" is a system in which a generative AI model constantly monitors children's behavior and judges it as abnormal if it deviates from normal behavior patterns.

[0649] The "device that sends emergency notifications when an abnormality is detected" is a system that sends emergency notifications to pre-registered contacts when the AI ​​model detects abnormal behavior in a child.

[0650] "Means for storing preprocessed data in an information storage device and updating the configuration of the generative AI model" refers to a technology for restoring preprocessed data to a database or storage device and using that data to update the algorithms and parameters of the generative AI model.

[0651] "Means for sending emergency notifications to pre-registered family members and welfare agencies" refers to a system that sends emergency messages to pre-designated family members and welfare agencies when a child exhibits abnormal behavior.

[0652] This invention is a system that provides a safe and secure environment for parents of intellectually disabled children. The system records interactions between parents and their children and trains a generative AI model to learn from this data, enabling the model to continue communicating with the child after the parent's death. The system also includes a function in which the generative AI model monitors for abnormal behavior and notifies pre-registered family members and welfare agencies in the event of an emergency.

[0653] Data collection and learning phase

[0654] First, users (parents and children) use devices such as smartphones and tablets to carry out everyday interactions. These interactions are recorded in audio, text, and video formats. For example, a parent might say, "Good morning, how was your day?", and a child might respond, "I drew a picture at school today."

[0655] The device then sends the recorded data to a server, which converts the received data into text using voice recognition technology (e.g., Google Speech-to-Text API), removes noise, filters out unnecessary information, and extracts only the important information. The server also structures the data by adding metadata such as time, location, and emotion tags.

[0656] The server then feeds the pre-processed data into a generative AI model to learn parent-child interaction patterns, allowing the model to mimic the support provided by the parent.

[0657] Communication phase using generative AI

[0658] In everyday communication, a user (child) will ask a device such as a tablet, "What shall we do today?" This question is converted into text data using speech recognition technology and sent to a server. The server inputs the text data into a generative AI model to generate an appropriate response. This response is sent back to the device and is returned to the child as a voice using speech synthesis technology (e.g., Google Text-to-Speech API).

[0659] For example, if a child asks, "What shall we do today?", the generative AI model might suggest, "How about going to the park today?"

[0660] Emergency Notification Phase

[0661] The system's generative AI model constantly monitors a child's behavior and state, and if it detects behavior that deviates from the norm, it will determine that something is abnormal, such as if the child loses consciousness or exhibits unusual behavioral patterns.

[0662] If an abnormality is detected, the server generates an emergency notification message and sends it to pre-defined contacts (family members or welfare agencies). For example, it could send a message via SMS or email saying, "Your child has lost consciousness. Please check the scene immediately."

[0663] In this way, the present invention is a system that realizes functions such as recording interactions with parents, learning using a generative AI model, communication, monitoring abnormal behavior, and emergency notifications, in order to provide an environment where intellectually disabled children can live safely even after their parents have passed away.

[0664] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0665] Step 1:

[0666] User initiates interaction

[0667] Input: Daily interactions between users (parents and children) (voice, text, video)

[0668] Action: Parent and child engage in everyday interactions, such as parent saying, "Good morning, how was your day?" and child responding, "I drew a picture at school today."

[0669] Output: Interaction data is recorded on a smartphone or tablet

[0670] Step 2:

[0671] Sending data

[0672] Input: Recorded interaction data (audio, text, video)

[0673] How it works: The device sends the recorded data to a server. For example, audio data recorded using a tablet's microphone is uploaded to a cloud server via Wi-Fi.

[0674] Output: The server receives the data

[0675] Step 3:

[0676] Data Preprocessing

[0677] Input: Data of the interaction received by the server (audio, text, video)

[0678] How it works: The server uses speech recognition technology (e.g., Google Speech-to-Text API) to convert voice data into text, remove noise, and filter out unnecessary information. It also structures the data by adding metadata such as time, location, and emotion tags. For example, voice data such as "Good morning, how was your day?" is converted to text like "Good morning, how was your day?", and metadata such as October 10, 2023, and the living room are added.

[0679] Output: Preprocessed text data

[0680] Step 4:

[0681] Training generative AI models

[0682] Input: Preprocessed text data

[0683] How it works: The server feeds the preprocessed data into a generative AI model that learns parent-child interaction patterns, such as what response a parent gets when they say "good morning."

[0684] Output: A trained generative AI model

[0685] Step 5:

[0686] Daily communication

[0687] Input: Questions (voice) asked by the user (child) to the device

[0688] How it works: The user (child) asks a tablet or other device, "What shall we do today?" This question is converted into text data using speech recognition technology and sent to a server. The server inputs the text data into a generative AI model and generates an appropriate response. This response is sent back to the device, and is replies to the child as audio using speech synthesis technology (e.g., Google Text-to-Speech API).

[0689] How it works: When a child asks, "What shall we do today?", the generative AI model suggests, "How about going to the park today?"

[0690] Output: Voice response

[0691] Step 6:

[0692] Abnormal behavior monitoring

[0693] Input: User (child) behavior data

[0694] How it works: The generative AI model constantly monitors the child's behavior and state, and if it detects any behavior that deviates from the norm, it will judge it to be abnormal. For example, if the child suddenly stops moving or exhibits an unusual behavior pattern, it will detect an abnormality.

[0695] Output: Anomaly detection alert

[0696] Step 7:

[0697] Sending emergency notifications

[0698] Input: Anomaly detection alert, pre-registered contact information

[0699] What it does: The server generates an emergency notification message and sends it to pre-defined contacts (family or welfare agencies), for example, an SMS or email message saying "Your child has lost consciousness. Please check the scene immediately."

[0700] Output: Send emergency notification message

[0701] (Application example 1)

[0702] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0703] There is a need to reduce the anxiety that users who particularly need supervision, such as people with intellectual disabilities and the elderly, experience in their daily lives. It is important to provide an environment where users can live safely even when their parents or caregivers are not always nearby. There is also a need for a system that can respond quickly in the event of abnormal behavior or an emergency. Furthermore, there is a need to ensure that users do not feel lonely even after their parents pass away, and that they continue to receive the necessary support.

[0704] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0705] In this invention, the server includes a means for recording interactions between parents and persons with intellectual disabilities, a means for preprocessing the recorded data and having the generation AI learn from it, a means for the generation AI to communicate with the user after the parent has passed away, a means for the generation AI to monitor the user's abnormal behavior, a means for sending an emergency notification when an abnormality is detected, a means for monitoring the user's behavior and status using the camera and microphone of the smart device, a means for the generation AI to communicate with the user in real time, and a means for sending a notification to pre-set contacts when abnormal behavior is detected. This provides an environment where users can live safely and enables rapid response in emergencies.

[0706] A "means for recording interactions between a parent and a child with an intellectual disability" is a device or method for recording, by audio, text, video, or other means, the conversations or activities between a parent and a child with an intellectual disability.

[0707] "Means for preprocessing recorded data and training a generative AI" refers to a device or method for converting recorded data into text, performing preprocessing such as noise removal and adding metadata, and then training the generative AI model.

[0708] "Means for the generative AI to communicate with the user after the parents have passed away" refers to a device or method that allows the generative AI to interact with the user on behalf of the parents and provide necessary support and advice.

[0709] "Means for monitoring abnormal user behavior by a generation AI" refers to a device or method by which a generation AI monitors a user's daily behavior and detects abnormal behavior.

[0710] The "means for sending an emergency notification when an abnormality is detected" refers to a device or method for sending an emergency notification to pre-defined contacts when abnormal user behavior is detected.

[0711] "Means for monitoring a user's behavior and status using a camera or microphone on a smart device" refers to a device or method for recording and monitoring a user's behavior and status using a camera and microphone on a device such as a smartphone or head-mounted display.

[0712] "Means for the generation AI to communicate with the user in real time" refers to a device or method that enables the generation AI to respond immediately to the user's questions or requests and realize real-time dialogue.

[0713] "Means for sending notifications to pre-defined contacts when abnormal behavior is detected" refers to a device or method for sending notifications to pre-defined emergency contacts (such as family members or welfare agencies) when abnormal behavior of a user is detected.

[0714] The embodiments for implementing the invention are carried out in the following methods and steps.

[0715] Hardware and Software Use

[0716] 1. Hardware

[0717] Mobile devices such as smartphones and tablets

[0718] head-mounted display

[0719] Cloud server (e.g. AWS, Google Cloud)

[0720] Camera and microphone

[0721] 2. Software

[0722] Data collection applications (e.g., applications developed with Flutter)

[0723] Data preprocessing script (e.g., Python)

[0724] Natural language processing libraries (e.g. NLTK)

[0725] Generative AI models (e.g., OpenAI GPT-4)

[0726] Emergency notification API (e.g. Twilio API)

[0727] Anomaly detection algorithms (e.g., TensorFlow)

[0728] Data collection

[0729] The device (smartphone or tablet) records the interactions between parents and their children with intellectual disabilities in audio, text, and video formats, and the recorded data is sent to a cloud server where it is stored.

[0730] Data Preprocessing

[0731] The server converts the transmitted data into text, removes noise, and structures the data by adding metadata such as time, location, and emotion tags. This preprocessed data is then stored in a database.

[0732] Generative AI training

[0733] The server inputs the preprocessed data into the generative AI's learning model, learning the interaction patterns between parents and children, allowing the generative AI to provide appropriate support as a substitute parent.

[0734] Communication with users

[0735] The user (child) asks everyday questions or requests to the device, such as "What shall we do today?"

[0736] Examples:

[0737] User: "What should I do today?"

[0738] Generative AI: "How about going to the park today?"

[0739] The server converts the user's question into text data, inputs it into a generation AI, and generates an appropriate response. This response is then sent back to the device and replies to the user via voice.

[0740] Abnormal behavior monitoring and emergency notification

[0741] The AI ​​constantly monitors the user's behavior and status, and if it detects any behavior that deviates from the norm, it judges it to be an anomaly. If an anomaly is detected, the server will send an emergency notification to pre-defined contacts.

[0742] Examples:

[0743] Generative AI: Detects when the user has lost consciousness.

[0744] Server: Send emergency notification to family: "Your child has lost consciousness. Please check the scene immediately."

[0745] Prompt Sentence Examples

[0746] An example of a prompt sentence might be:

[0747] "Design a system that constantly monitors the user's mood and state and sends emergency notifications at the appropriate time."

[0748] "How can I automatically notify family members or designated contacts when abnormal behavior is detected?"

[0749] This system allows users to live with peace of mind and enables quick response in emergencies.

[0750] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0751] Step 1:

[0752] Data collection

[0753] Subject: Terminal

[0754] Input: Interactions between parents and their intellectually disabled children (audio, text, video)

[0755] Specific actions: Daily conversations and activities between parents and children are recorded using a data collection application on a smartphone or tablet.

[0756] Data processing and calculation: Recorded audio data and captured video data are sent to a cloud server.

[0757] Output: Recorded audio and video data is saved on a cloud server.

[0758] Step 2:

[0759] Data Preprocessing

[0760] Subject: Server

[0761] Input: Recorded audio and video data

[0762] Specific operations: Convert the data into text (speech recognition), remove noise, and add appropriate metadata (date, time, location, emotion tags, etc.).

[0763] Data processing and computation: Convert the audio data to text using a natural language processing library (e.g., NLTK) and update the database to add metadata.

[0764] Output: Preprocessed text data and structured metadata

[0765] Step 3:

[0766] Generative AI training

[0767] Subject: Server

[0768] Input: Preprocessed text data, structured metadata

[0769] Specific operation: The preprocessed data is input into a generative AI model (e.g., OpenAI GPT-4) to learn the interaction patterns between parents and children.

[0770] Data processing and calculation: Preprocessing data is fed into the learning model in large quantities, allowing the generative AI to recognize and learn patterns, enabling the AI ​​model to provide appropriate support in place of a parent.

[0771] Output: Trained generative AI model

[0772] Step 4:

[0773] Communication with users

[0774] Subject: Terminal

[0775] Input: User question or request (voice, text)

[0776] Specific behavior: The user asks a question or speaks to the device. Example: "What shall we do today?"

[0777] Data processing and calculation: The voice recognition function converts the user's question into text data, and this text data is sent to the cloud server.

[0778] Output: The user's question text data is sent to the generation AI.

[0779] Step 5:

[0780] Response generation by generative AI

[0781] Subject: Server

[0782] Input: User question text data

[0783] Specific operation: The generation AI generates an appropriate response based on the user's question text data.

[0784] Data processing and calculation: Based on the patterns learned by the generative AI, appropriate answers are generated to the user's questions.

[0785] Output: Generated response text data

[0786] Step 6:

[0787] Providing a response

[0788] Subject: Terminal

[0789] Input: Generated response text data

[0790] Specific operation: The generated response text data is converted into speech and output to the user. Example: "How about going to the park today?"

[0791] Data processing and calculation: Convert text into audio data using a speech conversion engine (e.g., Google Text-to-Speech).

[0792] Output: A spoken response to the user

[0793] Step 7:

[0794] Abnormal behavior monitoring

[0795] Subject: Generation AI

[0796] Input: User behavior and state data (camera, microphone)

[0797] Specific behavior: The generative AI monitors the user's real-time behavior and state and detects behavior that deviates from the norm.

[0798] Data processing and calculation: Identify abnormal behavior using an abnormal behavior detection algorithm (e.g., TensorFlow).

[0799] Output: Anomaly notification data when abnormal behavior is detected

[0800] Step 8:

[0801] Sending emergency notifications

[0802] Subject: Server

[0803] Input: Error notification data

[0804] Specific behavior: When an abnormality is detected, an emergency notification will be sent to pre-defined contacts. For example, "Your child has lost consciousness. Please check the scene immediately."

[0805] Data processing and calculation: Use emergency notification APIs (e.g., Twilio API) to send notifications about abnormal behavior to pre-registered contacts.

[0806] Output: Emergency notification message

[0807] This processing flow allows users to live with peace of mind and provides a system that can respond quickly in emergencies.

[0808] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0809] This invention is a system that provides a safe and secure environment for parents of intellectually disabled children. This system records interactions between parents and their intellectually disabled children, and trains a generation AI to learn from them, enabling the generation AI to continue communicating with the children even after the parents have passed away. The generation AI also monitors the child's abnormal behavior and notifies pre-registered family members and welfare agencies in the event of an emergency. Furthermore, this invention incorporates an emotion engine that recognizes the user's emotions, allowing the generation AI to understand the user's emotional state and provide appropriate responses.

[0810] System Overview

[0811] Data collection and learning phase

[0812] 1. User initiates interaction

[0813] Device: Records conversations and actions between parents and children in audio, text, and video formats via smartphones and tablets used by family members.

[0814] Server: Receives data sent from the terminal in real time and performs preprocessing.

[0815] 2. Data Preprocessing

[0816] Server: Converts received data into text, removes noise, trims unnecessary parts, and extracts important information.

[0817] Server: Adds metadata (time, location, emotion tags, etc.) to the preprocessed data and stores it in a database.

[0818] 3. Generative AI Training

[0819] Server: The preprocessed data is input into the Generative AI's learning algorithm, which learns the interaction patterns between parents and children. This allows the Generative AI to imitate the support provided by parents.

[0820] Communication phase using generative AI

[0821] 1. Daily communication

[0822] User (child): In everyday life, the user asks questions such as "What should I do today?" to a device such as a tablet.

[0823] Terminal: Records the user's voice and converts it into text data (voice recognition function).

[0824] Terminal: Sends the converted text data to the server.

[0825] Server: The received text data is input into the generation AI, which generates an appropriate response. This response is sent back to the device and replied to the user by voice.

[0826] Examples:

[0827] User (child): "What shall we do today?"

[0828] Generative AI: "How about going to the park today?"

[0829] Emergency Notification Phase

[0830] 1. Monitoring for Abnormal Behavior

[0831] Generative AI: Monitors children's daily behavior and detects unusual behavior.

[0832] Generative AI: When it detects abnormal behavior, it sends a signal to the server that triggers an emergency notification.

[0833] 2. Sending emergency notifications

[0834] Server: Generates a message for emergency notification and sends it to pre-registered contacts (family members and welfare agencies).

[0835] Examples:

[0836] Generative AI: Detects when a child loses consciousness.

[0837] Server: Send emergency notification to family: "Your child has lost consciousness. Please check the scene immediately."

[0838] Use of emotion engine

[0839] sentiment analysis

[0840] Device: Analyzes emotions from the user's voice and text.

[0841] Server: The emotion engine detects the user's emotional state from their speaking style and tone of voice, and reflects this in the generative AI.

[0842] Generative AI: Based on detected emotional information, it generates more appropriate responses that take into account the user's psychological state.

[0843] Examples:

[0844] User (child): "My friend said something bad about me at school today."

[0845] Emotion Engine: Detects sadness from the tone and content of the user's voice.

[0846] Generating AI: "That's tough, but you're a wonderful person," it encourages.

[0847] Emotional abnormality monitoring

[0848] Generative AI: Continuously monitors the user's emotional state and sends an emergency notification if an abnormal state is detected.

[0849] Examples:

[0850] The user (child) frequently makes statements that indicate sadness or despair.

[0851] Emotion Engine: Detects emotional anomalies that exceed the normal range.

[0852] Generative AI: Detects an abnormality and the server sends an emergency notification: "There is something abnormal in the child's emotional state. Please check immediately."

[0853] The system's program records detailed interactions between parents and their intellectually disabled children, and by training the generative AI, it provides ongoing support to children even after the parents have passed away. Furthermore, by incorporating an emotion engine, it can recognize the user's emotional state and provide more appropriate support. This allows for a rapid response in emergencies, providing a safe and secure environment for children.

[0854] The processing flow will be explained below.

[0855] Data collection and learning phase

[0856] Step 1:

[0857] Users use devices such as smartphones and tablets to record and record conversations and actions between parents and children.

[0858] Step 2:

[0859] The device sends the recorded data to the server in real time.

[0860] Step 3:

[0861] The server preprocesses the received data, converting the audio data to text, removing noise, trimming unnecessary parts, and extracting important information.

[0862] Step 4:

[0863] The server adds metadata (time, location, emotion tags, etc.) to the preprocessed data and stores it in a database.

[0864] Step 5:

[0865] The server inputs the stored data into the Generative AI's learning algorithm, which learns characteristics from parent-child interactions and builds a model to generate future responses.

[0866] Communication phase using generative AI

[0867] Step 6:

[0868] In the course of daily interactions, the user (child) asks questions such as, "What shall we do today?"

[0869] Step 7:

[0870] The device records the user's voice and converts it into text data (voice recognition function).

[0871] Step 8:

[0872] The terminal transmits the converted text data to the server.

[0873] Step 9:

[0874] The server inputs the received text data into the generation AI, which then generates an appropriate response.

[0875] Step 10:

[0876] The server transmits the generated response to the terminal as text data.

[0877] Step 11:

[0878] The terminal converts the text data into speech and plays it back to the user.

[0879] Emergency Notification Phase

[0880] Step 12:

[0881] The generative AI monitors a child's daily behavior and detects any unusual behavior (e.g., suddenly collapsing).

[0882] Step 13:

[0883] When the generative AI detects abnormal behavior, it sends a signal to the server that triggers an emergency notification.

[0884] Step 14:

[0885] The server generates the content of the emergency notification and sends it to pre-registered family members and welfare agencies.

[0886] Step 15:

[0887] The user (family or welfare agency) receives a notification and promptly goes to the scene or takes the necessary action (e.g., calls 119).

[0888] Use of emotion engine

[0889] sentiment analysis

[0890] Step 16:

[0891] The device collects the user's voice and text data and sends it to the emotion engine.

[0892] Step 17:

[0893] The server uses an emotion engine to analyze the user's emotional state from the tone of voice and text, for example, identifying emotions such as joy, sadness, and anger from the tone of voice.

[0894] Step 18:

[0895] The generative AI generates an appropriate response based on the emotional information obtained from the emotion engine.

[0896] Examples:

[0897] User (child): "My friend said something bad about me at school today."

[0898] Emotion Engine: Detects sadness from the tone and content of the user's voice.

[0899] Generating AI: "That's tough, but you're a wonderful person," it encourages.

[0900] Emotional abnormality monitoring

[0901] Step 19:

[0902] The generative AI monitors the user's emotional state in real time, and if negative emotions are detected repeatedly, it will be deemed an abnormality.

[0903] Step 20:

[0904] When the generative AI detects an emotional anomaly, it sends a signal to the server that triggers an emergency notification.

[0905] Step 21:

[0906] The server generates emergency notifications about emotional abnormalities and sends them to pre-registered family members and welfare agencies.

[0907] Examples:

[0908] The user (child) frequently makes statements that express sadness and despair.

[0909] Emotion Engine: Detects abnormal emotional states that exceed the normal range.

[0910] Generative AI: Detects an abnormality and the server sends an emergency notification: "There is something abnormal in the child's emotional state. Please check immediately."

[0911] This system records detailed daily interactions between parents and their intellectually disabled children, and uses this information to train a generative AI, providing ongoing support for the child even after the parents have passed away. It also incorporates an emotion engine to recognize the user's emotional state and provide more appropriate support. It can respond quickly in emergencies, providing a safe and secure environment for children.

[0912] Example 2

[0913] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0914] There is a need to provide an environment where children with intellectual disabilities can live safely even after their parents pass away. However, conventional systems have difficulty completely replacing the role of parents, and are particularly limited in terms of communication between parents and emergency response. Furthermore, they lack the ability to appropriately understand and respond to children's emotional states, making it difficult to maintain a sense of psychological security for children.

[0915] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0916] In this invention, the server includes a means for recording interactions between a parent and their intellectually disabled child, a means for preprocessing the recorded data and training the generation AI, a means for the generation AI to communicate with the child after the parent's death, a means for the generation AI to monitor the child's behavior and detect abnormalities, a means for sending an emergency notification to contacts when an abnormality is detected, a means for analyzing the user's emotions, and a means for the generation AI to generate an appropriate response based on the analyzed emotional information. This allows the child to receive continuous support even after parental support is no longer available, and allows for rapid response in emergencies. Furthermore, the emotion engine can be used to understand the child's emotional state and provide appropriate psychological support.

[0917] "Recording means" means any device or software that collects and stores in audio, text, and video form communications between a parent and their child with an intellectual disability.

[0918] "Preprocessing means" refers to a device or program that removes noise from recorded data, converts it into text, and otherwise converts it into a format suitable for learning by generative AI.

[0919] "Generative AI" refers to an artificial intelligence model that can learn from input data and generate natural, human-like responses.

[0920] "Means of communication" refers to the device or software that enables the generative AI to generate appropriate responses to questions posed by the child and return them to the child in voice or text format.

[0921] "Behavioral monitoring means" refers to a device or program that continuously records a child's daily behavior and detects deviations from normal patterns.

[0922] "Means for detecting anomalies" refers to devices or software that identify anomalous patterns of behavior from monitored behavioral data and issue an alert if such patterns are detected.

[0923] "Means for sending emergency notifications" refers to a device or program that sends emergency notifications via SMS, email, etc. to pre-registered contacts (family members or welfare agencies) when an abnormality is detected.

[0924] "Means for analyzing emotions" refers to a device or software that analyzes the emotional state of a user from their voice or text and reflects the results in the generative AI's response.

[0925] "Means for generating an appropriate response" refers to a device or program that enables the generation AI to create a response that takes into account the user's psychological state based on analyzed emotional information and user input.

[0926] This invention is a system that records interactions between parents and intellectually disabled children and trains a generative AI to provide ongoing support for children even after their parents pass away. It also has the ability to monitor abnormal behavior and send prompt notifications in the event of an emergency. It can also analyze the user's emotions and provide appropriate responses based on that information.

[0927] System configuration

[0928] Data collection and learning phase

[0929] First, the user uses a device such as a smartphone or tablet to record conversations and behaviors between parents and their children in audio, text, and video formats. Recordings are made using the default recording or camera app. The recorded data is then sent in real time to a cloud server (e.g., AWS S3).

[0930] The server then preprocesses the received data. Audio data is converted to text using the Google Cloud Speech-to-Text API, and noise is removed using NLTK (Natural Language Toolkit). Specific frames of the video data are extracted using FFmpeg and saved as images. Metadata (e.g., recording date and time, recording location, emotion tags) is added to the preprocessed data, and it is then stored in a NoSQL database such as MongoDB.

[0931] The stored data is then fed into a generative AI (such as OpenAI's GPT-3) to learn the interaction patterns between parents and children. In the process, the generative AI will be able to mimic the support provided by parents.

[0932] Communication phase using generative AI

[0933] Users (children) can ask a device such as a tablet, "What shall we do today?" in their daily lives. The question is recorded on the device and converted into text data using the Google Cloud Speech-to-Text API. The converted text data is sent to a server and input into the generation AI.

[0934] The server generates an appropriate response using the AI ​​and sends it back to the device. The user is then told a voice response, using Amazon Polly for speech synthesis.

[0935] Examples:

[0936] User (child): "What shall we do today?"

[0937] Generative AI: "How about going to the park today?"

[0938] Emergency Notification Phase

[0939] The AI ​​monitors the child's daily activities and detects any unusual behavior. For example, if a child stays in the same position for a long time, it is detected as abnormal behavior. When abnormal behavior is detected, the AI ​​sends a signal to the server that triggers an emergency notification.

[0940] The server sends emergency notifications via SMS or email to pre-registered contacts (family members or welfare agencies) using services such as AWS SNS (Simple Notification Service).

[0941] Examples:

[0942] Generative AI: Detects when a child loses consciousness.

[0943] Server: Send emergency notification to family: "Your child has lost consciousness. Please check the scene immediately."

[0944] Use of emotion engine

[0945] The user's voice and text are analyzed for emotional state using IBM Watson Tone Analyzer. The server uses an emotion engine to detect emotional information from the user's speaking style and tone of writing, and reflects that information in the generation AI. Based on the detected emotional information, the generation AI generates a more appropriate response that takes the user's psychological state into consideration.

[0946] Examples:

[0947] User (child): "My friend said something bad about me at school today."

[0948] Emotion Engine: Detects sadness from the tone and content of the user's voice.

[0949] Generating AI: "That's tough, but you're a wonderful person," it encourages.

[0950] The emotion engine also continuously monitors the user's emotional state and provides emergency notifications if it detects an abnormal state.

[0951] Examples:

[0952] The user (child) frequently makes statements that indicate sadness or despair.

[0953] Emotion Engine: Detects emotional anomalies that exceed the normal range.

[0954] Generative AI: An abnormality is detected and the server sends an emergency notification: "There is something abnormal in the child's emotional state. Please check immediately."

[0955] As described above, this system records detailed interactions between parents and their intellectually disabled children and trains the generative AI to provide ongoing support for the child even after the parents have passed away. Furthermore, by combining it with an emotion engine, it can recognize the user's emotional state and provide more appropriate support. This allows for a rapid response in emergencies, providing an environment where children can live in peace of mind.

[0956] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0957] Program processing flow

[0958] Data collection and learning phase

[0959] Step 1: User initiates interaction

[0960] Device: A smartphone or tablet is used to record conversations and actions between parents and children in audio, text, and video formats. The input is the conversation or action, and the output is an audio file, text file, or video file. Specifically, audio recording is done using a recording app, and video recording is done using a camera app.

[0961] Step 2: Preprocessing the data

[0962] Server: Receives data sent from the device in real time and performs preprocessing. The inputs are audio files, text files, and video files, and the output is processed text data and image data. Specifically, the Google Cloud Speech-to-Text API is used to convert audio data into text, and NLTK is used to remove noise. FFmpeg is used to extract specific frames from video data and save them as image files.

[0963] Step 3: Training the generative AI

[0964] Server: The preprocessed data is given metadata (recording date and time, recording location, emotion tag) and stored in a database (e.g., MongoDB). The input is the preprocessed data, and the output is a database entry with the metadata. The stored data is fed into a generative AI (e.g., OpenAI's GPT-3), which learns the interaction patterns between parents and children. Through this learning, the generative AI is able to imitate the support provided by the parent.

[0965] Communication phase using generative AI

[0966] Step 4: Daily Communication

[0967] User (child): Asks a device such as a tablet, "What shall we do today?" The input is the user's voice, and the output is a recorded audio file.

[0968] Terminal: Convert the recorded audio into text data using the Google Cloud Speech-to-Text API. The input is an audio file and the output is text data.

[0969] Terminal: Transmits converted text data to the server. The input is text data, and the output is data transmission to the server.

[0970] Server: Inputs the received text data into the generation AI and generates an appropriate response. The input is text data, and the output is the generated response text.

[0971] Server: Sends the generated response to the device and replies to the user via speech synthesis (using Amazon Polly). The input is the generated response text and the output is the audio response.

[0972] Examples:

[0973] User (child): "What shall we do today?"

[0974] Generative AI: "How about going to the park today?"

[0975] Emergency Notification Phase

[0976] Step 5: Monitor for abnormal behavior

[0977] Generative AI: Monitors a child's daily behavior and detects any unusual behavior. The input is daily behavior data, and the output is an alert for abnormal behavior detection. For example, it monitors long periods of inactivity or abnormal behavior patterns.

[0978] Step 6: Send emergency notifications

[0979] Server: When abnormal behavior is detected, it generates an emergency notification and sends it to pre-registered contacts (family members or welfare agencies). The input is the abnormal behavior detection alert, and the output is an emergency notification message to the contacts. The notification is sent using AWS SNS (Simple Notification Service).

[0980] Examples:

[0981] Generative AI: Detects when a child loses consciousness.

[0982] Server: Send emergency notification to family: "Your child has lost consciousness. Please check the scene immediately."

[0983] Use of emotion engine

[0984] Step 7: Sentiment analysis

[0985] Terminal: Analyzes emotions from user voice and text using IBM Watson Tone Analyzer. The input is an audio file or text data, and the output is analyzed emotional information.

[0986] Server: Using the emotion engine, the server detects the user's emotional state from their speaking style and tone of voice, and then feeds that information into the generative AI. The input is analyzed emotional information, and the output is text data with emotion tags.

[0987] Step 8: Generate an appropriate response

[0988] Generative AI: Generates more appropriate responses based on detected emotional information. The input is emotion-tagged text data, and the output is an emotion-sensitive response.

[0989] Examples:

[0990] User (child): "My friend said something bad about me at school today."

[0991] Emotion Engine: Detects sadness from the tone and content of the user's voice.

[0992] Generating AI: "That's tough, but you're a wonderful person," it encourages.

[0993] Step 9: Monitor for emotional abnormalities

[0994] Generative AI: Continuously monitors the user's emotional state and sends an emergency notification if an abnormality is detected. The input is emotional state data, and the output is an alert for anomaly detection. A specific example is when a user frequently makes statements that express sadness or despair.

[0995] Server: Detects anomalies and sends emergency notifications. The input is an anomaly detection alert, and the output is an emergency notification message.

[0996] Examples:

[0997] Emotion Engine: Detects emotional anomalies that exceed the normal range.

[0998] Generative AI: An abnormality is detected and the server sends an emergency notification: "There is something abnormal in the child's emotional state. Please check immediately."

[0999] (Application example 2)

[1000] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1001] Conventional monitoring systems are not specialized for people with intellectual disabilities or the elderly, and may not respond appropriately in emergencies. Furthermore, there is a lack of systems that can detect changes in a user's emotional state in real time and provide appropriate responses. Therefore, new technology is needed to balance user safety and mental care.

[1002] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1003] In this invention, the server includes: means for recording interactions between a parent and their intellectually disabled child; means for preprocessing the recorded data and training a generation AI; means for the generation AI to communicate with the child after the parent has passed away; means for the generation AI to monitor the child's abnormal behavior; means for issuing an emergency notification if an abnormality is detected; means for recognizing the user's voice and converting it into text; means for analyzing the emotions of the converted text data using an emotion engine; means for generating an appropriate response based on the analyzed emotional data; means for replying to the user with the generated response by voice; and means for detecting an abnormal emotional state through emotion analysis and issuing an emergency notification. This enables both daily safety management and psychological care for the user.

[1004] An "interaction recording device" is any device or method that records, in audio, text, or video format, the conversations or activities between a parent and their child with an intellectual disability.

[1005] "Preprocessing" refers to the process of converting recorded data into text, removing noise, and extracting important information.

[1006] "Generative AI" is an artificial intelligence that uses machine learning models to mimic the interactions between parents and their intellectually disabled children, enabling continuous communication.

[1007] "Means of communication" refers to the ability to use generative AI to converse and interact with children.

[1008] The "means of monitoring abnormal behavior" is a function that allows the generating AI to monitor a child's daily behavior and detect any behavior that is out of the ordinary.

[1009] The "means for sending emergency notifications" is a function that sends an emergency message to pre-registered family members and welfare organizations when an abnormality is detected.

[1010] "Means for recognizing voice and converting it into text" refers to technology that analyzes the voice spoken by the user and converts it into text data.

[1011] An "emotion engine" is an artificial intelligence model that analyzes emotions from a user's voice and text.

[1012] "Means for generating an appropriate response" refers to the function by which the generative AI creates an appropriate response or reply based on the analyzed emotional data.

[1013] "Means for replying by voice" refers to a technology that uses voice synthesis technology to convey the generated response to the user by voice.

[1014] "Means for detecting emotional state" refers to the function by which the emotion engine detects changes in the user's emotions in real time and provides that information to the generation AI.

[1015] The present invention is a system that records interactions between parents and their intellectually disabled children, formats the data through preprocessing, trains a generating AI, and then enables the generating AI to communicate with the child even after the parents have passed away, monitors abnormal behavior or emotional abnormalities, and sends emergency notifications. Detailed embodiments of the system are described below.

[1016] In this invention, the terminal can be a smartphone, tablet, intercom, etc., which can be used to communicate with the user and record voice. The server is installed on the cloud, where the recorded data is stored, preprocessed, and the generative AI model is trained.

[1017] Hardware and software used

[1018] Hardware: Smartphone, tablet, intercom, computer with microphone

[1019] Software: Python, SpeechRecognition, Transformers (Huggingface), Twilio API

[1020] Program Overview and Processing

[1021] 1. Recording interactions

[1022] The device records conversations and behaviors between parents and their children with intellectual disabilities in audio, text, or video format, and the recorded data is sent to a server in real time.

[1023] 2. Data Preprocessing

[1024] The server converts the received data into text, removes noise, and extracts important information. The preprocessed data is then annotated with metadata (such as time, location, and emotion tags) and stored in a database.

[1025] 3. Generative AI Training

[1026] The server inputs the preprocessed data into the generative AI's learning algorithm, which learns the interaction patterns between parents and children. The generative AI is then able to mimic the support provided by the parent and generate appropriate responses.

[1027] 4. Daily communication

[1028] The user (child) asks questions to the device in their daily life. The device records the user's voice and converts it into text data. This data is sent to the server, where the AI ​​generates an appropriate response and sends it to the device, which then replies to the user by voice.

[1029] Example: User (child): "What shall we do today?"

[1030] Generative AI: "How about going to the park today?"

[1031] 5. Emotion Recognition and Analysis

[1032] The device analyzes the user's emotions from their voice and text, and the server uses an emotion engine to detect the user's emotional state and reflects this in the generating AI.

[1033] Example: User (child): "My friend said something bad about me at school today."

[1034] Emotion Engine: Detects sadness from the tone and content of the user's voice.

[1035] Generator: "That's tough, but you're a wonderful being."

[1036] 6. Monitoring for abnormal behavior and emotional abnormalities

[1037] The AI ​​monitors the child's daily behavior and detects abnormal behavior. It also continuously monitors the user's emotional state and sends an emergency notification if it detects an abnormal state.

[1038] Example: Generative AI detects when a child loses consciousness.

[1039] Server: Send emergency notification to family: "Your child has lost consciousness. Please check the scene immediately."

[1040] Example prompt: "I'm not feeling too great today." Please assess the emotion in this text. Please advise how to respond on behalf of the parent.

[1041] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1042] Step 1:

[1043] The user speaks to the device, which records the user's voice.

[1044] Input: User's voice

[1045] Output: Recorded audio data

[1046] What it does: Uses the device's microphone to record what the user says.

[1047] Step 2:

[1048] The device converts recorded audio into text.

[1049] Input: Recorded audio data

[1050] Output: Text data

[1051] What it does: Uses speech recognition software (e.g., Google Speech Recognition API) to convert voice data into text.

[1052] Step 3:

[1053] The terminal transmits the text data to the server.

[1054] Input: Text data

[1055] Output: Text data sent to the server

[1056] Operation: Uses the device's communication function to send text data to a server in the cloud.

[1057] Step 4:

[1058] The server preprocesses the received text data, removing noise and extracting important information.

[1059] Input: Text data sent to the server

[1060] Output: Preprocessed text data

[1061] Operation: The server performs noise removal and important information extraction on the text data, and then adds metadata.

[1062] Step 5:

[1063] The server inputs the preprocessed text data into the generation AI, which generates an appropriate response.

[1064] Input: Preprocessed text data

[1065] Output: Response by the generative AI

[1066] How it works: The server inputs text data into the generative AI model, which generates appropriate responses based on the parent's response patterns learned by the model.

[1067] Step 6:

[1068] The server generates a response and sends it to the terminal.

[1069] Input: Response by generative AI

[1070] Output: The response sent to the device

[1071] Operation: The server sends the response generated by the generation AI to the terminal via its communication function.

[1072] Step 7:

[1073] The terminal returns the response received from the server to the user by voice.

[1074] Input: The response sent to the terminal

[1075] Output: A spoken response to the user

[1076] How it works: The device uses speech synthesis software to convert the text response received from the server into audio and play it back to the user.

[1077] Step 8:

[1078] The terminal transmits the text data to an emotion engine to analyze the user's emotions.

[1079] Input: Text data

[1080] Output: Emotion analysis results

[1081] How it works: The device uses an emotion engine (e.g., a Transformers emotion analysis model) to analyze the emotional state of text data.

[1082] Step 9:

[1083] The server detects emotional abnormalities and sends an emergency notification if any abnormalities are detected.

[1084] Input: Sentiment analysis results

[1085] Output: Send emergency notification

[1086] How it works: The server analyzes the emotion analysis results, and if it detects an abnormal emotional state, it sends an emergency notification to pre-registered contacts.

[1087] Step 10:

[1088] Emergency notifications reach registered contacts.

[1089] Input: Send emergency notification

[1090] Output: Contact notification

[1091] What it does: Your server uses a communication service, such as the Twilio API, to send an emergency message to registered contacts.

[1092] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1093] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1094] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1095] [Third embodiment]

[1096] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1097] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1098] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1099] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1100] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1101] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1102] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1103] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1104] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1105] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1106] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1107] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1108] This invention is a system that aims to provide an environment where parents of intellectually disabled children can live with peace of mind. This system records interactions between parents and their intellectually disabled children and trains a generative AI to learn from this data, allowing the generative AI to continue communicating with the child even after the parent has passed away. It also includes a function to monitor abnormal behavior and notify pre-registered family members and welfare organizations in the event of an emergency.

[1109] System Overview

[1110] Data collection and learning phase

[1111] 1. User initiates interaction

[1112] Devices: Family members' smartphones and tablets record daily interactions between parents and children in audio, text, and video formats.

[1113] Server: Receives data sent from the terminal and performs preprocessing.

[1114] 2. Data Preprocessing

[1115] Server: Converts received data into text, removes noise and unnecessary information, and structures the data by adding metadata such as time, location, and emotion tags.

[1116] 3. Generative AI Training

[1117] Server: The preprocessed data is input into the generative AI's learning model to learn the interaction patterns between parents and children, allowing the generative AI to mimic the support provided by parents.

[1118] Communication phase using generative AI

[1119] 1. Daily communication

[1120] User (child): Asks questions such as "What should we do today?" to a device such as a tablet.

[1121] Terminal: Converts this question into text data and sends it to the server.

[1122] Server: Inputs text data into the AI ​​generator and generates an appropriate response. This response is then sent back to the device and spoken back to the child.

[1123] Examples:

[1124] User (child): "What shall we do today?"

[1125] Generative AI: "How about going to the park today?"

[1126] Emergency Notification Phase

[1127] 1. Monitoring for Abnormal Behavior

[1128] Generative AI: Constantly monitors the child's behavior and condition, and if it detects behavior that deviates from the norm, it determines that it is abnormal.

[1129] Trigger emergency notifications if an anomaly is detected.

[1130] 2. Sending emergency notifications

[1131] Server: Generates emergency notification messages and sends them to pre-defined contacts (family members and welfare agencies).

[1132] Examples:

[1133] Generative AI: Detects when a child loses consciousness.

[1134] Server: Send emergency notification to family: "Your child has lost consciousness. Please check the scene immediately."

[1135] Example

[1136] In a concrete example, the process works as follows:

[1137] 1. User (child): Communicate regularly with the generated AI in daily life.

[1138] 2. Generative AI: Acts as a parent, providing appropriate advice and instructions.

[1139] 3. In case of abnormality: The generated AI detects abnormal behavior of the child and the server sends an emergency notification.

[1140] 4. Who to notify: Families and welfare agencies that receive the notification will respond promptly.

[1141] In this way, this invention is a system that provides an environment where children with intellectual disabilities can live safely even after their parents pass away. The data processing and generative AI functions of each phase make continuous support possible.

[1142] The processing flow will be explained below.

[1143] Data collection and learning phase

[1144] Step 1:

[1145] Users use devices such as smartphones and tablets to record and record conversations and actions between parents and children.

[1146] Step 2:

[1147] The device sends the recorded data to the server in real time.

[1148] Step 3:

[1149] The server preprocesses the received data, converting the audio data to text, removing noise, trimming unnecessary parts, and extracting important information.

[1150] Step 4:

[1151] The server adds metadata (time, location, emotion tags, etc.) to the preprocessed data and stores it in a database.

[1152] Step 5:

[1153] The server inputs the stored data into the Generative AI's learning algorithm, which learns characteristics from parent-child interactions and builds a model to generate future responses.

[1154] Communication phase using generative AI

[1155] Step 6:

[1156] In the course of daily interactions, the user (child) asks questions such as, "What shall we do today?"

[1157] Step 7:

[1158] The device records the user's voice and converts it into text data (voice recognition function).

[1159] Step 8:

[1160] The terminal transmits the converted text data to the server.

[1161] Step 9:

[1162] The server inputs the received text data into the generation AI, which then generates an appropriate response.

[1163] Step 10:

[1164] The server transmits the generated response to the terminal as text data.

[1165] Step 11:

[1166] The terminal converts the text data into speech and plays it back to the user.

[1167] Emergency Notification Phase

[1168] Step 12:

[1169] The generative AI monitors a child's daily behavior and detects any unusual behavior (e.g., suddenly collapsing).

[1170] Step 13:

[1171] When the generative AI detects abnormal behavior, it sends a signal to the server that triggers an emergency notification.

[1172] Step 14:

[1173] The server generates the content of the emergency notification and sends it to pre-registered family members and welfare agencies.

[1174] Step 15:

[1175] The user (family or welfare agency) receives a notification and promptly goes to the scene or takes the necessary action (e.g., calls 119).

[1176] The system's program records detailed daily interactions between parents and their intellectually disabled children, and trains the AI ​​to learn from these interactions, providing ongoing support for the child even after the parents have passed away. It also has a built-in notification system for rapid response in emergencies.

[1177] Example 1

[1178] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1179] There is a lack of support for intellectually disabled children to live safely after their parents pass away. There is no way to continue communicating with their children, monitoring their behavior, or responding in the event of an emergency, as parents did while they were alive, which raises the issue of the possibility that the safety and quality of life of the children may decline.

[1180] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1181] In this invention, the server includes a device that records interactions between parents and their intellectually disabled children, a device that preprocesses the recorded data and trains a generative AI model, a device that allows the generative AI model to communicate with the child after the parents pass away, a device that allows the generative AI model to monitor abnormal behavior in the child, and a device that issues an emergency notification if an abnormality is detected. This enables intellectually disabled children to continue living in a safe and stable environment even after the parents pass away.

[1182] "Devices that record interactions between parents and children with intellectual disabilities" means devices that collect and retain audio, text, and video communications between parents and children with intellectual disabilities.

[1183] The "device that preprocesses recorded data and trains a generative AI model" is a system that performs preprocessing such as noise removal, text conversion, and metadata addition based on recorded interaction data, and then uses that data to train a generative AI model.

[1184] The "device that allows a generative AI model to communicate with children after their parents have passed away" is a system that allows a generative AI model to continuously interact with and support children with intellectual disabilities based on data learned, even when their parents are no longer present.

[1185] The "device in which a generative AI model monitors children's abnormal behavior" is a system in which a generative AI model constantly monitors children's behavior and judges it as abnormal if it deviates from normal behavior patterns.

[1186] The "device that sends emergency notifications when an abnormality is detected" is a system that sends emergency notifications to pre-registered contacts when the AI ​​model detects abnormal behavior in a child.

[1187] "Means for storing preprocessed data in an information storage device and updating the configuration of the generative AI model" refers to a technology for restoring preprocessed data to a database or storage device and using that data to update the algorithms and parameters of the generative AI model.

[1188] "Means for sending emergency notifications to pre-registered family members and welfare agencies" refers to a system that sends emergency messages to pre-designated family members and welfare agencies when a child exhibits abnormal behavior.

[1189] This invention is a system that provides a safe and secure environment for parents of intellectually disabled children. The system records interactions between parents and their children and trains a generative AI model to learn from this data, enabling the model to continue communicating with the child after the parent's death. The system also includes a function in which the generative AI model monitors for abnormal behavior and notifies pre-registered family members and welfare agencies in the event of an emergency.

[1190] Data collection and learning phase

[1191] First, users (parents and children) use devices such as smartphones and tablets to carry out everyday interactions. These interactions are recorded in audio, text, and video formats. For example, a parent might say, "Good morning, how was your day?", and a child might respond, "I drew a picture at school today."

[1192] The device then sends the recorded data to a server, which converts the received data into text using voice recognition technology (e.g., Google Speech-to-Text API), removes noise, filters out unnecessary information, and extracts only the important information. The server also structures the data by adding metadata such as time, location, and emotion tags.

[1193] The server then feeds the pre-processed data into a generative AI model to learn parent-child interaction patterns, allowing the model to mimic the support provided by the parent.

[1194] Communication phase using generative AI

[1195] In everyday communication, a user (child) will ask a device such as a tablet, "What shall we do today?" This question is converted into text data using speech recognition technology and sent to a server. The server inputs the text data into a generative AI model to generate an appropriate response. This response is sent back to the device and is returned to the child as a voice using speech synthesis technology (e.g., Google Text-to-Speech API).

[1196] For example, if a child asks, "What shall we do today?", the generative AI model might suggest, "How about going to the park today?"

[1197] Emergency Notification Phase

[1198] The system's generative AI model constantly monitors a child's behavior and state, and if it detects behavior that deviates from the norm, it will determine that something is abnormal, such as if the child loses consciousness or exhibits unusual behavioral patterns.

[1199] If an abnormality is detected, the server generates an emergency notification message and sends it to pre-defined contacts (family members or welfare agencies). For example, it could send a message via SMS or email saying, "Your child has lost consciousness. Please check the scene immediately."

[1200] In this way, the present invention is a system that realizes functions such as recording interactions with parents, learning using a generative AI model, communication, monitoring abnormal behavior, and emergency notifications, in order to provide an environment where intellectually disabled children can live safely even after their parents have passed away.

[1201] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1202] Step 1:

[1203] User initiates interaction

[1204] Input: Daily interactions between users (parents and children) (voice, text, video)

[1205] Action: Parent and child engage in everyday interactions, such as parent saying, "Good morning, how was your day?" and child responding, "I drew a picture at school today."

[1206] Output: Interaction data is recorded on a smartphone or tablet

[1207] Step 2:

[1208] Sending data

[1209] Input: Recorded interaction data (audio, text, video)

[1210] How it works: The device sends the recorded data to a server. For example, audio data recorded using a tablet's microphone is uploaded to a cloud server via Wi-Fi.

[1211] Output: The server receives the data

[1212] Step 3:

[1213] Data Preprocessing

[1214] Input: Data of the interaction received by the server (audio, text, video)

[1215] How it works: The server uses speech recognition technology (e.g., Google Speech-to-Text API) to convert voice data into text, remove noise, and filter out unnecessary information. It also structures the data by adding metadata such as time, location, and emotion tags. For example, voice data such as "Good morning, how was your day?" is converted to text like "Good morning, how was your day?", and metadata such as October 10, 2023, and the living room are added.

[1216] Output: Preprocessed text data

[1217] Step 4:

[1218] Training generative AI models

[1219] Input: Preprocessed text data

[1220] How it works: The server feeds the preprocessed data into a generative AI model that learns parent-child interaction patterns, such as what response a parent gets when they say "good morning."

[1221] Output: A trained generative AI model

[1222] Step 5:

[1223] Daily communication

[1224] Input: Questions (voice) asked by the user (child) to the device

[1225] How it works: The user (child) asks a tablet or other device, "What shall we do today?" This question is converted into text data using speech recognition technology and sent to a server. The server inputs the text data into a generative AI model and generates an appropriate response. This response is sent back to the device, and is replies to the child as audio using speech synthesis technology (e.g., Google Text-to-Speech API).

[1226] How it works: When a child asks, "What shall we do today?", the generative AI model suggests, "How about going to the park today?"

[1227] Output: Voice response

[1228] Step 6:

[1229] Abnormal behavior monitoring

[1230] Input: User (child) behavior data

[1231] How it works: The generative AI model constantly monitors the child's behavior and state, and if it detects any behavior that deviates from the norm, it will judge it to be abnormal. For example, if the child suddenly stops moving or exhibits an unusual behavior pattern, it will detect an abnormality.

[1232] Output: Anomaly detection alert

[1233] Step 7:

[1234] Sending emergency notifications

[1235] Input: Anomaly detection alert, pre-registered contact information

[1236] What it does: The server generates an emergency notification message and sends it to pre-defined contacts (family or welfare agencies), for example, an SMS or email message saying "Your child has lost consciousness. Please check the scene immediately."

[1237] Output: Send emergency notification message

[1238] (Application example 1)

[1239] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1240] There is a need to reduce the anxiety that users who particularly need supervision, such as people with intellectual disabilities and the elderly, experience in their daily lives. It is important to provide an environment where users can live safely even when their parents or caregivers are not always nearby. There is also a need for a system that can respond quickly in the event of abnormal behavior or an emergency. Furthermore, there is a need to ensure that users do not feel lonely even after their parents pass away, and that they continue to receive the necessary support.

[1241] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1242] In this invention, the server includes a means for recording interactions between parents and persons with intellectual disabilities, a means for preprocessing the recorded data and having the generation AI learn from it, a means for the generation AI to communicate with the user after the parent has passed away, a means for the generation AI to monitor the user's abnormal behavior, a means for sending an emergency notification when an abnormality is detected, a means for monitoring the user's behavior and status using the camera and microphone of the smart device, a means for the generation AI to communicate with the user in real time, and a means for sending a notification to pre-set contacts when abnormal behavior is detected. This provides an environment where users can live safely and enables rapid response in emergencies.

[1243] A "means for recording interactions between a parent and a child with an intellectual disability" is a device or method for recording, by audio, text, video, or other means, the conversations or activities between a parent and a child with an intellectual disability.

[1244] "Means for preprocessing recorded data and training a generative AI" refers to a device or method for converting recorded data into text, performing preprocessing such as noise removal and adding metadata, and then training the generative AI model.

[1245] "Means for the generative AI to communicate with the user after the parents have passed away" refers to a device or method that allows the generative AI to interact with the user on behalf of the parents and provide necessary support and advice.

[1246] "Means for monitoring abnormal user behavior by a generation AI" refers to a device or method by which a generation AI monitors a user's daily behavior and detects abnormal behavior.

[1247] The "means for sending an emergency notification when an abnormality is detected" refers to a device or method for sending an emergency notification to pre-defined contacts when abnormal user behavior is detected.

[1248] "Means for monitoring a user's behavior and status using a camera or microphone on a smart device" refers to a device or method for recording and monitoring a user's behavior and status using a camera and microphone on a device such as a smartphone or head-mounted display.

[1249] "Means for the generation AI to communicate with the user in real time" refers to a device or method that enables the generation AI to respond immediately to the user's questions or requests and realize real-time dialogue.

[1250] "Means for sending notifications to pre-defined contacts when abnormal behavior is detected" refers to a device or method for sending notifications to pre-defined emergency contacts (such as family members or welfare agencies) when abnormal behavior of a user is detected.

[1251] The embodiments for implementing the invention are carried out in the following methods and steps.

[1252] Hardware and Software Use

[1253] 1. Hardware

[1254] Mobile devices such as smartphones and tablets

[1255] head-mounted display

[1256] Cloud server (e.g. AWS, Google Cloud)

[1257] Camera and microphone

[1258] 2. Software

[1259] Data collection applications (e.g., applications developed with Flutter)

[1260] Data preprocessing script (e.g., Python)

[1261] Natural language processing libraries (e.g. NLTK)

[1262] Generative AI models (e.g., OpenAI GPT-4)

[1263] Emergency notification API (e.g. Twilio API)

[1264] Anomaly detection algorithms (e.g., TensorFlow)

[1265] Data collection

[1266] The device (smartphone or tablet) records the interactions between parents and their children with intellectual disabilities in audio, text, and video formats, and the recorded data is sent to a cloud server where it is stored.

[1267] Data Preprocessing

[1268] The server converts the transmitted data into text, removes noise, and structures the data by adding metadata such as time, location, and emotion tags. This preprocessed data is then stored in a database.

[1269] Generative AI training

[1270] The server inputs the preprocessed data into the generative AI's learning model, learning the interaction patterns between parents and children, allowing the generative AI to provide appropriate support as a substitute parent.

[1271] Communication with users

[1272] The user (child) asks everyday questions or requests to the device, such as "What shall we do today?"

[1273] Examples:

[1274] User: "What should I do today?"

[1275] Generative AI: "How about going to the park today?"

[1276] The server converts the user's question into text data, inputs it into a generation AI, and generates an appropriate response. This response is then sent back to the device and replies to the user via voice.

[1277] Abnormal behavior monitoring and emergency notification

[1278] The AI ​​constantly monitors the user's behavior and status, and if it detects any behavior that deviates from the norm, it judges it to be an anomaly. If an anomaly is detected, the server will send an emergency notification to pre-defined contacts.

[1279] Examples:

[1280] Generative AI: Detects when the user has lost consciousness.

[1281] Server: Send emergency notification to family: "Your child has lost consciousness. Please check the scene immediately."

[1282] Prompt Sentence Examples

[1283] An example of a prompt sentence might be:

[1284] "Design a system that constantly monitors the user's mood and state and sends emergency notifications at the appropriate time."

[1285] "How can I automatically notify family members or designated contacts when abnormal behavior is detected?"

[1286] This system allows users to live with peace of mind and enables quick response in emergencies.

[1287] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1288] Step 1:

[1289] Data collection

[1290] Subject: Terminal

[1291] Input: Interactions between parents and their intellectually disabled children (audio, text, video)

[1292] Specific actions: Daily conversations and activities between parents and children are recorded using a data collection application on a smartphone or tablet.

[1293] Data processing and calculation: Recorded audio data and captured video data are sent to a cloud server.

[1294] Output: Recorded audio and video data is saved on a cloud server.

[1295] Step 2:

[1296] Data Preprocessing

[1297] Subject: Server

[1298] Input: Recorded audio and video data

[1299] Specific operations: Convert the data into text (speech recognition), remove noise, and add appropriate metadata (date, time, location, emotion tags, etc.).

[1300] Data processing and computation: Convert the audio data to text using a natural language processing library (e.g., NLTK) and update the database to add metadata.

[1301] Output: Preprocessed text data and structured metadata

[1302] Step 3:

[1303] Generative AI training

[1304] Subject: Server

[1305] Input: Preprocessed text data, structured metadata

[1306] Specific operation: The preprocessed data is input into a generative AI model (e.g., OpenAI GPT-4) to learn the interaction patterns between parents and children.

[1307] Data processing and calculation: Preprocessing data is fed into the learning model in large quantities, allowing the generative AI to recognize and learn patterns, enabling the AI ​​model to provide appropriate support in place of a parent.

[1308] Output: Trained generative AI model

[1309] Step 4:

[1310] Communication with users

[1311] Subject: Terminal

[1312] Input: User question or request (voice, text)

[1313] Specific behavior: The user asks a question or speaks to the device. Example: "What shall we do today?"

[1314] Data processing and calculation: The voice recognition function converts the user's question into text data, and this text data is sent to the cloud server.

[1315] Output: The user's question text data is sent to the generation AI.

[1316] Step 5:

[1317] Response generation by generative AI

[1318] Subject: Server

[1319] Input: User question text data

[1320] Specific operation: The generation AI generates an appropriate response based on the user's question text data.

[1321] Data processing and calculation: Based on the patterns learned by the generative AI, appropriate answers are generated to the user's questions.

[1322] Output: Generated response text data

[1323] Step 6:

[1324] Providing a response

[1325] Subject: Terminal

[1326] Input: Generated response text data

[1327] Specific operation: The generated response text data is converted into speech and output to the user. Example: "How about going to the park today?"

[1328] Data processing and calculation: Convert text into audio data using a speech conversion engine (e.g., Google Text-to-Speech).

[1329] Output: A spoken response to the user

[1330] Step 7:

[1331] Abnormal behavior monitoring

[1332] Subject: Generation AI

[1333] Input: User behavior and state data (camera, microphone)

[1334] Specific behavior: The generative AI monitors the user's real-time behavior and state and detects behavior that deviates from the norm.

[1335] Data processing and calculation: Identify abnormal behavior using an abnormal behavior detection algorithm (e.g., TensorFlow).

[1336] Output: Anomaly notification data when abnormal behavior is detected

[1337] Step 8:

[1338] Sending emergency notifications

[1339] Subject: Server

[1340] Input: Error notification data

[1341] Specific behavior: When an abnormality is detected, an emergency notification will be sent to pre-defined contacts. For example, "Your child has lost consciousness. Please check the scene immediately."

[1342] Data processing and calculation: Use emergency notification APIs (e.g., Twilio API) to send notifications about abnormal behavior to pre-registered contacts.

[1343] Output: Emergency notification message

[1344] This processing flow allows users to live with peace of mind and provides a system that can respond quickly in emergencies.

[1345] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1346] This invention is a system that provides a safe and secure environment for parents of intellectually disabled children. This system records interactions between parents and their intellectually disabled children, and trains a generation AI to learn from them, enabling the generation AI to continue communicating with the children even after the parents have passed away. The generation AI also monitors the child's abnormal behavior and notifies pre-registered family members and welfare agencies in the event of an emergency. Furthermore, this invention incorporates an emotion engine that recognizes the user's emotions, allowing the generation AI to understand the user's emotional state and provide appropriate responses.

[1347] System Overview

[1348] Data collection and learning phase

[1349] 1. User initiates interaction

[1350] Device: Records conversations and actions between parents and children in audio, text, and video formats via smartphones and tablets used by family members.

[1351] Server: Receives data sent from the terminal in real time and performs preprocessing.

[1352] 2. Data Preprocessing

[1353] Server: Converts received data into text, removes noise, trims unnecessary parts, and extracts important information.

[1354] Server: Adds metadata (time, location, emotion tags, etc.) to the preprocessed data and stores it in a database.

[1355] 3. Generative AI Training

[1356] Server: The preprocessed data is input into the Generative AI's learning algorithm, which learns the interaction patterns between parents and children. This allows the Generative AI to imitate the support provided by parents.

[1357] Communication phase using generative AI

[1358] 1. Daily communication

[1359] User (child): In everyday life, the user asks questions such as "What should I do today?" to a device such as a tablet.

[1360] Terminal: Records the user's voice and converts it into text data (voice recognition function).

[1361] Terminal: Sends the converted text data to the server.

[1362] Server: The received text data is input into the generation AI, which generates an appropriate response. This response is sent back to the device and replied to the user by voice.

[1363] Examples:

[1364] User (child): "What shall we do today?"

[1365] Generative AI: "How about going to the park today?"

[1366] Emergency Notification Phase

[1367] 1. Monitoring for Abnormal Behavior

[1368] Generative AI: Monitors children's daily behavior and detects unusual behavior.

[1369] Generative AI: When it detects abnormal behavior, it sends a signal to the server that triggers an emergency notification.

[1370] 2. Sending emergency notifications

[1371] Server: Generates a message for emergency notification and sends it to pre-registered contacts (family members and welfare agencies).

[1372] Examples:

[1373] Generative AI: Detects when a child loses consciousness.

[1374] Server: Send emergency notification to family: "Your child has lost consciousness. Please check the scene immediately."

[1375] Use of emotion engine

[1376] sentiment analysis

[1377] Device: Analyzes emotions from the user's voice and text.

[1378] Server: The emotion engine detects the user's emotional state from their speaking style and tone of voice, and reflects this in the generative AI.

[1379] Generative AI: Based on detected emotional information, it generates more appropriate responses that take into account the user's psychological state.

[1380] Examples:

[1381] User (child): "My friend said something bad about me at school today."

[1382] Emotion Engine: Detects sadness from the tone and content of the user's voice.

[1383] Generating AI: "That's tough, but you're a wonderful person," it encourages.

[1384] Emotional abnormality monitoring

[1385] Generative AI: Continuously monitors the user's emotional state and sends an emergency notification if an abnormal state is detected.

[1386] Examples:

[1387] The user (child) frequently makes statements that indicate sadness or despair.

[1388] Emotion Engine: Detects emotional anomalies that exceed the normal range.

[1389] Generative AI: Detects an abnormality and the server sends an emergency notification: "There is something abnormal in the child's emotional state. Please check immediately."

[1390] The system's program records detailed interactions between parents and their intellectually disabled children, and by training the generative AI, it provides ongoing support to children even after the parents have passed away. Furthermore, by incorporating an emotion engine, it can recognize the user's emotional state and provide more appropriate support. This allows for a rapid response in emergencies, providing a safe and secure environment for children.

[1391] The processing flow will be explained below.

[1392] Data collection and learning phase

[1393] Step 1:

[1394] Users use devices such as smartphones and tablets to record and record conversations and actions between parents and children.

[1395] Step 2:

[1396] The device sends the recorded data to the server in real time.

[1397] Step 3:

[1398] The server preprocesses the received data, converting the audio data to text, removing noise, trimming unnecessary parts, and extracting important information.

[1399] Step 4:

[1400] The server adds metadata (time, location, emotion tags, etc.) to the preprocessed data and stores it in a database.

[1401] Step 5:

[1402] The server inputs the stored data into the Generative AI's learning algorithm, which learns characteristics from parent-child interactions and builds a model to generate future responses.

[1403] Communication phase using generative AI

[1404] Step 6:

[1405] In the course of daily interactions, the user (child) asks questions such as, "What shall we do today?"

[1406] Step 7:

[1407] The device records the user's voice and converts it into text data (voice recognition function).

[1408] Step 8:

[1409] The terminal transmits the converted text data to the server.

[1410] Step 9:

[1411] The server inputs the received text data into the generation AI, which then generates an appropriate response.

[1412] Step 10:

[1413] The server transmits the generated response to the terminal as text data.

[1414] Step 11:

[1415] The terminal converts the text data into speech and plays it back to the user.

[1416] Emergency Notification Phase

[1417] Step 12:

[1418] The generative AI monitors a child's daily behavior and detects any unusual behavior (e.g., suddenly collapsing).

[1419] Step 13:

[1420] When the generative AI detects abnormal behavior, it sends a signal to the server that triggers an emergency notification.

[1421] Step 14:

[1422] The server generates the content of the emergency notification and sends it to pre-registered family members and welfare agencies.

[1423] Step 15:

[1424] The user (family or welfare agency) receives a notification and promptly goes to the scene or takes the necessary action (e.g., calls 119).

[1425] Use of emotion engine

[1426] sentiment analysis

[1427] Step 16:

[1428] The device collects the user's voice and text data and sends it to the emotion engine.

[1429] Step 17:

[1430] The server uses an emotion engine to analyze the user's emotional state from the tone of voice and text, for example, identifying emotions such as joy, sadness, and anger from the tone of voice.

[1431] Step 18:

[1432] The generative AI generates an appropriate response based on the emotional information obtained from the emotion engine.

[1433] Examples:

[1434] User (child): "My friend said something bad about me at school today."

[1435] Emotion Engine: Detects sadness from the tone and content of the user's voice.

[1436] Generating AI: "That's tough, but you're a wonderful person," it encourages.

[1437] Emotional abnormality monitoring

[1438] Step 19:

[1439] The generative AI monitors the user's emotional state in real time, and if negative emotions are detected repeatedly, it will be deemed an abnormality.

[1440] Step 20:

[1441] When the generative AI detects an emotional anomaly, it sends a signal to the server that triggers an emergency notification.

[1442] Step 21:

[1443] The server generates emergency notifications about emotional abnormalities and sends them to pre-registered family members and welfare agencies.

[1444] Examples:

[1445] The user (child) frequently makes statements that express sadness and despair.

[1446] Emotion Engine: Detects abnormal emotional states that exceed the normal range.

[1447] Generative AI: Detects an abnormality and the server sends an emergency notification: "There is something abnormal in the child's emotional state. Please check immediately."

[1448] This system records detailed daily interactions between parents and their intellectually disabled children, and uses this information to train a generative AI, providing ongoing support for the child even after the parents have passed away. It also incorporates an emotion engine to recognize the user's emotional state and provide more appropriate support. It can respond quickly in emergencies, providing a safe and secure environment for children.

[1449] Example 2

[1450] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1451] There is a need to provide an environment where children with intellectual disabilities can live safely even after their parents pass away. However, conventional systems have difficulty completely replacing the role of parents, and are particularly limited in terms of communication between parents and emergency response. Furthermore, they lack the ability to appropriately understand and respond to children's emotional states, making it difficult to maintain a sense of psychological security for children.

[1452] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1453] In this invention, the server includes a means for recording interactions between a parent and their intellectually disabled child, a means for preprocessing the recorded data and training the generation AI, a means for the generation AI to communicate with the child after the parent's death, a means for the generation AI to monitor the child's behavior and detect abnormalities, a means for sending an emergency notification to contacts when an abnormality is detected, a means for analyzing the user's emotions, and a means for the generation AI to generate an appropriate response based on the analyzed emotional information. This allows the child to receive continuous support even after parental support is no longer available, and allows for rapid response in emergencies. Furthermore, the emotion engine can be used to understand the child's emotional state and provide appropriate psychological support.

[1454] "Recording means" means any device or software that collects and stores in audio, text, and video form communications between a parent and their child with an intellectual disability.

[1455] "Preprocessing means" refers to a device or program that removes noise from recorded data, converts it into text, and otherwise converts it into a format suitable for learning by generative AI.

[1456] "Generative AI" refers to an artificial intelligence model that can learn from input data and generate natural, human-like responses.

[1457] "Means of communication" refers to the device or software that enables the generative AI to generate appropriate responses to questions posed by the child and return them to the child in voice or text format.

[1458] "Behavioral monitoring means" refers to a device or program that continuously records a child's daily behavior and detects deviations from normal patterns.

[1459] "Means for detecting anomalies" refers to devices or software that identify anomalous patterns of behavior from monitored behavioral data and issue an alert if such patterns are detected.

[1460] "Means for sending emergency notifications" refers to a device or program that sends emergency notifications via SMS, email, etc. to pre-registered contacts (family members or welfare agencies) when an abnormality is detected.

[1461] "Means for analyzing emotions" refers to a device or software that analyzes the emotional state of a user from their voice or text and reflects the results in the generative AI's response.

[1462] "Means for generating an appropriate response" refers to a device or program that enables the generation AI to create a response that takes into account the user's psychological state based on analyzed emotional information and user input.

[1463] This invention is a system that records interactions between parents and intellectually disabled children and trains a generative AI to provide ongoing support for children even after their parents pass away. It also has the ability to monitor abnormal behavior and send prompt notifications in the event of an emergency. It can also analyze the user's emotions and provide appropriate responses based on that information.

[1464] System configuration

[1465] Data collection and learning phase

[1466] First, the user uses a device such as a smartphone or tablet to record conversations and behaviors between parents and their children in audio, text, and video formats. Recordings are made using the default recording or camera app. The recorded data is then sent in real time to a cloud server (e.g., AWS S3).

[1467] The server then preprocesses the received data. Audio data is converted to text using the Google Cloud Speech-to-Text API, and noise is removed using NLTK (Natural Language Toolkit). Specific frames of the video data are extracted using FFmpeg and saved as images. Metadata (e.g., recording date and time, recording location, emotion tags) is added to the preprocessed data, and it is then stored in a NoSQL database such as MongoDB.

[1468] The stored data is then fed into a generative AI (such as OpenAI's GPT-3) to learn the interaction patterns between parents and children. In the process, the generative AI will be able to mimic the support provided by parents.

[1469] Communication phase using generative AI

[1470] Users (children) can ask a device such as a tablet, "What shall we do today?" in their daily lives. The question is recorded on the device and converted into text data using the Google Cloud Speech-to-Text API. The converted text data is sent to a server and input into the generation AI.

[1471] The server generates an appropriate response using the AI ​​and sends it back to the device. The user is then told a voice response, using Amazon Polly for speech synthesis.

[1472] Examples:

[1473] User (child): "What shall we do today?"

[1474] Generative AI: "How about going to the park today?"

[1475] Emergency Notification Phase

[1476] The AI ​​monitors the child's daily activities and detects any unusual behavior. For example, if a child stays in the same position for a long time, it is detected as abnormal behavior. When abnormal behavior is detected, the AI ​​sends a signal to the server that triggers an emergency notification.

[1477] The server sends emergency notifications via SMS or email to pre-registered contacts (family members or welfare agencies) using services such as AWS SNS (Simple Notification Service).

[1478] Examples:

[1479] Generative AI: Detects when a child loses consciousness.

[1480] Server: Send emergency notification to family: "Your child has lost consciousness. Please check the scene immediately."

[1481] Use of emotion engine

[1482] The user's voice and text are analyzed for emotional state using IBM Watson Tone Analyzer. The server uses an emotion engine to detect emotional information from the user's speaking style and tone of writing, and reflects that information in the generation AI. Based on the detected emotional information, the generation AI generates a more appropriate response that takes the user's psychological state into consideration.

[1483] Examples:

[1484] User (child): "My friend said something bad about me at school today."

[1485] Emotion Engine: Detects sadness from the tone and content of the user's voice.

[1486] Generating AI: "That's tough, but you're a wonderful person," it encourages.

[1487] The emotion engine also continuously monitors the user's emotional state and provides emergency notifications if it detects an abnormal state.

[1488] Examples:

[1489] The user (child) frequently makes statements that indicate sadness or despair.

[1490] Emotion Engine: Detects emotional anomalies that exceed the normal range.

[1491] Generative AI: An abnormality is detected and the server sends an emergency notification: "There is something abnormal in the child's emotional state. Please check immediately."

[1492] As described above, this system records detailed interactions between parents and their intellectually disabled children and trains the generative AI to provide ongoing support for the child even after the parents have passed away. Furthermore, by combining it with an emotion engine, it can recognize the user's emotional state and provide more appropriate support. This allows for a rapid response in emergencies, providing an environment where children can live in peace of mind.

[1493] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1494] Program processing flow

[1495] Data collection and learning phase

[1496] Step 1: User initiates interaction

[1497] Device: A smartphone or tablet is used to record conversations and actions between parents and children in audio, text, and video formats. The input is the conversation or action, and the output is an audio file, text file, or video file. Specifically, audio recording is done using a recording app, and video recording is done using a camera app.

[1498] Step 2: Preprocessing the data

[1499] Server: Receives data sent from the device in real time and performs preprocessing. The inputs are audio files, text files, and video files, and the output is processed text data and image data. Specifically, the Google Cloud Speech-to-Text API is used to convert audio data into text, and NLTK is used to remove noise. FFmpeg is used to extract specific frames from video data and save them as image files.

[1500] Step 3: Training the generative AI

[1501] Server: The preprocessed data is given metadata (recording date and time, recording location, emotion tag) and stored in a database (e.g., MongoDB). The input is the preprocessed data, and the output is a database entry with the metadata. The stored data is fed into a generative AI (e.g., OpenAI's GPT-3), which learns the interaction patterns between parents and children. Through this learning, the generative AI is able to imitate the support provided by the parent.

[1502] Communication phase using generative AI

[1503] Step 4: Daily Communication

[1504] User (child): Asks a device such as a tablet, "What shall we do today?" The input is the user's voice, and the output is a recorded audio file.

[1505] Terminal: Convert the recorded audio into text data using the Google Cloud Speech-to-Text API. The input is an audio file and the output is text data.

[1506] Terminal: Transmits converted text data to the server. The input is text data, and the output is data transmission to the server.

[1507] Server: Inputs the received text data into the generation AI and generates an appropriate response. The input is text data, and the output is the generated response text.

[1508] Server: Sends the generated response to the device and replies to the user via speech synthesis (using Amazon Polly). The input is the generated response text and the output is the audio response.

[1509] Examples:

[1510] User (child): "What shall we do today?"

[1511] Generative AI: "How about going to the park today?"

[1512] Emergency Notification Phase

[1513] Step 5: Monitor for abnormal behavior

[1514] Generative AI: Monitors a child's daily behavior and detects any unusual behavior. The input is daily behavior data, and the output is an alert for abnormal behavior detection. For example, it monitors long periods of inactivity or abnormal behavior patterns.

[1515] Step 6: Send emergency notifications

[1516] Server: When abnormal behavior is detected, it generates an emergency notification and sends it to pre-registered contacts (family members or welfare agencies). The input is the abnormal behavior detection alert, and the output is an emergency notification message to the contacts. The notification is sent using AWS SNS (Simple Notification Service).

[1517] Examples:

[1518] Generative AI: Detects when a child loses consciousness.

[1519] Server: Send emergency notification to family: "Your child has lost consciousness. Please check the scene immediately."

[1520] Use of emotion engine

[1521] Step 7: Sentiment analysis

[1522] Terminal: Analyzes emotions from user voice and text using IBM Watson Tone Analyzer. The input is an audio file or text data, and the output is analyzed emotional information.

[1523] Server: Using the emotion engine, the server detects the user's emotional state from their speaking style and tone of voice, and then feeds that information into the generative AI. The input is analyzed emotional information, and the output is text data with emotion tags.

[1524] Step 8: Generate an appropriate response

[1525] Generative AI: Generates more appropriate responses based on detected emotional information. The input is emotion-tagged text data, and the output is an emotion-sensitive response.

[1526] Examples:

[1527] User (child): "My friend said something bad about me at school today."

[1528] Emotion Engine: Detects sadness from the tone and content of the user's voice.

[1529] Generating AI: "That's tough, but you're a wonderful person," it encourages.

[1530] Step 9: Monitor for emotional abnormalities

[1531] Generative AI: Continuously monitors the user's emotional state and sends an emergency notification if an abnormality is detected. The input is emotional state data, and the output is an alert for anomaly detection. A specific example is when a user frequently makes statements that express sadness or despair.

[1532] Server: Detects anomalies and sends emergency notifications. The input is an anomaly detection alert, and the output is an emergency notification message.

[1533] Examples:

[1534] Emotion Engine: Detects emotional anomalies that exceed the normal range.

[1535] Generative AI: An abnormality is detected and the server sends an emergency notification: "There is something abnormal in the child's emotional state. Please check immediately."

[1536] (Application example 2)

[1537] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1538] Conventional monitoring systems are not specialized for people with intellectual disabilities or the elderly, and may not respond appropriately in emergencies. Furthermore, there is a lack of systems that can detect changes in a user's emotional state in real time and provide appropriate responses. Therefore, new technology is needed to balance user safety and mental care.

[1539] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1540] In this invention, the server includes: means for recording interactions between a parent and their intellectually disabled child; means for preprocessing the recorded data and training a generation AI; means for the generation AI to communicate with the child after the parent has passed away; means for the generation AI to monitor the child's abnormal behavior; means for issuing an emergency notification if an abnormality is detected; means for recognizing the user's voice and converting it into text; means for analyzing the emotions of the converted text data using an emotion engine; means for generating an appropriate response based on the analyzed emotional data; means for replying to the user with the generated response by voice; and means for detecting an abnormal emotional state through emotion analysis and issuing an emergency notification. This enables both daily safety management and psychological care for the user.

[1541] An "interaction recording device" is any device or method that records, in audio, text, or video format, the conversations or activities between a parent and their child with an intellectual disability.

[1542] "Preprocessing" refers to the process of converting recorded data into text, removing noise, and extracting important information.

[1543] "Generative AI" is an artificial intelligence that uses machine learning models to mimic the interactions between parents and their intellectually disabled children, enabling continuous communication.

[1544] "Means of communication" refers to the ability to use generative AI to converse and interact with children.

[1545] The "means of monitoring abnormal behavior" is a function that allows the generating AI to monitor a child's daily behavior and detect any behavior that is out of the ordinary.

[1546] The "means for sending emergency notifications" is a function that sends an emergency message to pre-registered family members and welfare organizations when an abnormality is detected.

[1547] "Means for recognizing voice and converting it into text" refers to technology that analyzes the voice spoken by the user and converts it into text data.

[1548] An "emotion engine" is an artificial intelligence model that analyzes emotions from a user's voice and text.

[1549] "Means for generating an appropriate response" refers to the function by which the generative AI creates an appropriate response or reply based on the analyzed emotional data.

[1550] "Means for replying by voice" refers to a technology that uses voice synthesis technology to convey the generated response to the user by voice.

[1551] "Means for detecting emotional state" refers to the function by which the emotion engine detects changes in the user's emotions in real time and provides that information to the generation AI.

[1552] The present invention is a system that records interactions between parents and their intellectually disabled children, formats the data through preprocessing, trains a generating AI, and then enables the generating AI to communicate with the child even after the parents have passed away, monitors abnormal behavior or emotional abnormalities, and sends emergency notifications. Detailed embodiments of the system are described below.

[1553] In this invention, the terminal can be a smartphone, tablet, intercom, etc., which can be used to communicate with the user and record voice. The server is installed on the cloud, where the recorded data is stored, preprocessed, and the generative AI model is trained.

[1554] Hardware and software used

[1555] Hardware: Smartphone, tablet, intercom, computer with microphone

[1556] Software: Python, SpeechRecognition, Transformers (Huggingface), Twilio API

[1557] Program Overview and Processing

[1558] 1. Recording interactions

[1559] The device records conversations and behaviors between parents and their children with intellectual disabilities in audio, text, or video format, and the recorded data is sent to a server in real time.

[1560] 2. Data Preprocessing

[1561] The server converts the received data into text, removes noise, and extracts important information. The preprocessed data is then annotated with metadata (such as time, location, and emotion tags) and stored in a database.

[1562] 3. Generative AI Training

[1563] The server inputs the preprocessed data into the generative AI's learning algorithm, which learns the interaction patterns between parents and children. The generative AI is then able to mimic the support provided by the parent and generate appropriate responses.

[1564] 4. Daily communication

[1565] The user (child) asks questions to the device in their daily life. The device records the user's voice and converts it into text data. This data is sent to the server, where the AI ​​generates an appropriate response and sends it to the device, which then replies to the user by voice.

[1566] Example: User (child): "What shall we do today?"

[1567] Generative AI: "How about going to the park today?"

[1568] 5. Emotion Recognition and Analysis

[1569] The device analyzes the user's emotions from their voice and text, and the server uses an emotion engine to detect the user's emotional state and reflects this in the generating AI.

[1570] Example: User (child): "My friend said something bad about me at school today."

[1571] Emotion Engine: Detects sadness from the tone and content of the user's voice.

[1572] Generator: "That's tough, but you're a wonderful being."

[1573] 6. Monitoring for abnormal behavior and emotional abnormalities

[1574] The AI ​​monitors the child's daily behavior and detects abnormal behavior. It also continuously monitors the user's emotional state and sends an emergency notification if it detects an abnormal state.

[1575] Example: Generative AI detects when a child loses consciousness.

[1576] Server: Send emergency notification to family: "Your child has lost consciousness. Please check the scene immediately."

[1577] Example prompt: "I'm not feeling too great today." Please assess the emotion in this text. Please advise how to respond on behalf of the parent.

[1578] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1579] Step 1:

[1580] The user speaks to the device, which records the user's voice.

[1581] Input: User's voice

[1582] Output: Recorded audio data

[1583] What it does: Uses the device's microphone to record what the user says.

[1584] Step 2:

[1585] The device converts recorded audio into text.

[1586] Input: Recorded audio data

[1587] Output: Text data

[1588] What it does: Uses speech recognition software (e.g., Google Speech Recognition API) to convert voice data into text.

[1589] Step 3:

[1590] The terminal transmits the text data to the server.

[1591] Input: Text data

[1592] Output: Text data sent to the server

[1593] Operation: Uses the device's communication function to send text data to a server in the cloud.

[1594] Step 4:

[1595] The server preprocesses the received text data, removing noise and extracting important information.

[1596] Input: Text data sent to the server

[1597] Output: Preprocessed text data

[1598] Operation: The server performs noise removal and important information extraction on the text data, and then adds metadata.

[1599] Step 5:

[1600] The server inputs the preprocessed text data into the generation AI, which generates an appropriate response.

[1601] Input: Preprocessed text data

[1602] Output: Response by the generative AI

[1603] How it works: The server inputs text data into the generative AI model, which generates appropriate responses based on the parent's response patterns learned by the model.

[1604] Step 6:

[1605] The server generates a response and sends it to the terminal.

[1606] Input: Response by generative AI

[1607] Output: The response sent to the device

[1608] Operation: The server sends the response generated by the generation AI to the terminal via its communication function.

[1609] Step 7:

[1610] The terminal returns the response received from the server to the user by voice.

[1611] Input: The response sent to the terminal

[1612] Output: A spoken response to the user

[1613] How it works: The device uses speech synthesis software to convert the text response received from the server into audio and play it back to the user.

[1614] Step 8:

[1615] The terminal transmits the text data to an emotion engine to analyze the user's emotions.

[1616] Input: Text data

[1617] Output: Emotion analysis results

[1618] How it works: The device uses an emotion engine (e.g., a Transformers emotion analysis model) to analyze the emotional state of text data.

[1619] Step 9:

[1620] The server detects emotional abnormalities and sends an emergency notification if any abnormalities are detected.

[1621] Input: Sentiment analysis results

[1622] Output: Send emergency notification

[1623] How it works: The server analyzes the emotion analysis results, and if it detects an abnormal emotional state, it sends an emergency notification to pre-registered contacts.

[1624] Step 10:

[1625] Emergency notifications reach registered contacts.

[1626] Input: Send emergency notification

[1627] Output: Contact notification

[1628] What it does: Your server uses a communication service, such as the Twilio API, to send an emergency message to registered contacts.

[1629] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1630] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1631] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1632] [Fourth embodiment]

[1633] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1634] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1635] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1636] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1637] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1638] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1639] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1640] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1641] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1642] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1643] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1644] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1645] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1646] This invention is a system that aims to provide an environment where parents of intellectually disabled children can live with peace of mind. This system records interactions between parents and their intellectually disabled children and trains a generative AI to learn from this data, allowing the generative AI to continue communicating with the child even after the parent has passed away. It also includes a function to monitor abnormal behavior and notify pre-registered family members and welfare organizations in the event of an emergency.

[1647] System Overview

[1648] Data collection and learning phase

[1649] 1. User initiates interaction

[1650] Devices: Family members' smartphones and tablets record daily interactions between parents and children in audio, text, and video formats.

[1651] Server: Receives data sent from the terminal and performs preprocessing.

[1652] 2. Data Preprocessing

[1653] Server: Converts received data into text, removes noise and unnecessary information, and structures the data by adding metadata such as time, location, and emotion tags.

[1654] 3. Generative AI Training

[1655] Server: The preprocessed data is input into the generative AI's learning model to learn the interaction patterns between parents and children, allowing the generative AI to mimic the support provided by parents.

[1656] Communication phase using generative AI

[1657] 1. Daily communication

[1658] User (child): Asks questions such as "What should we do today?" to a device such as a tablet.

[1659] Terminal: Converts this question into text data and sends it to the server.

[1660] Server: Inputs text data into the AI ​​generator and generates an appropriate response. This response is then sent back to the device and spoken back to the child.

[1661] Examples:

[1662] User (child): "What shall we do today?"

[1663] Generative AI: "How about going to the park today?"

[1664] Emergency Notification Phase

[1665] 1. Monitoring for Abnormal Behavior

[1666] Generative AI: Constantly monitors the child's behavior and condition, and if it detects behavior that deviates from the norm, it determines that it is abnormal.

[1667] Trigger emergency notifications if an anomaly is detected.

[1668] 2. Sending emergency notifications

[1669] Server: Generates emergency notification messages and sends them to pre-defined contacts (family members and welfare agencies).

[1670] Examples:

[1671] Generative AI: Detects when a child loses consciousness.

[1672] Server: Send emergency notification to family: "Your child has lost consciousness. Please check the scene immediately."

[1673] Example

[1674] In a concrete example, the process works as follows:

[1675] 1. User (child): Communicate regularly with the generated AI in daily life.

[1676] 2. Generative AI: Acts as a parent, providing appropriate advice and instructions.

[1677] 3. In case of abnormality: The generated AI detects abnormal behavior of the child and the server sends an emergency notification.

[1678] 4. Who to notify: Families and welfare agencies that receive the notification will respond promptly.

[1679] In this way, this invention is a system that provides an environment where children with intellectual disabilities can live safely even after their parents pass away. The data processing and generative AI functions of each phase make continuous support possible.

[1680] The processing flow will be explained below.

[1681] Data collection and learning phase

[1682] Step 1:

[1683] Users use devices such as smartphones and tablets to record and record conversations and actions between parents and children.

[1684] Step 2:

[1685] The device sends the recorded data to the server in real time.

[1686] Step 3:

[1687] The server preprocesses the received data, converting the audio data to text, removing noise, trimming unnecessary parts, and extracting important information.

[1688] Step 4:

[1689] The server adds metadata (time, location, emotion tags, etc.) to the preprocessed data and stores it in a database.

[1690] Step 5:

[1691] The server inputs the stored data into the Generative AI's learning algorithm, which learns characteristics from parent-child interactions and builds a model to generate future responses.

[1692] Communication phase using generative AI

[1693] Step 6:

[1694] In the course of daily interactions, the user (child) asks questions such as, "What shall we do today?"

[1695] Step 7:

[1696] The device records the user's voice and converts it into text data (voice recognition function).

[1697] Step 8:

[1698] The terminal transmits the converted text data to the server.

[1699] Step 9:

[1700] The server inputs the received text data into the generation AI, which then generates an appropriate response.

[1701] Step 10:

[1702] The server transmits the generated response to the terminal as text data.

[1703] Step 11:

[1704] The terminal converts the text data into speech and plays it back to the user.

[1705] Emergency Notification Phase

[1706] Step 12:

[1707] The generative AI monitors a child's daily behavior and detects any unusual behavior (e.g., suddenly collapsing).

[1708] Step 13:

[1709] When the generative AI detects abnormal behavior, it sends a signal to the server that triggers an emergency notification.

[1710] Step 14:

[1711] The server generates the content of the emergency notification and sends it to pre-registered family members and welfare agencies.

[1712] Step 15:

[1713] The user (family or welfare agency) receives a notification and promptly goes to the scene or takes the necessary action (e.g., calls 119).

[1714] The system's program records detailed daily interactions between parents and their intellectually disabled children, and trains the AI ​​to learn from these interactions, providing ongoing support for the child even after the parents have passed away. It also has a built-in notification system for rapid response in emergencies.

[1715] Example 1

[1716] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1717] There is a lack of support for intellectually disabled children to live safely after their parents pass away. There is no way to continue communicating with their children, monitoring their behavior, or responding in the event of an emergency, as parents did while they were alive, which raises the issue of the possibility that the safety and quality of life of the children may decline.

[1718] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1719] In this invention, the server includes a device that records interactions between parents and their intellectually disabled children, a device that preprocesses the recorded data and trains a generative AI model, a device that allows the generative AI model to communicate with the child after the parents pass away, a device that allows the generative AI model to monitor abnormal behavior in the child, and a device that issues an emergency notification if an abnormality is detected. This enables intellectually disabled children to continue living in a safe and stable environment even after the parents pass away.

[1720] "Devices that record interactions between parents and children with intellectual disabilities" means devices that collect and retain audio, text, and video communications between parents and children with intellectual disabilities.

[1721] The "device that preprocesses recorded data and trains a generative AI model" is a system that performs preprocessing such as noise removal, text conversion, and metadata addition based on recorded interaction data, and then uses that data to train a generative AI model.

[1722] The "device that allows a generative AI model to communicate with children after their parents have passed away" is a system that allows a generative AI model to continuously interact with and support children with intellectual disabilities based on data learned, even when their parents are no longer present.

[1723] The "device in which a generative AI model monitors children's abnormal behavior" is a system in which a generative AI model constantly monitors children's behavior and judges it as abnormal if it deviates from normal behavior patterns.

[1724] The "device that sends emergency notifications when an abnormality is detected" is a system that sends emergency notifications to pre-registered contacts when the AI ​​model detects abnormal behavior in a child.

[1725] "Means for storing preprocessed data in an information storage device and updating the configuration of the generative AI model" refers to a technology for restoring preprocessed data to a database or storage device and using that data to update the algorithms and parameters of the generative AI model.

[1726] "Means for sending emergency notifications to pre-registered family members and welfare agencies" refers to a system that sends emergency messages to pre-designated family members and welfare agencies when a child exhibits abnormal behavior.

[1727] This invention is a system that provides a safe and secure environment for parents of intellectually disabled children. The system records interactions between parents and their children and trains a generative AI model to learn from this data, enabling the model to continue communicating with the child after the parent's death. The system also includes a function in which the generative AI model monitors for abnormal behavior and notifies pre-registered family members and welfare agencies in the event of an emergency.

[1728] Data collection and learning phase

[1729] First, users (parents and children) use devices such as smartphones and tablets to carry out everyday interactions. These interactions are recorded in audio, text, and video formats. For example, a parent might say, "Good morning, how was your day?", and a child might respond, "I drew a picture at school today."

[1730] The device then sends the recorded data to a server, which converts the received data into text using voice recognition technology (e.g., Google Speech-to-Text API), removes noise, filters out unnecessary information, and extracts only the important information. The server also structures the data by adding metadata such as time, location, and emotion tags.

[1731] The server then feeds the pre-processed data into a generative AI model to learn parent-child interaction patterns, allowing the model to mimic the support provided by the parent.

[1732] Communication phase using generative AI

[1733] In everyday communication, a user (child) will ask a device such as a tablet, "What shall we do today?" This question is converted into text data using speech recognition technology and sent to a server. The server inputs the text data into a generative AI model to generate an appropriate response. This response is sent back to the device and is returned to the child as a voice using speech synthesis technology (e.g., Google Text-to-Speech API).

[1734] For example, if a child asks, "What shall we do today?", the generative AI model might suggest, "How about going to the park today?"

[1735] Emergency Notification Phase

[1736] The system's generative AI model constantly monitors a child's behavior and state, and if it detects behavior that deviates from the norm, it will determine that something is abnormal, such as if the child loses consciousness or exhibits unusual behavioral patterns.

[1737] If an abnormality is detected, the server generates an emergency notification message and sends it to pre-defined contacts (family members or welfare agencies). For example, it could send a message via SMS or email saying, "Your child has lost consciousness. Please check the scene immediately."

[1738] In this way, the present invention is a system that realizes functions such as recording interactions with parents, learning using a generative AI model, communication, monitoring abnormal behavior, and emergency notifications, in order to provide an environment where intellectually disabled children can live safely even after their parents have passed away.

[1739] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1740] Step 1:

[1741] User initiates interaction

[1742] Input: Daily interactions between users (parents and children) (voice, text, video)

[1743] Action: Parent and child engage in everyday interactions, such as parent saying, "Good morning, how was your day?" and child responding, "I drew a picture at school today."

[1744] Output: Interaction data is recorded on a smartphone or tablet

[1745] Step 2:

[1746] Sending data

[1747] Input: Recorded interaction data (audio, text, video)

[1748] How it works: The device sends the recorded data to a server. For example, audio data recorded using a tablet's microphone is uploaded to a cloud server via Wi-Fi.

[1749] Output: The server receives the data

[1750] Step 3:

[1751] Data Preprocessing

[1752] Input: Data of the interaction received by the server (audio, text, video)

[1753] How it works: The server uses speech recognition technology (e.g., Google Speech-to-Text API) to convert voice data into text, remove noise, and filter out unnecessary information. It also structures the data by adding metadata such as time, location, and emotion tags. For example, voice data such as "Good morning, how was your day?" is converted to text like "Good morning, how was your day?", and metadata such as October 10, 2023, and the living room are added.

[1754] Output: Preprocessed text data

[1755] Step 4:

[1756] Training generative AI models

[1757] Input: Preprocessed text data

[1758] How it works: The server feeds the preprocessed data into a generative AI model that learns parent-child interaction patterns, such as what response a parent gets when they say "good morning."

[1759] Output: A trained generative AI model

[1760] Step 5:

[1761] Daily communication

[1762] Input: Questions (voice) asked by the user (child) to the device

[1763] How it works: The user (child) asks a tablet or other device, "What shall we do today?" This question is converted into text data using speech recognition technology and sent to a server. The server inputs the text data into a generative AI model and generates an appropriate response. This response is sent back to the device, and is replies to the child as audio using speech synthesis technology (e.g., Google Text-to-Speech API).

[1764] How it works: When a child asks, "What shall we do today?", the generative AI model suggests, "How about going to the park today?"

[1765] Output: Voice response

[1766] Step 6:

[1767] Abnormal behavior monitoring

[1768] Input: User (child) behavior data

[1769] How it works: The generative AI model constantly monitors the child's behavior and state, and if it detects any behavior that deviates from the norm, it will judge it to be abnormal. For example, if the child suddenly stops moving or exhibits an unusual behavior pattern, it will detect an abnormality.

[1770] Output: Anomaly detection alert

[1771] Step 7:

[1772] Sending emergency notifications

[1773] Input: Anomaly detection alert, pre-registered contact information

[1774] What it does: The server generates an emergency notification message and sends it to pre-defined contacts (family or welfare agencies), for example, an SMS or email message saying "Your child has lost consciousness. Please check the scene immediately."

[1775] Output: Send emergency notification message

[1776] (Application example 1)

[1777] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1778] There is a need to reduce the anxiety that users who particularly need supervision, such as people with intellectual disabilities and the elderly, experience in their daily lives. It is important to provide an environment where users can live safely even when their parents or caregivers are not always nearby. There is also a need for a system that can respond quickly in the event of abnormal behavior or an emergency. Furthermore, there is a need to ensure that users do not feel lonely even after their parents pass away, and that they continue to receive the necessary support.

[1779] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1780] In this invention, the server includes a means for recording interactions between parents and persons with intellectual disabilities, a means for preprocessing the recorded data and having the generation AI learn from it, a means for the generation AI to communicate with the user after the parent has passed away, a means for the generation AI to monitor the user's abnormal behavior, a means for sending an emergency notification when an abnormality is detected, a means for monitoring the user's behavior and status using the camera and microphone of the smart device, a means for the generation AI to communicate with the user in real time, and a means for sending a notification to pre-set contacts when abnormal behavior is detected. This provides an environment where users can live safely and enables rapid response in emergencies.

[1781] A "means for recording interactions between a parent and a child with an intellectual disability" is a device or method for recording, by audio, text, video, or other means, the conversations or activities between a parent and a child with an intellectual disability.

[1782] "Means for preprocessing recorded data and training a generative AI" refers to a device or method for converting recorded data into text, performing preprocessing such as noise removal and adding metadata, and then training the generative AI model.

[1783] "Means for the generative AI to communicate with the user after the parents have passed away" refers to a device or method that allows the generative AI to interact with the user on behalf of the parents and provide necessary support and advice.

[1784] "Means for monitoring abnormal user behavior by a generation AI" refers to a device or method by which a generation AI monitors a user's daily behavior and detects abnormal behavior.

[1785] The "means for sending an emergency notification when an abnormality is detected" refers to a device or method for sending an emergency notification to pre-defined contacts when abnormal user behavior is detected.

[1786] "Means for monitoring a user's behavior and status using a camera or microphone on a smart device" refers to a device or method for recording and monitoring a user's behavior and status using a camera and microphone on a device such as a smartphone or head-mounted display.

[1787] "Means for the generation AI to communicate with the user in real time" refers to a device or method that enables the generation AI to respond immediately to the user's questions or requests and realize real-time dialogue.

[1788] "Means for sending notifications to pre-defined contacts when abnormal behavior is detected" refers to a device or method for sending notifications to pre-defined emergency contacts (such as family members or welfare agencies) when abnormal behavior of a user is detected.

[1789] The embodiments for implementing the invention are carried out in the following methods and steps.

[1790] Hardware and Software Use

[1791] 1. Hardware

[1792] Mobile devices such as smartphones and tablets

[1793] head-mounted display

[1794] Cloud server (e.g. AWS, Google Cloud)

[1795] Camera and microphone

[1796] 2. Software

[1797] Data collection applications (e.g., applications developed with Flutter)

[1798] Data preprocessing script (e.g., Python)

[1799] Natural language processing libraries (e.g. NLTK)

[1800] Generative AI models (e.g., OpenAI GPT-4)

[1801] Emergency notification API (e.g. Twilio API)

[1802] Anomaly detection algorithms (e.g., TensorFlow)

[1803] Data collection

[1804] The device (smartphone or tablet) records the interactions between parents and their children with intellectual disabilities in audio, text, and video formats, and the recorded data is sent to a cloud server where it is stored.

[1805] Data Preprocessing

[1806] The server converts the transmitted data into text, removes noise, and structures the data by adding metadata such as time, location, and emotion tags. This preprocessed data is then stored in a database.

[1807] Generative AI training

[1808] The server inputs the preprocessed data into the generative AI's learning model, learning the interaction patterns between parents and children, allowing the generative AI to provide appropriate support as a substitute parent.

[1809] Communication with users

[1810] The user (child) asks everyday questions or requests to the device, such as "What shall we do today?"

[1811] Examples:

[1812] User: "What should I do today?"

[1813] Generative AI: "How about going to the park today?"

[1814] The server converts the user's question into text data, inputs it into a generation AI, and generates an appropriate response. This response is then sent back to the device and replies to the user via voice.

[1815] Abnormal behavior monitoring and emergency notification

[1816] The AI ​​constantly monitors the user's behavior and status, and if it detects any behavior that deviates from the norm, it judges it to be an anomaly. If an anomaly is detected, the server will send an emergency notification to pre-defined contacts.

[1817] Examples:

[1818] Generative AI: Detects when the user has lost consciousness.

[1819] Server: Send emergency notification to family: "Your child has lost consciousness. Please check the scene immediately."

[1820] Prompt Sentence Examples

[1821] An example of a prompt sentence might be:

[1822] "Design a system that constantly monitors the user's mood and state and sends emergency notifications at the appropriate time."

[1823] "How can I automatically notify family members or designated contacts when abnormal behavior is detected?"

[1824] This system allows users to live with peace of mind and enables quick response in emergencies.

[1825] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1826] Step 1:

[1827] Data collection

[1828] Subject: Terminal

[1829] Input: Interactions between parents and their intellectually disabled children (audio, text, video)

[1830] Specific actions: Daily conversations and activities between parents and children are recorded using a data collection application on a smartphone or tablet.

[1831] Data processing and calculation: Recorded audio data and captured video data are sent to a cloud server.

[1832] Output: Recorded audio and video data is saved on a cloud server.

[1833] Step 2:

[1834] Data Preprocessing

[1835] Subject: Server

[1836] Input: Recorded audio and video data

[1837] Specific operations: Convert the data into text (speech recognition), remove noise, and add appropriate metadata (date, time, location, emotion tags, etc.).

[1838] Data processing and computation: Convert the audio data to text using a natural language processing library (e.g., NLTK) and update the database to add metadata.

[1839] Output: Preprocessed text data and structured metadata

[1840] Step 3:

[1841] Generative AI training

[1842] Subject: Server

[1843] Input: Preprocessed text data, structured metadata

[1844] Specific operation: The preprocessed data is input into a generative AI model (e.g., OpenAI GPT-4) to learn the interaction patterns between parents and children.

[1845] Data processing and calculation: Preprocessing data is fed into the learning model in large quantities, allowing the generative AI to recognize and learn patterns, enabling the AI ​​model to provide appropriate support in place of a parent.

[1846] Output: Trained generative AI model

[1847] Step 4:

[1848] Communication with users

[1849] Subject: Terminal

[1850] Input: User question or request (voice, text)

[1851] Specific behavior: The user asks a question or speaks to the device. Example: "What shall we do today?"

[1852] Data processing and calculation: The voice recognition function converts the user's question into text data, and this text data is sent to the cloud server.

[1853] Output: The user's question text data is sent to the generation AI.

[1854] Step 5:

[1855] Response generation by generative AI

[1856] Subject: Server

[1857] Input: User question text data

[1858] Specific operation: The generation AI generates an appropriate response based on the user's question text data.

[1859] Data processing and calculation: Based on the patterns learned by the generative AI, appropriate answers are generated to the user's questions.

[1860] Output: Generated response text data

[1861] Step 6:

[1862] Providing a response

[1863] Subject: Terminal

[1864] Input: Generated response text data

[1865] Specific operation: The generated response text data is converted into speech and output to the user. Example: "How about going to the park today?"

[1866] Data processing and calculation: Convert text into audio data using a speech conversion engine (e.g., Google Text-to-Speech).

[1867] Output: A spoken response to the user

[1868] Step 7:

[1869] Abnormal behavior monitoring

[1870] Subject: Generation AI

[1871] Input: User behavior and state data (camera, microphone)

[1872] Specific behavior: The generative AI monitors the user's real-time behavior and state and detects behavior that deviates from the norm.

[1873] Data processing and calculation: Identify abnormal behavior using an abnormal behavior detection algorithm (e.g., TensorFlow).

[1874] Output: Anomaly notification data when abnormal behavior is detected

[1875] Step 8:

[1876] Sending emergency notifications

[1877] Subject: Server

[1878] Input: Error notification data

[1879] Specific behavior: When an abnormality is detected, an emergency notification will be sent to pre-defined contacts. For example, "Your child has lost consciousness. Please check the scene immediately."

[1880] Data processing and calculation: Use emergency notification APIs (e.g., Twilio API) to send notifications about abnormal behavior to pre-registered contacts.

[1881] Output: Emergency notification message

[1882] This processing flow allows users to live with peace of mind and provides a system that can respond quickly in emergencies.

[1883] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1884] This invention is a system that provides a safe and secure environment for parents of intellectually disabled children. This system records interactions between parents and their intellectually disabled children, and trains a generation AI to learn from them, enabling the generation AI to continue communicating with the children even after the parents have passed away. The generation AI also monitors the child's abnormal behavior and notifies pre-registered family members and welfare agencies in the event of an emergency. Furthermore, this invention incorporates an emotion engine that recognizes the user's emotions, allowing the generation AI to understand the user's emotional state and provide appropriate responses.

[1885] System Overview

[1886] Data collection and learning phase

[1887] 1. User initiates interaction

[1888] Device: Records conversations and actions between parents and children in audio, text, and video formats via smartphones and tablets used by family members.

[1889] Server: Receives data sent from the terminal in real time and performs preprocessing.

[1890] 2. Data Preprocessing

[1891] Server: Converts received data into text, removes noise, trims unnecessary parts, and extracts important information.

[1892] Server: Adds metadata (time, location, emotion tags, etc.) to the preprocessed data and stores it in a database.

[1893] 3. Generative AI Training

[1894] Server: The preprocessed data is input into the Generative AI's learning algorithm, which learns the interaction patterns between parents and children. This allows the Generative AI to imitate the support provided by parents.

[1895] Communication phase using generative AI

[1896] 1. Daily communication

[1897] User (child): In everyday life, the user asks questions such as "What should I do today?" to a device such as a tablet.

[1898] Terminal: Records the user's voice and converts it into text data (voice recognition function).

[1899] Terminal: Sends the converted text data to the server.

[1900] Server: The received text data is input into the generation AI, which generates an appropriate response. This response is sent back to the device and replied to the user by voice.

[1901] Examples:

[1902] User (child): "What shall we do today?"

[1903] Generative AI: "How about going to the park today?"

[1904] Emergency Notification Phase

[1905] 1. Monitoring for Abnormal Behavior

[1906] Generative AI: Monitors children's daily behavior and detects unusual behavior.

[1907] Generative AI: When it detects abnormal behavior, it sends a signal to the server that triggers an emergency notification.

[1908] 2. Sending emergency notifications

[1909] Server: Generates a message for emergency notification and sends it to pre-registered contacts (family members and welfare agencies).

[1910] Examples:

[1911] Generative AI: Detects when a child loses consciousness.

[1912] Server: Send emergency notification to family: "Your child has lost consciousness. Please check the scene immediately."

[1913] Use of emotion engine

[1914] sentiment analysis

[1915] Device: Analyzes emotions from the user's voice and text.

[1916] Server: The emotion engine detects the user's emotional state from their speaking style and tone of voice, and reflects this in the generative AI.

[1917] Generative AI: Based on detected emotional information, it generates more appropriate responses that take into account the user's psychological state.

[1918] Examples:

[1919] User (child): "My friend said something bad about me at school today."

[1920] Emotion Engine: Detects sadness from the tone and content of the user's voice.

[1921] Generating AI: "That's tough, but you're a wonderful person," it encourages.

[1922] Emotional abnormality monitoring

[1923] Generative AI: Continuously monitors the user's emotional state and sends an emergency notification if an abnormal state is detected.

[1924] Examples:

[1925] The user (child) frequently makes statements that indicate sadness or despair.

[1926] Emotion Engine: Detects emotional anomalies that exceed the normal range.

[1927] Generative AI: Detects an abnormality and the server sends an emergency notification: "There is something abnormal in the child's emotional state. Please check immediately."

[1928] The system's program records detailed interactions between parents and their intellectually disabled children, and by training the generative AI, it provides ongoing support to children even after the parents have passed away. Furthermore, by incorporating an emotion engine, it can recognize the user's emotional state and provide more appropriate support. This allows for a rapid response in emergencies, providing a safe and secure environment for children.

[1929] The processing flow will be explained below.

[1930] Data collection and learning phase

[1931] Step 1:

[1932] Users use devices such as smartphones and tablets to record and record conversations and actions between parents and children.

[1933] Step 2:

[1934] The device sends the recorded data to the server in real time.

[1935] Step 3:

[1936] The server preprocesses the received data, converting the audio data to text, removing noise, trimming unnecessary parts, and extracting important information.

[1937] Step 4:

[1938] The server adds metadata (time, location, emotion tags, etc.) to the preprocessed data and stores it in a database.

[1939] Step 5:

[1940] The server inputs the stored data into the Generative AI's learning algorithm, which learns characteristics from parent-child interactions and builds a model to generate future responses.

[1941] Communication phase using generative AI

[1942] Step 6:

[1943] In the course of daily interactions, the user (child) asks questions such as, "What shall we do today?"

[1944] Step 7:

[1945] The device records the user's voice and converts it into text data (voice recognition function).

[1946] Step 8:

[1947] The terminal transmits the converted text data to the server.

[1948] Step 9:

[1949] The server inputs the received text data into the generation AI, which then generates an appropriate response.

[1950] Step 10:

[1951] The server transmits the generated response to the terminal as text data.

[1952] Step 11:

[1953] The terminal converts the text data into speech and plays it back to the user.

[1954] Emergency Notification Phase

[1955] Step 12:

[1956] The generative AI monitors a child's daily behavior and detects any unusual behavior (e.g., suddenly collapsing).

[1957] Step 13:

[1958] When the generative AI detects abnormal behavior, it sends a signal to the server that triggers an emergency notification.

[1959] Step 14:

[1960] The server generates the content of the emergency notification and sends it to pre-registered family members and welfare agencies.

[1961] Step 15:

[1962] The user (family or welfare agency) receives a notification and promptly goes to the scene or takes the necessary action (e.g., calls 119).

[1963] Use of emotion engine

[1964] sentiment analysis

[1965] Step 16:

[1966] The device collects the user's voice and text data and sends it to the emotion engine.

[1967] Step 17:

[1968] The server uses an emotion engine to analyze the user's emotional state from the tone of voice and text, for example, identifying emotions such as joy, sadness, and anger from the tone of voice.

[1969] Step 18:

[1970] The generative AI generates an appropriate response based on the emotional information obtained from the emotion engine.

[1971] Examples:

[1972] User (child): "My friend said something bad about me at school today."

[1973] Emotion Engine: Detects sadness from the tone and content of the user's voice.

[1974] Generating AI: "That's tough, but you're a wonderful person," it encourages.

[1975] Emotional abnormality monitoring

[1976] Step 19:

[1977] The generative AI monitors the user's emotional state in real time, and if negative emotions are detected repeatedly, it will be deemed an abnormality.

[1978] Step 20:

[1979] When the generative AI detects an emotional anomaly, it sends a signal to the server that triggers an emergency notification.

[1980] Step 21:

[1981] The server generates emergency notifications about emotional abnormalities and sends them to pre-registered family members and welfare agencies.

[1982] Examples:

[1983] The user (child) frequently makes statements that express sadness and despair.

[1984] Emotion Engine: Detects abnormal emotional states that exceed the normal range.

[1985] Generative AI: Detects an abnormality and the server sends an emergency notification: "There is something abnormal in the child's emotional state. Please check immediately."

[1986] This system records detailed daily interactions between parents and their intellectually disabled children, and uses this information to train a generative AI, providing ongoing support for the child even after the parents have passed away. It also incorporates an emotion engine to recognize the user's emotional state and provide more appropriate support. It can respond quickly in emergencies, providing a safe and secure environment for children.

[1987] Example 2

[1988] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1989] There is a need to provide an environment where children with intellectual disabilities can live safely even after their parents pass away. However, conventional systems have difficulty completely replacing the role of parents, and are particularly limited in terms of communication between parents and emergency response. Furthermore, they lack the ability to appropriately understand and respond to children's emotional states, making it difficult to maintain a sense of psychological security for children.

[1990] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1991] In this invention, the server includes a means for recording interactions between a parent and their intellectually disabled child, a means for preprocessing the recorded data and training the generation AI, a means for the generation AI to communicate with the child after the parent's death, a means for the generation AI to monitor the child's behavior and detect abnormalities, a means for sending an emergency notification to contacts when an abnormality is detected, a means for analyzing the user's emotions, and a means for the generation AI to generate an appropriate response based on the analyzed emotional information. This allows the child to receive continuous support even after parental support is no longer available, and allows for rapid response in emergencies. Furthermore, the emotion engine can be used to understand the child's emotional state and provide appropriate psychological support.

[1992] "Recording means" means any device or software that collects and stores in audio, text, and video form communications between a parent and their child with an intellectual disability.

[1993] "Preprocessing means" refers to a device or program that removes noise from recorded data, converts it into text, and otherwise converts it into a format suitable for learning by generative AI.

[1994] "Generative AI" refers to an artificial intelligence model that can learn from input data and generate natural, human-like responses.

[1995] "Means of communication" refers to the device or software that enables the generative AI to generate appropriate responses to questions posed by the child and return them to the child in voice or text format.

[1996] "Behavioral monitoring means" refers to a device or program that continuously records a child's daily behavior and detects deviations from normal patterns.

[1997] "Means for detecting anomalies" refers to devices or software that identify anomalous patterns of behavior from monitored behavioral data and issue an alert if such patterns are detected.

[1998] "Means for sending emergency notifications" refers to a device or program that sends emergency notifications via SMS, email, etc. to pre-registered contacts (family members or welfare agencies) when an abnormality is detected.

[1999] "Means for analyzing emotions" refers to a device or software that analyzes the emotional state of a user from their voice or text and reflects the results in the generative AI's response.

[2000] "Means for generating an appropriate response" refers to a device or program that enables the generation AI to create a response that takes into account the user's psychological state based on analyzed emotional information and user input.

[2001] This invention is a system that records interactions between parents and intellectually disabled children and trains a generative AI to provide ongoing support for children even after their parents pass away. It also has the ability to monitor abnormal behavior and send prompt notifications in the event of an emergency. It can also analyze the user's emotions and provide appropriate responses based on that information.

[2002] System configuration

[2003] Data collection and learning phase

[2004] First, the user uses a device such as a smartphone or tablet to record conversations and behaviors between parents and their children in audio, text, and video formats. Recordings are made using the default recording or camera app. The recorded data is then sent in real time to a cloud server (e.g., AWS S3).

[2005] The server then preprocesses the received data. Audio data is converted to text using the Google Cloud Speech-to-Text API, and noise is removed using NLTK (Natural Language Toolkit). Specific frames of the video data are extracted using FFmpeg and saved as images. Metadata (e.g., recording date and time, recording location, emotion tags) is added to the preprocessed data, and it is then stored in a NoSQL database such as MongoDB.

[2006] The stored data is then fed into a generative AI (such as OpenAI's GPT-3) to learn the interaction patterns between parents and children. In the process, the generative AI will be able to mimic the support provided by parents.

[2007] Communication phase using generative AI

[2008] Users (children) can ask a device such as a tablet, "What shall we do today?" in their daily lives. The question is recorded on the device and converted into text data using the Google Cloud Speech-to-Text API. The converted text data is sent to a server and input into the generation AI.

[2009] The server generates an appropriate response using the AI ​​and sends it back to the device. The user is then told a voice response, using Amazon Polly for speech synthesis.

[2010] Examples:

[2011] User (child): "What shall we do today?"

[2012] Generative AI: "How about going to the park today?"

[2013] Emergency Notification Phase

[2014] The AI ​​monitors the child's daily activities and detects any unusual behavior. For example, if a child stays in the same position for a long time, it is detected as abnormal behavior. When abnormal behavior is detected, the AI ​​sends a signal to the server that triggers an emergency notification.

[2015] The server sends emergency notifications via SMS or email to pre-registered contacts (family members or welfare agencies) using services such as AWS SNS (Simple Notification Service).

[2016] Examples:

[2017] Generative AI: Detects when a child loses consciousness.

[2018] Server: Send emergency notification to family: "Your child has lost consciousness. Please check the scene immediately."

[2019] Use of emotion engine

[2020] The user's voice and text are analyzed for emotional state using IBM Watson Tone Analyzer. The server uses an emotion engine to detect emotional information from the user's speaking style and tone of writing, and reflects that information in the generation AI. Based on the detected emotional information, the generation AI generates a more appropriate response that takes the user's psychological state into consideration.

[2021] Examples:

[2022] User (child): "My friend said something bad about me at school today."

[2023] Emotion Engine: Detects sadness from the tone and content of the user's voice.

[2024] Generating AI: "That's tough, but you're a wonderful person," it encourages.

[2025] The emotion engine also continuously monitors the user's emotional state and provides emergency notifications if it detects an abnormal state.

[2026] Examples:

[2027] The user (child) frequently makes statements that indicate sadness or despair.

[2028] Emotion Engine: Detects emotional anomalies that exceed the normal range.

[2029] Generative AI: An abnormality is detected and the server sends an emergency notification: "There is something abnormal in the child's emotional state. Please check immediately."

[2030] As described above, this system records detailed interactions between parents and their intellectually disabled children and trains the generative AI to provide ongoing support for the child even after the parents have passed away. Furthermore, by combining it with an emotion engine, it can recognize the user's emotional state and provide more appropriate support. This allows for a rapid response in emergencies, providing an environment where children can live in peace of mind.

[2031] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2032] Program processing flow

[2033] Data collection and learning phase

[2034] Step 1: User initiates interaction

[2035] Device: A smartphone or tablet is used to record conversations and actions between parents and children in audio, text, and video formats. The input is the conversation or action, and the output is an audio file, text file, or video file. Specifically, audio recording is done using a recording app, and video recording is done using a camera app.

[2036] Step 2: Preprocessing the data

[2037] Server: Receives data sent from the device in real time and performs preprocessing. The inputs are audio files, text files, and video files, and the output is processed text data and image data. Specifically, the Google Cloud Speech-to-Text API is used to convert audio data into text, and NLTK is used to remove noise. FFmpeg is used to extract specific frames from video data and save them as image files.

[2038] Step 3: Training the generative AI

[2039] Server: The preprocessed data is given metadata (recording date and time, recording location, emotion tag) and stored in a database (e.g., MongoDB). The input is the preprocessed data, and the output is a database entry with the metadata. The stored data is fed into a generative AI (e.g., OpenAI's GPT-3), which learns the interaction patterns between parents and children. Through this learning, the generative AI is able to imitate the support provided by the parent.

[2040] Communication phase using generative AI

[2041] Step 4: Daily Communication

[2042] User (child): Asks a device such as a tablet, "What shall we do today?" The input is the user's voice, and the output is a recorded audio file.

[2043] Terminal: Convert the recorded audio into text data using the Google Cloud Speech-to-Text API. The input is an audio file and the output is text data.

[2044] Terminal: Transmits converted text data to the server. The input is text data, and the output is data transmission to the server.

[2045] Server: Inputs the received text data into the generation AI and generates an appropriate response. The input is text data, and the output is the generated response text.

[2046] Server: Sends the generated response to the device and replies to the user via speech synthesis (using Amazon Polly). The input is the generated response text and the output is the audio response.

[2047] Examples:

[2048] User (child): "What shall we do today?"

[2049] Generative AI: "How about going to the park today?"

[2050] Emergency Notification Phase

[2051] Step 5: Monitor for abnormal behavior

[2052] Generative AI: Monitors a child's daily behavior and detects any unusual behavior. The input is daily behavior data, and the output is an alert for abnormal behavior detection. For example, it monitors long periods of inactivity or abnormal behavior patterns.

[2053] Step 6: Send emergency notifications

[2054] Server: When abnormal behavior is detected, it generates an emergency notification and sends it to pre-registered contacts (family members or welfare agencies). The input is the abnormal behavior detection alert, and the output is an emergency notification message to the contacts. The notification is sent using AWS SNS (Simple Notification Service).

[2055] Examples:

[2056] Generative AI: Detects when a child loses consciousness.

[2057] Server: Send emergency notification to family: "Your child has lost consciousness. Please check the scene immediately."

[2058] Use of emotion engine

[2059] Step 7: Sentiment analysis

[2060] Terminal: Analyzes emotions from user voice and text using IBM Watson Tone Analyzer. The input is an audio file or text data, and the output is analyzed emotional information.

[2061] Server: Using the emotion engine, the server detects the user's emotional state from their speaking style and tone of voice, and then feeds that information into the generative AI. The input is analyzed emotional information, and the output is text data with emotion tags.

[2062] Step 8: Generate an appropriate response

[2063] Generative AI: Generates more appropriate responses based on detected emotional information. The input is emotion-tagged text data, and the output is an emotion-sensitive response.

[2064] Examples:

[2065] User (child): "My friend said something bad about me at school today."

[2066] Emotion Engine: Detects sadness from the tone and content of the user's voice.

[2067] Generating AI: "That's tough, but you're a wonderful person," it encourages.

[2068] Step 9: Monitor for emotional abnormalities

[2069] Generative AI: Continuously monitors the user's emotional state and sends an emergency notification if an abnormality is detected. The input is emotional state data, and the output is an alert for anomaly detection. A specific example is when a user frequently makes statements that express sadness or despair.

[2070] Server: Detects anomalies and sends emergency notifications. The input is an anomaly detection alert, and the output is an emergency notification message.

[2071] Examples:

[2072] Emotion Engine: Detects emotional anomalies that exceed the normal range.

[2073] Generative AI: An abnormality is detected and the server sends an emergency notification: "There is something abnormal in the child's emotional state. Please check immediately."

[2074] (Application example 2)

[2075] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2076] Conventional monitoring systems are not specialized for people with intellectual disabilities or the elderly, and may not respond appropriately in emergencies. Furthermore, there is a lack of systems that can detect changes in a user's emotional state in real time and provide appropriate responses. Therefore, new technology is needed to balance user safety and mental care.

[2077] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[2078] In this invention, the server includes: means for recording interactions between a parent and their intellectually disabled child; means for preprocessing the recorded data and training a generation AI; means for the generation AI to communicate with the child after the parent has passed away; means for the generation AI to monitor the child's abnormal behavior; means for issuing an emergency notification if an abnormality is detected; means for recognizing the user's voice and converting it into text; means for analyzing the emotions of the converted text data using an emotion engine; means for generating an appropriate response based on the analyzed emotional data; means for replying to the user with the generated response by voice; and means for detecting an abnormal emotional state through emotion analysis and issuing an emergency notification. This enables both daily safety management and psychological care for the user.

[2079] An "interaction recording device" is any device or method that records, in audio, text, or video format, the conversations or activities between a parent and their child with an intellectual disability.

[2080] "Preprocessing" refers to the process of converting recorded data into text, removing noise, and extracting important information.

[2081] "Generative AI" is an artificial intelligence that uses machine learning models to mimic the interactions between parents and their intellectually disabled children, enabling continuous communication.

[2082] "Means of communication" refers to the ability to use generative AI to converse and interact with children.

[2083] The "means of monitoring abnormal behavior" is a function that allows the generating AI to monitor a child's daily behavior and detect any behavior that is out of the ordinary.

[2084] The "means for sending emergency notifications" is a function that sends an emergency message to pre-registered family members and welfare organizations when an abnormality is detected.

[2085] "Means for recognizing voice and converting it into text" refers to technology that analyzes the voice spoken by the user and converts it into text data.

[2086] An "emotion engine" is an artificial intelligence model that analyzes emotions from a user's voice and text.

[2087] "Means for generating an appropriate response" refers to the function by which the generative AI creates an appropriate response or reply based on the analyzed emotional data.

[2088] "Means for replying by voice" refers to a technology that uses voice synthesis technology to convey the generated response to the user by voice.

[2089] "Means for detecting emotional state" refers to the function by which the emotion engine detects changes in the user's emotions in real time and provides that information to the generation AI.

[2090] The present invention is a system that records interactions between parents and their intellectually disabled children, formats the data through preprocessing, trains a generating AI, and then enables the generating AI to communicate with the child even after the parents have passed away, monitors abnormal behavior or emotional abnormalities, and sends emergency notifications. Detailed embodiments of the system are described below.

[2091] In this invention, the terminal can be a smartphone, tablet, intercom, etc., which can be used to communicate with the user and record voice. The server is installed on the cloud, where the recorded data is stored, preprocessed, and the generative AI model is trained.

[2092] Hardware and software used

[2093] Hardware: Smartphone, tablet, intercom, computer with microphone

[2094] Software: Python, SpeechRecognition, Transformers (Huggingface), Twilio API

[2095] Program Overview and Processing

[2096] 1. Recording interactions

[2097] The device records conversations and behaviors between parents and their children with intellectual disabilities in audio, text, or video format, and the recorded data is sent to a server in real time.

[2098] 2. Data Preprocessing

[2099] The server converts the received data into text, removes noise, and extracts important information. The preprocessed data is then annotated with metadata (such as time, location, and emotion tags) and stored in a database.

[2100] 3. Generative AI Training

[2101] The server inputs the preprocessed data into the generative AI's learning algorithm, which learns the interaction patterns between parents and children. The generative AI is then able to mimic the support provided by the parent and generate appropriate responses.

[2102] 4. Daily communication

[2103] The user (child) asks questions to the device in their daily life. The device records the user's voice and converts it into text data. This data is sent to the server, where the AI ​​generates an appropriate response and sends it to the device, which then replies to the user by voice.

[2104] Example: User (child): "What shall we do today?"

[2105] Generative AI: "How about going to the park today?"

[2106] 5. Emotion Recognition and Analysis

[2107] The device analyzes the user's emotions from their voice and text, and the server uses an emotion engine to detect the user's emotional state and reflects this in the generating AI.

[2108] Example: User (child): "My friend said something bad about me at school today."

[2109] Emotion Engine: Detects sadness from the tone and content of the user's voice.

[2110] Generator: "That's tough, but you're a wonderful being."

[2111] 6. Monitoring for abnormal behavior and emotional abnormalities

[2112] The AI ​​monitors the child's daily behavior and detects abnormal behavior. It also continuously monitors the user's emotional state and sends an emergency notification if it detects an abnormal state.

[2113] Example: Generative AI detects when a child loses consciousness.

[2114] Server: Send emergency notification to family: "Your child has lost consciousness. Please check the scene immediately."

[2115] Example prompt: "I'm not feeling too great today." Please assess the emotion in this text. Please advise how to respond on behalf of the parent.

[2116] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2117] Step 1:

[2118] The user speaks to the device, which records the user's voice.

[2119] Input: User's voice

[2120] Output: Recorded audio data

[2121] What it does: Uses the device's microphone to record what the user says.

[2122] Step 2:

[2123] The device converts recorded audio into text.

[2124] Input: Recorded audio data

[2125] Output: Text data

[2126] What it does: Uses speech recognition software (e.g., Google Speech Recognition API) to convert voice data into text.

[2127] Step 3:

[2128] The terminal transmits the text data to the server.

[2129] Input: Text data

[2130] Output: Text data sent to the server

[2131] Operation: Uses the device's communication function to send text data to a server in the cloud.

[2132] Step 4:

[2133] The server preprocesses the received text data, removing noise and extracting important information.

[2134] Input: Text data sent to the server

[2135] Output: Preprocessed text data

[2136] Operation: The server performs noise removal and important information extraction on the text data, and then adds metadata.

[2137] Step 5:

[2138] The server inputs the preprocessed text data into the generation AI, which generates an appropriate response.

[2139] Input: Preprocessed text data

[2140] Output: Response by the generative AI

[2141] How it works: The server inputs text data into the generative AI model, which generates appropriate responses based on the parent's response patterns learned by the model.

[2142] Step 6:

[2143] The server generates a response and sends it to the terminal.

[2144] Input: Response by generative AI

[2145] Output: The response sent to the device

[2146] Operation: The server sends the response generated by the generation AI to the terminal via its communication function.

[2147] Step 7:

[2148] The terminal returns the response received from the server to the user by voice.

[2149] Input: The response sent to the terminal

[2150] Output: A spoken response to the user

[2151] How it works: The device uses speech synthesis software to convert the text response received from the server into audio and play it back to the user.

[2152] Step 8:

[2153] The terminal transmits the text data to an emotion engine to analyze the user's emotions.

[2154] Input: Text data

[2155] Output: Emotion analysis results

[2156] How it works: The device uses an emotion engine (e.g., a Transformers emotion analysis model) to analyze the emotional state of text data.

[2157] Step 9:

[2158] The server detects emotional abnormalities and sends an emergency notification if any abnormalities are detected.

[2159] Input: Sentiment analysis results

[2160] Output: Send emergency notification

[2161] How it works: The server analyzes the emotion analysis results, and if it detects an abnormal emotional state, it sends an emergency notification to pre-registered contacts.

[2162] Step 10:

[2163] Emergency notifications reach registered contacts.

[2164] Input: Send emergency notification

[2165] Output: Contact notification

[2166] What it does: Your server uses a communication service, such as the Twilio API, to send an emergency message to registered contacts.

[2167] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2168] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2169] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2170] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2171] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2172] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2173] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2174] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2175] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2176] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2177] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2178] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2179] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2180] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2181] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[2182] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[2183] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[2184] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[2185] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[2186] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[2187] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[2188] The following is further disclosed regarding the above embodiment.

[2189] (Claim 1)

[2190] a means of recording interactions between parents and their intellectually disabled children;

[2191] A means of preprocessing the recorded data and training the generative AI;

[2192] After the parents are gone, how can the AI ​​communicate with the child?

[2193] A means for the generative AI to monitor abnormal behavior in children,

[2194] A system that includes a means for providing emergency notification if an anomaly is detected.

[2195] (Claim 2)

[2196] 10. The system of claim 1, further comprising means for storing the preprocessed data in a database and updating a model of the generative AI.

[2197] (Claim 3)

[2198] 10. The system of claim 1, further comprising means for sending emergency notifications to pre-registered family members and welfare agencies.

[2199] "Example 1"

[2200] (Claim 1)

[2201] A device that records interactions between parents and their intellectually disabled children;

[2202] A device that preprocesses the recorded data and trains a generative AI model;

[2203] A device that allows a generative AI model to communicate with children after their parents have passed away,

[2204] A device that uses a generative AI model to monitor abnormal behavior in children,

[2205] A system including a device that provides emergency notification when an abnormality is detected.

[2206] (Claim 2)

[2207] 10. The system of claim 1, further comprising means for storing the preprocessed data in an information storage device and updating the configuration of the generative AI model.

[2208] (Claim 3)

[2209] 10. The system of claim 1, further comprising means for sending emergency notifications to pre-registered family members and welfare agencies.

[2210] "Application Example 1"

[2211] (Claim 1)

[2212] a means of recording interactions between parents and their intellectually disabled children;

[2213] A means of preprocessing the recorded data and training the generative AI;

[2214] After the parents are gone, how can the AI ​​communicate with the child?

[2215] A means for the generative AI to monitor abnormal behavior in children,

[2216] a means for providing an emergency notification when an abnormality is detected;

[2217] A means for monitoring the user's behavior and status using the camera and microphone of the smart device;

[2218] A m...

Claims

1. a means of recording interactions between parents and their intellectually disabled children; A means of preprocessing the recorded data and training the generative AI; After the parents are gone, how can the AI ​​communicate with the child? A means for the generative AI to monitor abnormal behavior in children, A system that includes a means for providing emergency notification if an anomaly is detected.

2. The system of claim 1 , further comprising means for storing the preprocessed data in a database and updating a model of the generative AI.

3. The system of claim 1 , further comprising means for sending emergency notifications to pre-registered family members and welfare agencies.

Images