system

Abstract

We provide the system. [Solution] Means for receiving audio data and video data from an acquisition device, A means for analyzing the aforementioned audio and video data to infer the subject's behavior or emotional state, A means of determining and proposing appropriate countermeasures based on the aforementioned inferences, A means for predicting the future state of the subject based on past data, Means for controlling a smart device that automatically executes the aforementioned appropriate countermeasures, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In recent years, as one of the main problems faced by caregivers in child-rearing, it is difficult to immediately understand the emotions and conditions of a baby and take appropriate actions. In particular, for modern caregivers who are required to balance daily child-rearing tasks with work and housework, it is important to predict the baby's condition and take preemptive actions. However, due to lack of experience and time constraints, this is often difficult to achieve. In addition, when the understanding of the baby's condition is shallow, there is also a problem that it is difficult for those who participate in the care other than relatives and caregivers to take appropriate actions.

Means for Solving the Problems

[0005] This invention provides a system that analyzes a baby's behavior and emotional state using audio and video data received from an acquisition device and proposes appropriate responses. Specifically, it includes means for analyzing audio and video data to infer the subject's behavior and emotional state. Furthermore, it reduces the burden on parents by determining appropriate countermeasures based on the inferred state and automatically executing them by controlling a smart device. It also enables more effective childcare support by predicting future states based on past data. In addition, it generates and provides a manual describing optimal methods for relatives who interact with the baby, thereby facilitating temporary childcare by relatives. This reduces stress from childcare and improves the childcare environment.

[0006] An "acquisition device" is a device that collects data such as audio and video and transmits it for later analysis.

[0007] "Audio data" refers to sound signals recorded by microphones or other sound acquisition devices, and is used to infer emotions and states through analysis.

[0008] "Video data" refers to the signal of images recorded by cameras or other video acquisition devices, and is used for analyzing movements and facial expressions.

[0009] "Analysis" is the process of extracting information from acquired data and assigning meaning to it according to a specific purpose.

[0010] "Behavioral or emotional state" refers to the psychological or physiological state judged from the actions and sounds exhibited by the subject, particularly a baby.

[0011] "Inference" is the process of logically considering and judging an unknown state based on the data obtained.

[0012] "Countermeasures" refer to the actions or means that should be taken to appropriately address the situation based on the inferred results obtained.

[0013] "Predicting future states" refers to predicting future states using a specific algorithm based on past data and current conditions.

[0014] A "smart device" refers to a digital device that is connected to the internet and can be controlled remotely.

[0015] "Control" refers to the operation of adjusting or changing the behavior of a device to a state that conforms to instructions. [Brief explanation of the drawing]

[0016] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when combined with an emotion engine. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when combined with an emotion engine.

Mode for Carrying Out the Invention

[0017] Hereinafter, an example of an embodiment of the system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be one arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be one type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), APU (Accelerated Processing Unit), etc.

[0020] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0021] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0022] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0024] [First Embodiment]

[0025] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0026] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0029] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0032] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0036] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0037] The childcare support system of the present invention analyzes the baby's behavior and emotional state using audio and video data acquired around the baby, and proposes appropriate responses. An embodiment of this system is described below.

[0038] System Overview

[0039] 1. Data Acquisition Method

[0040] A device (such as a smartphone) is placed near the baby and uses a microphone and camera to acquire audio and video data. The device then sends the acquired data to a cloud server. At this stage, the data is processed in real time, so technologies are applied to minimize communication delays.

[0041] 2. Data Analysis Process

[0042] The server analyzes the received data. For audio data, it analyzes the tone, volume, and sound patterns of the voice to infer emotional states such as hunger or discomfort from the crying. For video data, it uses computer vision technology to analyze the baby's movements and facial expressions to identify behavioral patterns.

[0043] 3. Making predictions and deciding on countermeasures

[0044] Based on the analysis results, the server estimates the baby's current condition and determines the optimal course of action. This is done using a dedicated machine learning algorithm that provides suggestions tailored to the individual characteristics of each baby.

[0045] 4. Integration with smart devices

[0046] The device then implements the countermeasures received from the server. This includes steps such as coordinating with smart devices like smart speakers and smart lights to create an appropriate environment. For example, it might play soft music to calm a baby or adjust the room lighting to create a sleep-friendly environment.

[0047] Specific example

[0048] Example 1: Analysis and response to crying sounds

[0049] The device detects the baby's crying and immediately sends the signal to the server. The server analyzes the crying pattern and determines that the baby is likely hungry. Based on this, the action "notify the user to feed the baby milk" is selected. The device then sends a notification to the user saying, "The baby seems hungry. Please prepare some milk."

[0050] Example 2: Environmental adjustments to promote sleep

[0051] If the video data detects that the baby is frequently yawning or rubbing its eyes, the server infers that the baby is sleepy. Based on this, the device instructs the smart light to dim the room's brightness and play relaxing music.

[0052] In this way, the present invention can provide detailed childcare support tailored to the baby's condition while working in conjunction with a variety of devices.

[0053] The following describes the processing flow.

[0054] Step 1:

[0055] The device is placed in the baby's environment and acquires audio and video data in real time via a microphone and camera. This includes the baby's cries, ambient sounds, and video of the baby's movements and facial expressions.

[0056] Step 2:

[0057] The device temporarily stores the acquired data, performs preprocessing such as noise reduction and data compression, and then sends the data to a server in the cloud via the internet.

[0058] Step 3:

[0059] When the server receives audio data, it begins analyzing the crying patterns using speech recognition technology. Here, it analyzes the volume, rhythm, and pitch of the sounds to infer the baby's emotional state (e.g., hunger, sleepiness, discomfort).

[0060] Step 4:

[0061] Simultaneously, the server analyzes the video data and uses computer vision algorithms to identify the baby's movements and posture. For example, it can detect actions such as rubbing the eyes or flailing the arms and legs.

[0062] Step 5:

[0063] The server analyzes the audio and video results to estimate the baby's current condition. This involves applying machine learning models and combining them with predictive models based on historical data to achieve highly accurate predictions.

[0064] Step 6:

[0065] The server determines the appropriate course of action based on the prediction results. For each situation, it formulates a specific action plan, such as "give milk" or "prepare for a nap."

[0066] Step 7:

[0067] The device executes instructions received from the server and sends a notification to the user. This notification includes recommended actions and provides guidance to facilitate childcare.

[0068] Step 8:

[0069] The device connects with smart devices and, when necessary, plays music or adjusts lighting to create a comfortable environment for the baby. The goal is to ensure that the proposed countermeasures are effectively implemented.

[0070] (Example 1)

[0071] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0072] In modern society, the burden of childcare is significant, and there is a particular need to quickly and accurately understand the physical and emotional state of babies. However, current technology makes it difficult to analyze a baby's condition in real time and automatically implement appropriate countermeasures tailored to their individual characteristics. Furthermore, there is the challenge that relatives and caregivers often have difficulty receiving immediate support regarding childcare while going about their daily lives.

[0073] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0074] In this invention, the server includes means for receiving acoustic and image information from an acquisition device, means for analyzing the acoustic and image information to infer the subject's behavior or emotional state, and means for determining and displaying appropriate action based on the inference. This enables real-time analysis of the baby's condition and highly accurate childcare support tailored to individual characteristics.

[0075] An "acquisition device" is a device used to receive acoustic and image information, and is mainly equipped with a microphone and a camera.

[0076] "Acoustic information" refers to the sound emitted from an object, acquired as digital data, and is characterized by its tone, volume, and pattern.

[0077] "Image information" refers to still images or video data captured by a camera or similar device, which visually records the actions and facial expressions of a subject.

[0078] The "subject" is the person being monitored for childcare support purposes, usually a baby, but can also include other individuals designated by the user.

[0079] "Inference" is the process of analyzing acquired acoustic and visual information to predict the state or emotions of a subject.

[0080] "Appropriate action" refers to suggestions for actions or environmental adjustments that should be taken for the subject based on their presumed condition.

[0081] "Intelligent devices" are devices that are connected to a network and have the ability to automatically switch their operation based on external commands. Examples include smart speakers and smart lights.

[0082] "Real-time" refers to a process where data acquisition, analysis, and result delivery occur immediately, with processing completed within a timeframe that minimizes delays.

[0083] A "generative AI model" is a form of artificial intelligence that learns from past data and has the ability to make predictions and suggestions based on new data.

[0084] This invention is a childcare support system that reduces the burden of childcare and analyzes the baby's condition in real time. This system uses a terminal (e.g., a smartphone) as an acquisition device to collect acoustic and image information. The terminal is placed near the baby and acquires data using a microphone and camera. The acquired data is transmitted in real time to a server in the cloud. The server performs high-speed data processing and uses efficient data compression technology to minimize communication delays.

[0085] The server uses a speech analysis engine to analyze acoustic information. Specifically, it analyzes the tone, volume, and sound patterns of the voice to infer the baby's emotional state. Image information is analyzed using computer vision technology through a video analysis engine to identify the baby's movements and facial expressions.

[0086] The server inputs the analysis results into a generating AI model to generate countermeasures tailored to the individual characteristics of each baby. These countermeasures are optimized based on the child's individual condition and may include, for example, environmental adjustments to promote sleep or offering milk when the baby is hungry. The generated countermeasures are notified to the user via the terminal.

[0087] Furthermore, the device can automatically create an appropriate environment by linking with intelligent devices. For example, it can adjust the brightness of smart lights and instruct smart speakers to play relaxation music.

[0088] As a concrete example, a device that detects a baby crying sends the acoustic information to a server, and a generating AI model infers that the crying is due to hunger. In this case, the device notifies the user, "Please prepare to feed the baby milk." An example of a prompt message is, "Analyze the baby's condition from the audio and video data when the baby is crying and suggest appropriate childcare support."

[0089] In this way, the invention provides support tailored to the baby's condition and reduces the burden on caregivers.

[0090] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0091] Step 1:

[0092] The device is placed near the baby and uses a microphone to acquire ambient acoustic information and a camera to collect image information. The input to this process is the baby's cries and movements, and the output is digitized acoustic and image information. The device transmits this data to a cloud server in real time.

[0093] Step 2:

[0094] The server receives acoustic information transmitted from the terminal. Using the acquired acoustic data as input, the audio analysis engine analyzes the data. Specifically, it analyzes the tone, volume, and pattern of the sound to infer the baby's emotional state. The output is a possible state indicated by the crying (e.g., hunger, discomfort).

[0095] Step 3:

[0096] The server also receives image information and performs analysis using computer vision technology. Based on the image data obtained as input, the video analysis engine analyzes the baby's movements (e.g., yawning, rubbing eyes) and facial expressions. This identifies the baby's state from its behavioral patterns and facial expressions, and provides insights into that state as output.

[0097] Step 4:

[0098] The server inputs the results of the analysis of acoustic and image information into a generating AI model. This model accurately predicts the baby's current state from this data and determines the appropriate course of action based on that prediction. The output is the optimal course of action tailored to the baby's characteristics (e.g., a suggestion to change the diaper, instructions to prepare milk).

[0099] Step 5:

[0100] The device receives a notification of a course of action from the server. Based on this notification, it displays a message on the device's screen prompting the user to take specific action and can, if necessary, link with intelligent devices. For example, the device might send a notification to the user saying, "Please feed your baby milk," and instruct them to adjust the brightness of a smart light or play relaxing music.

[0101] Step 6:

[0102] After receiving a notification from the device, the user takes action on the suggested course of action. Specifically, this might involve preparing milk or changing diapers. They can also observe the baby's condition within the environment set up by the smart device and take additional action as needed.

[0103] Through these steps, the system provides users with immediate support tailored to the baby's various needs.

[0104] (Application Example 1)

[0105] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0106] Traditional product inspection methods have limitations when it comes to manual quality control. In particular, detecting defects using sound and video is labor-intensive and time-consuming, and its accuracy is also limited. There is a need to provide a method that effectively solves these problems and enables rapid and precise product quality control.

[0107] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0108] In this invention, the server includes means for receiving audio and video data from an acquisition device, means for analyzing the audio and video data to infer the behavior or emotional state of the object, and means for detecting specific disorder in a product and proposing countermeasures. This enables automated, highly accurate audio and video-based product inspection and rapid notification to managers.

[0109] An "acquisition device" is a device used to acquire audio and video data from a target in real time.

[0110] "Audio data" refers to sound information obtained through an acquisition device, and is data for analysis, including volume and sound quality.

[0111] "Video data" refers to visual information obtained through an acquisition device, and is data used for analyzing movement and appearance.

[0112] "Object" refers to an item or subject that is monitored or analyzed by the system.

[0113] "Intelligent devices" refer to devices that have the ability to take appropriate actions or give instructions, and include smart devices.

[0114] "Disorder" refers to abnormalities or defects in the state of a product or object that deviate from the normal pattern.

[0115] "Countermeasures" refer to specific actions or measures that are determined to be taken based on the results that the system has inferred or detected.

[0116] "Administrator" refers to the person or department responsible for the operation of the system and the implementation of countermeasures.

[0117] The system implementing this application primarily aims to automatically manage the quality of objects by analyzing audio and video data. The server receives audio and video data from acquisition devices in real time and analyzes this data. For audio data, an audio analysis algorithm is used to detect abnormal or unusual sounds. For video data, computer vision technology is used to identify abnormalities in the surface condition and shape of the product.

[0118] This process utilizes cloud servers and intelligent devices to streamline data analysis. The calculations employ programming languages ​​such as Python and image processing techniques using libraries like OpenCV. Based on the analysis results, the server controls the intelligent devices and proposes appropriate countermeasures.

[0119] As a concrete example, in product inspection at a factory, an acquisition device moves along the production line, acquiring audio and video data of the products. The acquired data is sent to a server, and if any abnormalities in the sound or image are automatically detected, the system notifies the administrator and instructs them on how to respond.

[0120] An example of a prompt is, "Please tell me about the most important anomaly detection system for product inspection." This prompt is used to provide specific information about anomaly detection using a generative AI model.

[0121] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0122] Step 1:

[0123] The terminal collects audio and video data of the target object through an acquisition device. The audio data, as input, is the waveform data of sound detected by the sensor, and the video data includes frame images from the camera. This data is then prepared to be transmitted to the server in real time.

[0124] Step 2:

[0125] The server receives audio data transmitted from the terminal and applies an audio analysis algorithm to analyze any unusual sounds or patterns. If the sound pattern differs from the normal pattern, this analysis detects the abnormal pattern as an output.

[0126] Step 3:

[0127] The server receives video data transmitted from the terminal and inspects the surface condition of the object using image analysis. Computer vision technology is used to process the image data and identify abnormal shapes, scratches, and other defects. The results of this analysis are then generated as output data.

[0128] Step 4:

[0129] The server uses a generative AI model to determine the necessary countermeasures based on the results of audio and video analysis. At this stage, the analysis results are converted into prompts, and the generative AI model generates the optimal countermeasure as output.

[0130] Step 5:

[0131] The server controls the intelligent device and executes the determined response. This control includes sending notifications and issuing instructions for the device's operation. The user is also notified of the anomaly and the recommended course of action.

[0132] Step 6:

[0133] The user receives notifications from the server and takes actual corrective actions based on the proposed solutions. Efficient responses are confirmed through the system's operation and user interface.

[0134] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0135] This invention provides a system for childcare support that takes into account not only the emotional state and behavior of the baby, but also the emotions of the user, the caregiver. The embodiments of this system are described in detail below.

[0136] Overall system configuration

[0137] 1. Data Acquisition and Analysis

[0138] The device acquires audio and video data about the baby. This includes crying, ambient sounds, movements, and facial expressions. This data is sent to a server in the cloud and analyzed in real time by a dedicated analysis engine.

[0139] 2. Predicting the baby's emotional state

[0140] The server infers the baby's emotions and behavior based on the analyzed audio and video. This uses an emotion analysis algorithm, which makes it possible to understand what needs the baby has.

[0141] 3. User emotion recognition

[0142] The device further uses an emotion engine to recognize the user's emotional state through the user's voice input and actions (e.g., tone of voice and word choice). This is done to identify emotions such as stress, fatigue, or relief.

[0143] 4. Decision and presentation of countermeasures

[0144] The server determines a course of action that takes into account the emotional states of both the baby and the user. This course of action is tailored to the baby's needs while also including measures to soothe and support the user's emotions.

[0145] 5. Smart device control

[0146] The device controls connected smart devices based on the countermeasures taken. For example, if the user is feeling stressed, relaxing music may be played or the lighting may be adjusted.

[0147] Specific example

[0148] Example 1: Baby's hunger and user fatigue

[0149] The server detects a baby crying and infers that the baby is hungry. At the same time, the emotion engine recognizes signs of fatigue in the user's voice. In this case, the server decides on a course of action: "gently notify the user to prepare the baby's milk and play relaxing music in the meantime." The device follows these instructions, notifying the user and playing the music.

[0150] Example 2: A baby who wants to play and a user who feels safe.

[0151] When video analysis recognizes the baby's desire to play and the user's voice indicates they feel at ease, the server suggests a solution such as extending playtime. The device then plays upbeat music and automatically adjusts the room temperature to a suitable level for playtime.

[0152] Thus, this invention grasps the emotional states of both the baby and the user, providing new value to conventional childcare support by taking the user's emotions into consideration.

[0153] The following describes the processing flow.

[0154] Step 1:

[0155] The device is placed near the baby and uses its built-in microphone and camera to capture audio and video data. This includes the baby's cries, facial expressions, and body movements. The data is converted into a digital format and prepared for analysis.

[0156] Step 2:

[0157] The device transmits the acquired audio and video data to a cloud server. This communication uses a secure protocol to minimize the risk of data leakage. The transmitted data is processed in real time, taking into account any temporal delays.

[0158] Step 3:

[0159] When the server receives audio data, it uses a speech recognition system to analyze the crying patterns. From the identified audio features, it infers the baby's emotional state (e.g., hunger, sleepiness). A pre-trained emotion model is used for this analysis.

[0160] Step 4:

[0161] The server processes the video data and uses computer vision algorithms to analyze the baby's facial expressions and movements. For example, it can determine whether the baby wants to play or is sleepy based on their facial expressions and limb movements.

[0162] Step 5:

[0163] The server recognizes the user's emotions based on voice input and behavioral logs, in addition to the baby's emotional state. The emotion engine identifies stress and feelings of security from the user's voice and sends the resulting data to the emotion analysis system.

[0164] Step 6:

[0165] By combining this data, the server generates a response plan that takes into account the emotional state of both the baby and the user. For example, specific actions such as "feed the baby milk" or "recommend playing relaxation music" are determined.

[0166] Step 7:

[0167] The device sends a notification to the user based on the countermeasures received from the server. The notification presents an action plan tailored to the baby's condition and the user's emotions.

[0168] Step 8:

[0169] The device connects with smart devices and automatically adjusts the environment as needed. For example, it optimizes the childcare environment by lowering the brightness of smart lights or playing music.

[0170] Through this series of processes, the system provides childcare support that meets the needs of both the baby and the user.

[0171] (Example 2)

[0172] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0173] Conventional childcare support systems have struggled to provide comprehensive solutions that consider not only the baby's emotions and needs, but also the parents' emotions and stress levels. Furthermore, providing optimal solutions in real time, based on the conditions of both the baby and the parents, has been a technical challenge.

[0174] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0175] In this invention, the server includes means for receiving audio and video data from an acquisition device, means for analyzing the audio and video data to estimate the emotional states of the baby and the caregiver, and means for determining appropriate countermeasures that take into account the emotional states of the baby and the caregiver based on the estimation results. This enables comprehensive childcare support that simultaneously satisfies the needs of the baby and the emotions of the caregiver.

[0176] An "acquisition device" is a device for receiving audio and video data.

[0177] "Analyzing audio and video data" means processing the received data to determine the subject's behavior and emotional state.

[0178] "Inferring behavioral and emotional states" means estimating the subject's current psychological or physical state based on analyzed data.

[0179] "Determining appropriate countermeasures" means using the prediction results to select the optimal actions and instructions for the target and users.

[0180] "Controlling smart devices" means operating or managing connected electronic devices based on predetermined countermeasures.

[0181] A "generative AI model" refers to an algorithm that learns from data and accurately predicts the state of a situation.

[0182] "Generating optimal action plans" means creating specific action guidelines tailored to the target audience and the user's situation.

[0183] This invention is a system aimed at supporting childcare, and it provides functions that take into account not only the emotional state and behavior of the baby, but also the emotions of the caregiver who uses the system. This system is implemented as follows.

[0184] About the configuration and equipment

[0185] The device is equipped with a camera and microphone to acquire audio and video data about the baby. The data acquired from these sensors includes crying, ambient sounds, movements, and facial expressions. This collected data is transmitted to a server via the internet.

[0186] About data analysis

[0187] The server analyzes data in real time using an analysis engine built on the cloud. This analysis uses a generative AI model to infer the baby's emotional state and behavior from audio and video. The analyzed information is processed by an emotion analysis algorithm within the system to specifically identify the baby's needs. Similarly, the user's emotional state is also analyzed from the voice input.

[0188] Proposal of countermeasures

[0189] The server takes into account the emotional states of both the baby and the user to determine the appropriate course of action. For example, if the server detects both the baby's hunger and the user's fatigue simultaneously, it will prompt the user to prepare milk and send instructions to the device recommending relaxing music.

[0190] Specific examples and prompt statements

[0191] For example, if a baby cries, and the server determines that the baby is hungry, and at the same time the user's calm tone indicates low stress levels, the server will notify the user, "Please enjoy some soothing music while you prepare your baby's milk."

[0192] Example prompt: "The baby is crying continuously. It may need milk. Gently notify the user and instruct them to play relaxing music."

[0193] In this way, the system aims to provide environmentally conscious childcare support and create a comfortable environment for both the baby and the user.

[0194] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0195] Step 1:

[0196] The device uses a camera and microphone to collect audio and video data of the baby's surroundings. Inputs include ambient sounds, the baby's movements, and cries, which are captured as digital data. The collected data is temporarily stored within the device and prepared for analysis.

[0197] Step 2:

[0198] The terminal encrypts the acquired audio and video data and transmits it to the server via the internet. The input is the raw data acquired in step 1, which is delivered to the server while maintaining security during transmission. The output is an encrypted data flow.

[0199] Step 3:

[0200] The server inputs the received audio and video data into its analysis engine, where a generative AI model performs real-time analysis. The input consists of digital audio and video, which the AI ​​model processes and outputs the baby's emotional state (e.g., hunger, sleepiness, desire to play, etc.). This output is data indicating the baby's specific emotional state.

[0201] Step 4:

[0202] The device acquires the parent's voice and analyzes it using an emotion engine. The input is the parent's words and tone of voice, which is then subjected to emotion analysis to output the user's emotional state (e.g., stress, fatigue, feeling of security). This output is data that indicates the user's feelings.

[0203] Step 5:

[0204] The server integrates and analyzes the emotional states of both the baby and the user to determine the optimal course of action. The input is the emotional data obtained in steps 3 and 4, and an AI model is used to design responses that address the baby's needs while supporting the user. The output is a set of specific instructions and recommendations.

[0205] Step 6:

[0206] The server sends the decided course of action to the terminal. The input is the course of action generated on the server side, which is then transmitted to the terminal as a procedure manual. The output is a digital message containing instructions to be followed.

[0207] Step 7:

[0208] The terminal controls connected smart devices to implement the received countermeasures. The input is a command from the server, which executes specific actions such as playing music or adjusting lighting. The output is the optimized state of the environment settings.

[0209] (Application Example 2)

[0210] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0211] There is a need for a system that can detect abnormal situations and safety concerns within the home based on a comprehensive assessment that takes into account the emotional state of each individual and the overall home environment, and to respond appropriately. Conventional technologies only focus on individual emotions and behaviors, making it difficult to comprehensively grasp the safety situation within the home and provide appropriate countermeasures in real time.

[0212] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0213] In this invention, the server includes means for receiving audio and video data from an acquisition device, means for analyzing the audio and video data and inferring the subject's behavior or emotional state, and means for evaluating the safety status within the home based on the audio characteristics of the subject and neighbors. This makes it possible to ensure safety and provide prompt countermeasures tailored to the situation and individual emotions within the home.

[0214] An "acquisition device" is a device used to receive audio and video data, and is responsible for acquiring data within a home or specific environment.

[0215] "Audio data" refers to audio or sound information represented in digital format, and is used for analysis and evaluation.

[0216] "Video data" refers to visual information represented in digital format, and is used for analysis and evaluation.

[0217] A "subject" is an individual observed through audio and video data, whose emotional state and behavior are inferred from it.

[0218] "Inference" is the act of analyzing acquired data to determine the emotional state and behavior of a subject.

[0219] "Vocal features" are specific elements extracted from voice data and serve as indicators of an individual's emotional state and environmental characteristics.

[0220] A "means for evaluating the safety status within the home" is a system that analyzes the overall safety of the environment based on acquired data and provides an appropriate assessment.

[0221] In the system implementing this invention, a server and a terminal play important roles. First, the terminal collects audio and video data through an acquisition device installed in the home. The acquired data is then transmitted to the server. The server uses audio analysis software (e.g., speech recognition API) to analyze the audio data and image analysis software (e.g., image recognition API) to analyze the video data to infer the behavior and emotional state of the subject.

[0222] The server assesses the safety status of the home based on the inferred emotional state of the subject, while also considering voice characteristics. For example, an AI-powered analysis platform learns voices and sounds within the home to detect anomalies. Next, a generative AI model is used to determine appropriate countermeasures based on the safety status assessment. This makes it possible to present appropriate countermeasures in real time. For example, if someone in the family is experiencing stress, it will suggest relaxing content or activities.

[0223] These countermeasures are communicated to the user via the device, and automatically controlled devices are appropriately managed. For example, actions such as adjusting lighting or playing music can be performed. This gives users a sense of security within their homes and enables proactive safety management.

[0224] As a concrete example, when a baby starts crying, the device collects the audio data, and the server analyzes it. If it determines that the baby is hungry, the server notifies the user to prepare milk, and the system then starts playing relaxing music. This entire process is achieved by utilizing IoT devices and cloud computing technology.

[0225] Example of a prompt:

[0226] "To ensure safety within the home, analyze audio and video in real time and suggest appropriate responses. Explain how to control smart devices, taking into account the emotions of both the baby and the caregiver."

[0227] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0228] Step 1:

[0229] The terminal collects audio and video data using acquisition equipment installed in the home. This input data consists of audio and video captures from within the home and is necessary for subsequent analysis. The collected data is transmitted to the server in real time.

[0230] Step 2:

[0231] The server analyzes the received audio data using speech analysis software. This extracts speech features and generates basic information for determining the subject's emotions and the situation within the home. The output of this speech analysis includes feature quantities, such as crying patterns and voice tone.

[0232] Step 3:

[0233] The server analyzes the video data using image analysis software. It analyzes the subject's actions in detail from the video data and saves the results to the server's database. The analysis results output indicators that show abnormalities or warning signs of behavior.

[0234] Step 4:

[0235] The server integrates features derived from audio and video data and uses a generative AI model to infer the subject's emotional state. In this step, prompt sentences are used to enable the AI ​​model to infer emotion, providing the basis for appropriate responses.

[0236] Step 5:

[0237] The server assesses the safety status and determines appropriate countermeasures based on inferred emotional states and voice characteristics within the home. Using a generative AI model, it generates instructions to quickly provide responses in the event of an anomaly.

[0238] Step 6:

[0239] The device notifies the user of the countermeasures output from the server and executes control of the smart device based on those countermeasures. This includes specific actions such as playing music or adjusting lighting. The aim is to give the user a sense of security within their home.

[0240] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0241] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0242] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0243] [Second Embodiment]

[0244] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0245] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0246] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0247] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0248] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0249] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0250] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0251] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0252] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0253] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0254] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0255] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0256] The childcare support system of the present invention analyzes the baby's behavior and emotional state using audio and video data acquired around the baby, and proposes appropriate responses. An embodiment of this system is described below.

[0257] System Overview

[0258] 1. Data Acquisition Method

[0259] A device (such as a smartphone) is placed near the baby and uses a microphone and camera to acquire audio and video data. The device then sends the acquired data to a cloud server. At this stage, the data is processed in real time, so technologies are applied to minimize communication delays.

[0260] 2. Data Analysis Process

[0261] The server analyzes the received data. For audio data, it analyzes the tone, volume, and sound patterns of the voice to infer emotional states such as hunger or discomfort from the crying. For video data, it uses computer vision technology to analyze the baby's movements and facial expressions to identify behavioral patterns.

[0262] 3. Making predictions and deciding on countermeasures

[0263] Based on the analysis results, the server estimates the baby's current condition and determines the optimal course of action. This is done using a dedicated machine learning algorithm that provides suggestions tailored to the individual characteristics of each baby.

[0264] 4. Integration with smart devices

[0265] The device then implements the countermeasures received from the server. This includes steps such as coordinating with smart devices like smart speakers and smart lights to create an appropriate environment. For example, it might play soft music to calm a baby or adjust the room lighting to create a sleep-friendly environment.

[0266] Specific example

[0267] Example 1: Analysis and response to crying sounds

[0268] The device detects the baby's crying and immediately sends the signal to the server. The server analyzes the crying pattern and determines that the baby is likely hungry. Based on this, the action "notify the user to feed the baby milk" is selected. The device then sends a notification to the user saying, "The baby seems hungry. Please prepare some milk."

[0269] Example 2: Environmental adjustments to promote sleep

[0270] If the video data detects that the baby is frequently yawning or rubbing its eyes, the server infers that the baby is sleepy. Based on this, the device instructs the smart light to dim the room's brightness and play relaxing music.

[0271] In this way, the present invention can provide detailed childcare support tailored to the baby's condition while working in conjunction with a variety of devices.

[0272] The following describes the processing flow.

[0273] Step 1:

[0274] The device is placed in the baby's environment and acquires audio and video data in real time via a microphone and camera. This includes the baby's cries, ambient sounds, and video of the baby's movements and facial expressions.

[0275] Step 2:

[0276] The device temporarily stores the acquired data, performs preprocessing such as noise reduction and data compression, and then sends the data to a server in the cloud via the internet.

[0277] Step 3:

[0278] When the server receives the voice data, it starts the pattern analysis of the crying sound using voice recognition technology. Here, it analyzes the intensity, rhythm, pitch, etc. of the sound to infer the emotional state of the baby (e.g., hunger, sleepiness, discomfort).

[0279] Step 4:

[0280] At the same time, the server analyzes the video data and uses computer vision algorithms to identify the baby's actions and postures. As specific examples, it detects actions such as blinking eyes and flapping hands and feet.

[0281] Step 5:

[0282] The server synthesizes the analysis results of the voice and video and estimates the current state of the baby. To do this, it applies a machine learning model and combines a prediction model based on past data to make a highly accurate inference.

[0283] Step 6:

[0284] Based on the inference result, the server determines a countermeasure. For each state, it formulates a specific action plan such as "give milk" or "prepare for a nap".

[0285] Step 7:

[0286] The terminal proceeds to execute the instructions received from the server and sends a notification to the user. This notification describes the recommended countermeasures and provides guidance for smooth childcare.

[0287] Step 8:

[0288] The terminal coordinates with smart devices and, if necessary, plays music or adjusts lighting to create a comfortable environment for the baby. The aim is to effectively implement the proposed countermeasures.

[0289] (Example 1)

[0290] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0291] In modern society, the burden of childcare is significant, and there is a particular need to quickly and accurately understand the physical and emotional state of babies. However, current technology makes it difficult to analyze a baby's condition in real time and automatically implement appropriate countermeasures tailored to their individual characteristics. Furthermore, there is the challenge that relatives and caregivers often have difficulty receiving immediate support regarding childcare while going about their daily lives.

[0292] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0293] In this invention, the server includes means for receiving acoustic and image information from an acquisition device, means for analyzing the acoustic and image information to infer the subject's behavior or emotional state, and means for determining and displaying appropriate action based on the inference. This enables real-time analysis of the baby's condition and highly accurate childcare support tailored to individual characteristics.

[0294] An "acquisition device" is a device used to receive acoustic and image information, and is mainly equipped with a microphone and a camera.

[0295] "Acoustic information" refers to sound emitted from an object, acquired as digital data, characterized by its tone, volume, and pattern.

[0296] "Image information" refers to still images or video data captured by a camera or similar device, which visually records the actions and facial expressions of a subject.

[0297] The "subject" is the person being monitored for childcare support purposes, usually a baby, but can also include other individuals designated by the user.

[0298] "Inference" is the process of analyzing acquired acoustic and visual information to predict the state or emotions of a subject.

[0299] "Appropriate measures" refer to suggestions for actions or environmental adjustments that should be taken for the subject, based on their presumed condition.

[0300] "Intelligent devices" are devices that are connected to a network and have the ability to automatically switch their operation based on external commands. Examples include smart speakers and smart lights.

[0301] "Real-time" refers to a process where data acquisition, analysis, and result delivery occur immediately, with processing completed within a timeframe that minimizes delays.

[0302] A "generative AI model" is a form of artificial intelligence that learns from past data and has the ability to make predictions and suggestions based on new data.

[0303] This invention is a childcare support system that reduces the burden of childcare and analyzes the baby's condition in real time. This system uses a terminal (e.g., a smartphone) as an acquisition device to collect acoustic and image information. The terminal is placed near the baby and acquires data using a microphone and camera. The acquired data is transmitted in real time to a server in the cloud. The server performs high-speed data processing and uses efficient data compression technology to minimize communication delays.

[0304] The server uses a speech analysis engine to analyze acoustic information. Specifically, it analyzes the tone, volume, and sound patterns of the voice to infer the baby's emotional state. Image information is analyzed using computer vision technology through a video analysis engine to identify the baby's movements and facial expressions.

[0305] The server inputs the analysis results into the generative AI model and generates countermeasures according to the individual characteristics of the baby. These countermeasures are optimized based on the individual state of the child and include, for example, environmental adjustments for sleep promotion and the presentation of milk during hunger. The generated countermeasures are notified to the user through the terminal.

[0306] Furthermore, by collaborating with intelligent devices, the terminal can also automatically arrange an appropriate environment. For example, it can adjust the brightness of a smart light and issue instructions to play relaxation music through a smart speaker.

[0307] As a specific example, a terminal that detects the crying sound of a baby transmits acoustic information to the server, and the generative AI model推测 that the crying is due to hunger. In this case, the terminal notifies the user, "Please prepare to feed the baby milk." An example of a prompt sentence is, "Analyze the state from the audio data and video data when the baby is crying and propose appropriate childcare support."

[0308] In this way, the invention provides support tailored to the state of the baby and reduces the burden on caregivers.

[0309] The flow of the specific process in Example 1 will be described using FIG. 11.

[0310] Step 1:

[0311] The terminal is placed near the baby, uses a microphone to acquire ambient acoustic information, and uses a camera to collect image information. The input of this process is the crying sound and movements of the baby, and the output is digitized acoustic information and image information. The terminal transmits this data to the cloud server in real time.

[0312] Step 2:

[0313] The server receives acoustic information transmitted from the terminal. Using the acquired acoustic data as input, the audio analysis engine analyzes the data. Specifically, it analyzes the tone, volume, and pattern of the sound to infer the baby's emotional state. The output is a possible state indicated by the crying (e.g., hunger, discomfort).

[0314] Step 3:

[0315] The server also receives image information and performs analysis using computer vision technology. Based on the image data obtained as input, the video analysis engine analyzes the baby's movements (e.g., yawning, rubbing eyes) and facial expressions. This identifies the baby's state from its behavioral patterns and facial expressions, and provides insights into that state as output.

[0316] Step 4:

[0317] The server inputs the results of the analysis of acoustic and image information into a generating AI model. This model accurately predicts the baby's current state from this data and determines the appropriate course of action based on that prediction. The output is the optimal course of action tailored to the baby's characteristics (e.g., a suggestion to change the diaper, instructions to prepare milk).

[0318] Step 5:

[0319] The device receives a notification of a course of action from the server. Based on this notification, it displays a message on the device's screen prompting the user to take specific action and can, if necessary, link with intelligent devices. For example, the device might send a notification to the user saying, "Please feed your baby milk," and instruct them to adjust the brightness of a smart light or play relaxing music.

[0320] Step 6:

[0321] After receiving a notification from the device, the user takes action on the suggested course of action. Specifically, this might involve preparing milk or changing diapers. They can also observe the baby's condition within the environment set up by the smart device and take additional action as needed.

[0322] Through these steps, the system provides users with immediate support tailored to the baby's various needs.

[0323] (Application Example 1)

[0324] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0325] Traditional product inspection methods have limitations when it comes to manual quality control. In particular, detecting defects using sound and video is labor-intensive and time-consuming, and its accuracy is also limited. There is a need to provide a method that effectively solves these problems and enables rapid and precise product quality control.

[0326] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0327] In this invention, the server includes means for receiving audio and video data from an acquisition device, means for analyzing the audio and video data to infer the behavior or emotional state of the object, and means for detecting specific disorder in a product and proposing countermeasures. This enables automated, highly accurate audio and video-based product inspection and rapid notification to managers.

[0328] An "acquisition device" is a device used to acquire audio and video data from a target in real time.

[0329] "Audio data" refers to sound information obtained through an acquisition device, and is data for analysis, including volume and sound quality.

[0330] "Video data" refers to visual information obtained through an acquisition device, and is data used for analyzing movement and appearance.

[0331] "Object" refers to an item or subject that is monitored or analyzed by the system.

[0332] "Intelligent devices" refer to devices that have the ability to take appropriate actions or give instructions, and include smart devices.

[0333] "Disorder" refers to abnormalities or defects in the state of a product or object that deviate from the normal pattern.

[0334] "Countermeasures" refer to specific actions or measures that are determined to be taken based on the results that the system has inferred or detected.

[0335] "Administrator" refers to the person or department responsible for the operation of the system and the implementation of countermeasures.

[0336] The system implementing this application primarily aims to automatically manage the quality of objects by analyzing audio and video data. The server receives audio and video data from acquisition devices in real time and analyzes this data. For audio data, an audio analysis algorithm is used to detect abnormal or unusual sounds. For video data, computer vision technology is used to identify abnormalities in the surface condition and shape of the product.

[0337] This process utilizes cloud servers and intelligent devices to streamline data analysis. The calculations employ programming languages ​​such as Python and image processing techniques using libraries like OpenCV. Based on the analysis results, the server controls the intelligent devices and proposes appropriate countermeasures.

[0338] As a concrete example, in product inspection at a factory, an acquisition device moves along the production line, acquiring audio and video data of the products. The acquired data is sent to a server, and if any abnormalities in the sound or image are automatically detected, the system notifies the administrator and instructs them on how to respond.

[0339] An example of a prompt is, "Please tell me about the most important anomaly detection system for product inspection." This prompt is used to provide specific information about anomaly detection using a generative AI model.

[0340] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0341] Step 1:

[0342] The terminal collects audio and video data of the target object through an acquisition device. The audio data, as input, is the waveform data of sound detected by the sensor, and the video data includes frame images from the camera. This data is then prepared to be transmitted to the server in real time.

[0343] Step 2:

[0344] The server receives audio data transmitted from the terminal and applies an audio analysis algorithm to analyze any unusual sounds or patterns. If the sound pattern differs from the normal pattern, this analysis detects the abnormal pattern as an output.

[0345] Step 3:

[0346] The server receives video data transmitted from the terminal and inspects the surface condition of the object using image analysis. Computer vision technology is used to process the image data and identify abnormal shapes, scratches, and other defects. The results of this analysis are then generated as output data.

[0347] Step 4:

[0348] The server uses a generative AI model to determine the necessary countermeasures based on the results of audio and video analysis. At this stage, the analysis results are converted into prompts, and the generative AI model generates the optimal countermeasure as output.

[0349] Step 5:

[0350] The server controls the intelligent device and executes the determined response. This control includes sending notifications and issuing instructions for the device's operation. The user is also notified of the anomaly and the recommended course of action.

[0351] Step 6:

[0352] The user receives notifications from the server and takes actual corrective actions based on the proposed solutions. Efficient responses are confirmed through the system's operation and user interface.

[0353] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0354] This invention provides a system for childcare support that takes into account not only the emotional state and behavior of the baby, but also the emotions of the user, the caregiver. The embodiments of this system are described in detail below.

[0355] Overall system configuration

[0356] 1. Data Acquisition and Analysis

[0357] The device acquires audio and video data about the baby. This includes crying, ambient sounds, movements, and facial expressions. This data is sent to a server in the cloud and analyzed in real time by a dedicated analysis engine.

[0358] 2. Predicting the baby's emotional state

[0359] The server infers the baby's emotions and behavior based on the analyzed audio and video. This uses an emotion analysis algorithm, which makes it possible to understand what needs the baby has.

[0360] 3. User emotion recognition

[0361] The device further uses an emotion engine to recognize the user's emotional state through the user's voice input and actions (e.g., tone of voice and word choice). This is done to identify emotions such as stress, fatigue, or relief.

[0362] 4. Decision and presentation of countermeasures

[0363] The server determines a course of action that takes into account the emotional states of both the baby and the user. This course of action is tailored to the baby's needs while also including measures to soothe and support the user's emotions.

[0364] 5. Smart device control

[0365] The device controls connected smart devices based on the countermeasures taken. For example, if the user is feeling stressed, relaxing music may be played or the lighting may be adjusted.

[0366] Specific example

[0367] Example 1: Baby's hunger and user fatigue

[0368] The server detects a baby crying and infers that the baby is hungry. At the same time, the emotion engine recognizes signs of fatigue in the user's voice. In this case, the server decides on a course of action: "gently notify the user to prepare the baby's milk and play relaxing music in the meantime." The device follows these instructions, notifying the user and playing the music.

[0369] Example 2: A baby who wants to play and a user who feels safe.

[0370] When video analysis recognizes the baby's desire to play and the user's voice indicates they feel at ease, the server suggests a solution such as extending playtime. The device then plays upbeat music and automatically adjusts the room temperature to a suitable level for playtime.

[0371] Thus, the present invention grasps the emotional states of both the baby and the user, providing new value to conventional childcare support by taking the user's emotions into consideration.

[0372] The following describes the processing flow.

[0373] Step 1:

[0374] The device is placed near the baby and uses its built-in microphone and camera to capture audio and video data. This includes the baby's cries, facial expressions, and body movements. The data is converted into a digital format and prepared for analysis.

[0375] Step 2:

[0376] The device transmits the acquired audio and video data to a cloud server. This communication uses a secure protocol to minimize the risk of data leakage. The transmitted data is processed in real time, taking into account any temporal delays.

[0377] Step 3:

[0378] When the server receives audio data, it uses a speech recognition system to analyze the crying patterns. From the identified audio features, it infers the baby's emotional state (e.g., hunger, sleepiness). A pre-trained emotion model is used for this analysis.

[0379] Step 4:

[0380] The server processes the video data and uses computer vision algorithms to analyze the baby's facial expressions and movements. For example, it can determine whether the baby wants to play or is sleepy based on their facial expressions and limb movements.

[0381] Step 5:

[0382] The server recognizes the user's emotions based on voice input and behavioral logs, in addition to the baby's emotional state. The emotion engine identifies stress and feelings of security from the user's voice and sends the resulting data to the emotion analysis system.

[0383] Step 6:

[0384] By combining this data, the server generates a response plan that takes into account the emotional state of both the baby and the user. For example, specific actions such as "feed the baby milk" or "recommend playing relaxation music" are determined.

[0385] Step 7:

[0386] The device sends a notification to the user based on the countermeasures received from the server. The notification presents an action plan tailored to the baby's condition and the user's emotions.

[0387] Step 8:

[0388] The device connects with smart devices and automatically adjusts the environment as needed. For example, it optimizes the childcare environment by lowering the brightness of smart lights or playing music.

[0389] Through this series of processes, the system provides childcare support that meets the needs of both the baby and the user.

[0390] (Example 2)

[0391] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0392] Conventional childcare support systems have struggled to provide comprehensive solutions that consider not only the baby's emotions and needs, but also the parents' emotions and stress levels. Furthermore, providing optimal solutions in real time, based on the conditions of both the baby and the parents, has been a technical challenge.

[0393] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0394] In this invention, the server includes means for receiving audio and video data from an acquisition device, means for analyzing the audio and video data to estimate the emotional states of the baby and the caregiver, and means for determining appropriate countermeasures that take into account the emotional states of the baby and the caregiver based on the estimation results. This enables comprehensive childcare support that simultaneously satisfies the needs of the baby and the emotions of the caregiver.

[0395] An "acquisition device" is a device for receiving audio and video data.

[0396] "Analyzing audio and video data" means processing the received data to determine the subject's behavior and emotional state.

[0397] "Inferring behavioral and emotional states" means estimating the subject's current psychological or physical state based on analyzed data.

[0398] "Determining appropriate countermeasures" means using the prediction results to select the optimal actions and instructions for the target and users.

[0399] "Controlling smart devices" means operating or managing connected electronic devices based on predetermined countermeasures.

[0400] A "generative AI model" refers to an algorithm that learns from data and accurately predicts the state of a situation.

[0401] "Generating optimal action plans" means creating specific action guidelines tailored to the target audience and the user's situation.

[0402] This invention is a system aimed at supporting childcare, and it provides functions that take into account not only the emotional state and behavior of the baby, but also the emotions of the caregiver who uses the system. This system is implemented as follows.

[0403] About the configuration and equipment

[0404] The device is equipped with a camera and microphone to acquire audio and video data about the baby. The data acquired from these sensors includes crying, ambient sounds, movements, and facial expressions. This collected data is transmitted to a server via the internet.

[0405] About data analysis

[0406] The server analyzes data in real time using an analysis engine built on the cloud. This analysis uses a generative AI model to infer the baby's emotional state and behavior from audio and video. The analyzed information is processed by an emotion analysis algorithm within the system to specifically identify the baby's needs. Similarly, the user's emotional state is also analyzed from the voice input.

[0407] Proposal of countermeasures

[0408] The server takes into account the emotional states of both the baby and the user to determine the appropriate course of action. For example, if the server detects both the baby's hunger and the user's fatigue simultaneously, it will prompt the user to prepare milk and send instructions to the device recommending relaxing music.

[0409] Specific examples and prompt statements

[0410] For example, if a baby cries, and the server determines that the baby is hungry, and at the same time the user's calm tone indicates low stress levels, the server will notify the user, "Please enjoy some soothing music while you prepare your baby's milk."

[0411] Example prompt: "The baby is crying continuously. It may need milk. Gently notify the user and instruct them to play relaxing music."

[0412] In this way, the system aims to provide environmentally conscious childcare support and create a comfortable environment for both the baby and the user.

[0413] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0414] Step 1:

[0415] The device uses a camera and microphone to collect audio and video data of the baby's surroundings. Inputs include ambient sounds, the baby's movements, and cries, which are captured as digital data. The collected data is temporarily stored within the device and prepared for analysis.

[0416] Step 2:

[0417] The terminal encrypts the acquired audio and video data and transmits it to the server via the internet. The input is the raw data acquired in step 1, which is delivered to the server while maintaining security during transmission. The output is an encrypted data flow.

[0418] Step 3:

[0419] The server inputs the received audio and video data into its analysis engine, where a generative AI model performs real-time analysis. The input consists of digital audio and video, which the AI ​​model processes and outputs the baby's emotional state (e.g., hunger, sleepiness, desire to play, etc.). This output is data indicating the baby's specific emotional state.

[0420] Step 4:

[0421] The device acquires the parent's voice and analyzes it using an emotion engine. The input is the parent's words and tone of voice, which is then subjected to emotion analysis to output the user's emotional state (e.g., stress, fatigue, feeling of security). This output is data that indicates the user's feelings.

[0422] Step 5:

[0423] The server integrates and analyzes the emotional states of both the baby and the user to determine the optimal course of action. The input is the emotional data obtained in steps 3 and 4, and an AI model is used to design responses that address the baby's needs while supporting the user. The output is a set of specific instructions and recommendations.

[0424] Step 6:

[0425] The server sends the decided course of action to the terminal. The input is the course of action generated on the server side, which is then transmitted to the terminal as a procedure manual. The output is a digital message containing instructions to be followed.

[0426] Step 7:

[0427] The terminal controls connected smart devices to implement the received countermeasures. The input consists of instructions from the server, which execute specific actions such as playing music or adjusting lighting. The output is the optimized state of the environment settings.

[0428] (Application Example 2)

[0429] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0430] There is a need for a system that can detect abnormal situations and safety concerns within the home based on a comprehensive assessment that takes into account the emotional state of each individual and the overall home environment, and to respond appropriately. Conventional technologies only focus on individual emotions and behaviors, making it difficult to comprehensively grasp the safety situation within the home and provide appropriate countermeasures in real time.

[0431] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0432] In this invention, the server includes means for receiving audio and video data from an acquisition device, means for analyzing the audio and video data and inferring the subject's behavior or emotional state, and means for evaluating the safety status within the home based on the audio characteristics of the subject and neighbors. This makes it possible to ensure safety and provide prompt countermeasures tailored to the situation and individual emotions within the home.

[0433] An "acquisition device" is a device used to receive audio and video data, and is responsible for acquiring data within a home or specific environment.

[0434] "Audio data" refers to audio or sound information represented in digital format, and is used for analysis and evaluation.

[0435] "Video data" refers to visual information represented in digital format, and is used for analysis and evaluation.

[0436] A "subject" is an individual whose emotional state and behavior are inferred from the object observed through audio and video data.

[0437] "Inference" is the act of analyzing acquired data to determine the emotional state and behavior of a subject.

[0438] "Vocal features" are specific elements extracted from voice data and serve as indicators of an individual's emotional state and environmental characteristics.

[0439] A "means for evaluating the safety status within the home" is a system that analyzes the overall safety of the environment based on acquired data and provides an appropriate assessment.

[0440] In the system implementing this invention, a server and a terminal play important roles. First, the terminal collects audio and video data through an acquisition device installed in the home. The acquired data is then transmitted to the server. The server uses audio analysis software (e.g., speech recognition API) to analyze the audio data and image analysis software (e.g., image recognition API) to analyze the video data to infer the behavior and emotional state of the subject.

[0441] The server assesses the safety status of the home based on the inferred emotional state of the subject, while also considering voice characteristics. For example, an AI-powered analysis platform learns voices and sounds within the home to detect anomalies. Next, a generative AI model is used to determine appropriate countermeasures based on the safety status assessment. This makes it possible to present appropriate countermeasures in real time. For example, if someone in the family is experiencing stress, it will suggest relaxing content or activities.

[0442] These countermeasures are communicated to the user via the device, and automatically controlled devices are appropriately managed. For example, actions such as adjusting lighting or playing music can be performed. This gives users a sense of security within their homes and enables proactive safety management.

[0443] As a concrete example, when a baby starts crying, the device collects the audio data, and the server analyzes it. If it determines that the baby is hungry, the server notifies the user to prepare milk, and the system then starts playing relaxing music. This entire process is achieved by utilizing IoT devices and cloud computing technology.

[0444] Example of a prompt:

[0445] "To ensure safety within the home, analyze audio and video in real time and suggest appropriate responses. Explain how to control smart devices, taking into account the emotions of both the baby and the caregiver."

[0446] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0447] Step 1:

[0448] The terminal collects audio and video data using acquisition equipment installed in the home. This input data consists of audio and video captures from within the home and is necessary for subsequent analysis. The collected data is transmitted to the server in real time.

[0449] Step 2:

[0450] The server analyzes the received audio data using speech analysis software. This extracts speech features and generates basic information for determining the subject's emotions and the situation within the home. The output of this speech analysis includes feature quantities, such as crying patterns and voice tone.

[0451] Step 3:

[0452] The server analyzes the video data using image analysis software. It analyzes the subject's actions in detail from the video data and saves the results to the server's database. The analysis results output indicators that show abnormalities or warning signs of behavior.

[0453] Step 4:

[0454] The server integrates features derived from audio and video data and uses a generative AI model to infer the subject's emotional state. In this step, prompt sentences are used to enable the AI ​​model to infer emotion, providing the basis for appropriate responses.

[0455] Step 5:

[0456] The server assesses the safety status and determines appropriate countermeasures based on inferred emotional states and voice characteristics within the home. Using a generative AI model, it generates instructions to quickly provide responses in the event of an anomaly.

[0457] Step 6:

[0458] The device notifies the user of the countermeasures output from the server and executes control of the smart device based on those countermeasures. This includes specific actions such as playing music or adjusting lighting. The aim is to give the user a sense of security within their home.

[0459] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0460] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0461] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0462] [Third Embodiment]

[0463] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0464] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0465] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0466] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0467] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0468] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0469] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0470] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0471] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0472] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0473] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0474] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0475] The childcare support system of the present invention analyzes the baby's behavior and emotional state using audio and video data acquired around the baby, and proposes appropriate responses. An embodiment of this system is described below.

[0476] System Overview

[0477] 1. Data Acquisition Method

[0478] A device (such as a smartphone) is placed near the baby and uses a microphone and camera to acquire audio and video data. The device then sends the acquired data to a cloud server. At this stage, the data is processed in real time, so technologies are applied to minimize communication delays.

[0479] 2. Data Analysis Process

[0480] The server analyzes the received data. For audio data, it analyzes the tone, volume, and sound patterns of the voice to infer emotional states such as hunger or discomfort from the crying. For video data, it uses computer vision technology to analyze the baby's movements and facial expressions to identify behavioral patterns.

[0481] 3. Making predictions and deciding on countermeasures

[0482] Based on the analysis results, the server estimates the baby's current condition and determines the optimal course of action. This is done using a dedicated machine learning algorithm that provides suggestions tailored to the individual characteristics of each baby.

[0483] 4. Integration with smart devices

[0484] The device then implements the countermeasures received from the server. This includes steps such as coordinating with smart devices like smart speakers and smart lights to create an appropriate environment. For example, it might play soft music to calm a baby or adjust the room lighting to create a sleep-friendly environment.

[0485] Specific example

[0486] Example 1: Analysis and response to crying sounds

[0487] The device detects the baby's crying and immediately sends the signal to the server. The server analyzes the crying pattern and determines that the baby is likely hungry. Based on this, the action "notify the user to feed the baby milk" is selected. The device then sends a notification to the user saying, "The baby seems hungry. Please prepare some milk."

[0488] Example 2: Environmental adjustments to promote sleep

[0489] If the video data detects that the baby is frequently yawning or rubbing its eyes, the server infers that the baby is sleepy. Based on this, the device instructs the smart light to dim the room's brightness and play relaxing music.

[0490] In this way, the present invention can provide detailed childcare support tailored to the baby's condition while working in conjunction with a variety of devices.

[0491] The following describes the processing flow.

[0492] Step 1:

[0493] The device is placed in the baby's environment and acquires audio and video data in real time via a microphone and camera. This includes the baby's cries, ambient sounds, and video of the baby's movements and facial expressions.

[0494] Step 2:

[0495] The device temporarily stores the acquired data, performs preprocessing such as noise reduction and data compression, and then sends the data to a server in the cloud via the internet.

[0496] Step 3:

[0497] When the server receives audio data, it begins analyzing the crying patterns using speech recognition technology. Here, it analyzes the volume, rhythm, and pitch of the sounds to infer the baby's emotional state (e.g., hunger, sleepiness, discomfort).

[0498] Step 4:

[0499] Simultaneously, the server analyzes the video data and uses computer vision algorithms to identify the baby's movements and posture. For example, it can detect actions such as rubbing the eyes or flailing the arms and legs.

[0500] Step 5:

[0501] The server analyzes the audio and video results to estimate the baby's current condition. This involves applying machine learning models and combining them with predictive models based on historical data to achieve highly accurate predictions.

[0502] Step 6:

[0503] The server determines the appropriate course of action based on the prediction results. For each situation, it formulates a specific action plan, such as "give milk" or "prepare for a nap."

[0504] Step 7:

[0505] The device executes instructions received from the server and sends a notification to the user. This notification includes recommended actions and provides guidance to facilitate childcare.

[0506] Step 8:

[0507] The device connects with smart devices and, when necessary, plays music or adjusts lighting to create a comfortable environment for the baby. The goal is to ensure that the proposed countermeasures are effectively implemented.

[0508] (Example 1)

[0509] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0510] In modern society, the burden of childcare is significant, and there is a particular need to quickly and accurately understand the physical and emotional state of babies. However, current technology makes it difficult to analyze a baby's condition in real time and automatically implement appropriate countermeasures tailored to their individual characteristics. Furthermore, there is the challenge that relatives and caregivers often have difficulty receiving immediate support regarding childcare while going about their daily lives.

[0511] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0512] In this invention, the server includes means for receiving acoustic and image information from an acquisition device, means for analyzing the acoustic and image information to infer the subject's behavior or emotional state, and means for determining and displaying appropriate action based on the inference. This enables real-time analysis of the baby's condition and highly accurate childcare support tailored to individual characteristics.

[0513] An "acquisition device" is a device used to receive acoustic and image information, and is mainly equipped with a microphone and a camera.

[0514] "Acoustic information" refers to the sound emitted from an object, acquired as digital data, and is characterized by its tone, volume, and pattern.

[0515] "Image information" refers to still images or video data captured by a camera or similar device, which visually records the actions and facial expressions of a subject.

[0516] The "subject" is the person being monitored for childcare support purposes, usually a baby, but can also include other individuals designated by the user.

[0517] "Inference" is the process of analyzing acquired acoustic and visual information to predict the state or emotions of a subject.

[0518] "Appropriate action" refers to suggestions for actions or environmental adjustments that should be taken for the subject based on their presumed condition.

[0519] "Intelligent devices" are devices that are connected to a network and have the ability to automatically switch their operation based on external commands. Examples include smart speakers and smart lights.

[0520] "Real-time" refers to a process where data acquisition, analysis, and result delivery occur immediately, with processing completed within a timeframe that minimizes delays.

[0521] A "generative AI model" is a form of artificial intelligence that learns from past data and has the ability to make predictions and suggestions based on new data.

[0522] This invention is a childcare support system that reduces the burden of childcare and analyzes the baby's condition in real time. This system uses a terminal (e.g., a smartphone) as an acquisition device to collect acoustic and image information. The terminal is placed near the baby and acquires data using a microphone and camera. The acquired data is transmitted in real time to a server in the cloud. The server performs high-speed data processing and uses efficient data compression technology to minimize communication delays.

[0523] The server uses a speech analysis engine to analyze acoustic information. Specifically, it analyzes the tone, volume, and sound patterns of the voice to infer the baby's emotional state. Image information is analyzed using computer vision technology through a video analysis engine to identify the baby's movements and facial expressions.

[0524] The server inputs the analysis results into a generating AI model to generate countermeasures tailored to the individual characteristics of each baby. These countermeasures are optimized based on the child's individual condition and may include, for example, environmental adjustments to promote sleep or offering milk when the baby is hungry. The generated countermeasures are notified to the user via the terminal.

[0525] Furthermore, the device can automatically create an appropriate environment by linking with intelligent devices. For example, it can adjust the brightness of smart lights and instruct smart speakers to play relaxation music.

[0526] As a concrete example, a device that detects a baby crying sends the acoustic information to a server, and a generating AI model infers that the crying is due to hunger. In this case, the device notifies the user, "Please prepare to feed the baby milk." An example of a prompt message is, "Analyze the baby's condition from the audio and video data when the baby is crying and suggest appropriate childcare support."

[0527] In this way, the invention provides support tailored to the baby's condition and reduces the burden on caregivers.

[0528] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0529] Step 1:

[0530] The device is placed near the baby and uses a microphone to acquire ambient acoustic information and a camera to collect image information. The input to this process is the baby's cries and movements, and the output is digitized acoustic and image information. The device transmits this data to a cloud server in real time.

[0531] Step 2:

[0532] The server receives acoustic information transmitted from the terminal. Using the acquired acoustic data as input, the audio analysis engine analyzes the data. Specifically, it analyzes the tone, volume, and pattern of the sound to infer the baby's emotional state. The output is a possible state indicated by the crying (e.g., hunger, discomfort).

[0533] Step 3:

[0534] The server also receives image information and performs analysis using computer vision technology. Based on the image data obtained as input, the video analysis engine analyzes the baby's movements (e.g., yawning, rubbing eyes) and facial expressions. This identifies the baby's state from its behavioral patterns and facial expressions, and provides insights into that state as output.

[0535] Step 4:

[0536] The server inputs the results of the analysis of acoustic and image information into a generating AI model. This model accurately predicts the baby's current state from this data and determines the appropriate course of action based on that prediction. The output is the optimal course of action tailored to the baby's characteristics (e.g., a suggestion to change the diaper, instructions to prepare milk).

[0537] Step 5:

[0538] The device receives a notification of a course of action from the server. Based on this notification, it displays a message on the device's screen prompting the user to take specific action and can, if necessary, link with intelligent devices. For example, the device might send a notification to the user saying, "Please feed your baby milk," and instruct them to adjust the brightness of a smart light or play relaxing music.

[0539] Step 6:

[0540] After receiving a notification from the device, the user takes action on the suggested course of action. Specifically, this might involve preparing milk or changing diapers. They can also observe the baby's condition within the environment set up by the smart device and take additional action as needed.

[0541] Through these steps, the system provides users with immediate support tailored to the baby's various needs.

[0542] (Application Example 1)

[0543] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0544] Traditional product inspection methods have limitations when it comes to manual quality control. In particular, detecting defects using sound and video is labor-intensive and time-consuming, and its accuracy is also limited. There is a need to provide a method that effectively solves these problems and enables rapid and precise product quality control.

[0545] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0546] In this invention, the server includes means for receiving audio and video data from an acquisition device, means for analyzing the audio and video data to infer the behavior or emotional state of the object, and means for detecting specific disorder in a product and proposing countermeasures. This enables automated, highly accurate audio and video-based product inspection and rapid notification to managers.

[0547] An "acquisition device" is a device used to acquire audio and video data from a target in real time.

[0548] "Audio data" refers to sound information obtained through an acquisition device, and is data for analysis, including volume and sound quality.

[0549] "Video data" refers to visual information obtained through an acquisition device, and is data used for analyzing movement and appearance.

[0550] "Object" refers to an item or subject that is monitored or analyzed by the system.

[0551] "Intelligent devices" refer to devices that have the ability to take appropriate actions or give instructions, and include smart devices.

[0552] "Disorder" refers to abnormalities or defects in the state of a product or object that deviate from the normal pattern.

[0553] "Countermeasures" refer to specific actions or measures that are determined to be taken based on the results that the system has inferred or detected.

[0554] "Administrator" refers to the person or department responsible for the operation of the system and the implementation of countermeasures.

[0555] The system implementing this application primarily aims to automatically manage the quality of objects by analyzing audio and video data. The server receives audio and video data from acquisition devices in real time and analyzes this data. For audio data, an audio analysis algorithm is used to detect abnormal or unusual sounds. For video data, computer vision technology is used to identify abnormalities in the surface condition and shape of the product.

[0556] This process utilizes cloud servers and intelligent devices to streamline data analysis. The calculations employ programming languages ​​such as Python and image processing techniques using libraries like OpenCV. Based on the analysis results, the server controls the intelligent devices and proposes appropriate countermeasures.

[0557] As a concrete example, in product inspection at a factory, an acquisition device moves along the production line, acquiring audio and video data of the products. The acquired data is sent to a server, and if any abnormalities in the sound or image are automatically detected, the system notifies the administrator and instructs them on how to respond.

[0558] An example of a prompt is, "Please tell me about the most important anomaly detection system for product inspection." This prompt is used to provide specific information about anomaly detection using a generative AI model.

[0559] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0560] Step 1:

[0561] The terminal collects audio and video data of the target object through an acquisition device. The audio data, as input, is the waveform data of sound detected by the sensor, and the video data includes frame images from the camera. This data is then prepared to be transmitted to the server in real time.

[0562] Step 2:

[0563] The server receives audio data transmitted from the terminal and applies an audio analysis algorithm to analyze any unusual sounds or patterns. If the sound pattern differs from the normal pattern, this analysis detects the abnormal pattern as an output.

[0564] Step 3:

[0565] The server receives video data transmitted from the terminal and inspects the surface condition of the object using image analysis. Computer vision technology is used to process the image data and identify abnormal shapes, scratches, and other defects. The results of this analysis are then generated as output data.

[0566] Step 4:

[0567] The server uses a generative AI model to determine the necessary countermeasures based on the results of audio and video analysis. At this stage, the analysis results are converted into prompts, and the generative AI model generates the optimal countermeasure as output.

[0568] Step 5:

[0569] The server controls the intelligent device and executes the determined response. This control includes sending notifications and issuing instructions for the device's operation. The user is also notified of the anomaly and the recommended course of action.

[0570] Step 6:

[0571] The user receives notifications from the server and takes actual corrective actions based on the proposed solutions. Efficient responses are confirmed through the system's operation and user interface.

[0572] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0573] This invention provides a system for childcare support that takes into account not only the emotional state and behavior of the baby, but also the emotions of the user, the caregiver. The embodiments of this system are described in detail below.

[0574] Overall system configuration

[0575] 1. Data Acquisition and Analysis

[0576] The device acquires audio and video data about the baby. This includes crying, ambient sounds, movements, and facial expressions. This data is sent to a server in the cloud and analyzed in real time by a dedicated analysis engine.

[0577] 2. Predicting the baby's emotional state

[0578] The server infers the baby's emotions and behavior based on the analyzed audio and video. This uses an emotion analysis algorithm, which makes it possible to understand what needs the baby has.

[0579] 3. User emotion recognition

[0580] The device further uses an emotion engine to recognize the user's emotional state through the user's voice input and actions (e.g., tone of voice and word choice). This is done to identify emotions such as stress, fatigue, or relief.

[0581] 4. Decision and presentation of countermeasures

[0582] The server determines a course of action that takes into account the emotional states of both the baby and the user. This course of action is tailored to the baby's needs while also including measures to soothe and support the user's emotions.

[0583] 5. Smart device control

[0584] The device controls connected smart devices based on the countermeasures taken. For example, if the user is feeling stressed, relaxing music may be played or the lighting may be adjusted.

[0585] Specific example

[0586] Example 1: Baby's hunger and user fatigue

[0587] The server detects a baby crying and infers that the baby is hungry. At the same time, the emotion engine recognizes signs of fatigue in the user's voice. In this case, the server decides on a course of action: "gently notify the user to prepare the baby's milk and play relaxing music in the meantime." The device follows these instructions, notifying the user and playing the music.

[0588] Example 2: A baby who wants to play and a user who feels safe.

[0589] When video analysis recognizes the baby's desire to play and the user's voice indicates they feel at ease, the server suggests a solution such as extending playtime. The device then plays upbeat music and automatically adjusts the room temperature to a suitable level for playtime.

[0590] Thus, this invention grasps the emotional states of both the baby and the user, providing new value to conventional childcare support by taking the user's emotions into consideration.

[0591] The following describes the processing flow.

[0592] Step 1:

[0593] The device is placed near the baby and uses its built-in microphone and camera to capture audio and video data. This includes the baby's cries, facial expressions, and body movements. The data is converted into a digital format and prepared for analysis.

[0594] Step 2:

[0595] The device transmits the acquired audio and video data to a cloud server. This communication uses a secure protocol to minimize the risk of data leakage. The transmitted data is processed in real time, taking into account any temporal delays.

[0596] Step 3:

[0597] When the server receives audio data, it uses a speech recognition system to analyze the crying patterns. From the identified audio features, it infers the baby's emotional state (e.g., hunger, sleepiness). A pre-trained emotion model is used for this analysis.

[0598] Step 4:

[0599] The server processes the video data and uses computer vision algorithms to analyze the baby's facial expressions and movements. For example, it can determine whether the baby wants to play or is sleepy based on their facial expressions and limb movements.

[0600] Step 5:

[0601] The server recognizes the user's emotions based on voice input and behavioral logs, in addition to the baby's emotional state. The emotion engine identifies stress and feelings of security from the user's voice and sends the resulting data to the emotion analysis system.

[0602] Step 6:

[0603] By combining this data, the server generates a response plan that takes into account the emotional state of both the baby and the user. For example, specific actions such as "feed the baby milk" or "recommend playing relaxation music" are determined.

[0604] Step 7:

[0605] The device sends a notification to the user based on the countermeasures received from the server. The notification presents an action plan tailored to the baby's condition and the user's emotions.

[0606] Step 8:

[0607] The device connects with smart devices and automatically adjusts the environment as needed. For example, it optimizes the childcare environment by lowering the brightness of smart lights or playing music.

[0608] Through this series of processes, the system provides childcare support that meets the needs of both the baby and the user.

[0609] (Example 2)

[0610] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0611] Conventional childcare support systems have struggled to provide comprehensive solutions that consider not only the baby's emotions and needs, but also the parents' emotions and stress levels. Furthermore, providing optimal solutions in real time, based on the conditions of both the baby and the parents, has been a technical challenge.

[0612] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0613] In this invention, the server includes means for receiving audio and video data from an acquisition device, means for analyzing the audio and video data to estimate the emotional states of the baby and the caregiver, and means for determining appropriate countermeasures that take into account the emotional states of the baby and the caregiver based on the estimation results. This enables comprehensive childcare support that simultaneously satisfies the needs of the baby and the emotions of the caregiver.

[0614] An "acquisition device" is a device for receiving audio and video data.

[0615] "Analyzing audio and video data" means processing the received data to determine the subject's behavior and emotional state.

[0616] "Inferring behavioral and emotional states" means estimating the subject's current psychological or physical state based on analyzed data.

[0617] "Determining appropriate countermeasures" means using the prediction results to select the optimal actions and instructions for the target and users.

[0618] "Controlling smart devices" means operating or managing connected electronic devices based on predetermined countermeasures.

[0619] A "generative AI model" refers to an algorithm that learns from data and accurately predicts the state of a situation.

[0620] "Generating optimal action plans" means creating specific action guidelines tailored to the target audience and the user's situation.

[0621] This invention is a system aimed at supporting childcare, and it provides functions that take into account not only the emotional state and behavior of the baby, but also the emotions of the caregiver who uses the system. This system is implemented as follows.

[0622] About the configuration and equipment

[0623] The device is equipped with a camera and microphone to acquire audio and video data about the baby. The data acquired from these sensors includes crying, ambient sounds, movements, and facial expressions. This collected data is transmitted to a server via the internet.

[0624] About data analysis

[0625] The server analyzes data in real time using an analysis engine built on the cloud. This analysis uses a generative AI model to infer the baby's emotional state and behavior from audio and video. The analyzed information is processed by an emotion analysis algorithm within the system to specifically identify the baby's needs. Similarly, the user's emotional state is also analyzed from the voice input.

[0626] Proposal of countermeasures

[0627] The server takes into account the emotional states of both the baby and the user to determine the appropriate course of action. For example, if the server detects both the baby's hunger and the user's fatigue simultaneously, it will prompt the user to prepare milk and send instructions to the device recommending relaxing music.

[0628] Specific examples and prompt statements

[0629] For example, if a baby cries, and the server determines that the baby is hungry, and at the same time the user's calm tone indicates low stress levels, the server will notify the user, "Please enjoy some soothing music while you prepare your baby's milk."

[0630] Example prompt: "The baby is crying continuously. It may need milk. Gently notify the user and instruct them to play relaxing music."

[0631] In this way, the system aims to provide environmentally conscious childcare support and create a comfortable environment for both the baby and the user.

[0632] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0633] Step 1:

[0634] The device uses a camera and microphone to collect audio and video data of the baby's surroundings. Inputs include ambient sounds, the baby's movements, and cries, which are captured as digital data. The collected data is temporarily stored within the device and prepared for analysis.

[0635] Step 2:

[0636] The terminal encrypts the acquired audio and video data and transmits it to the server via the internet. The input is the raw data acquired in step 1, which is delivered to the server while maintaining security during transmission. The output is an encrypted data flow.

[0637] Step 3:

[0638] The server inputs the received audio and video data into its analysis engine, where a generative AI model performs real-time analysis. The input consists of digital audio and video, which the AI ​​model processes and outputs the baby's emotional state (e.g., hunger, sleepiness, desire to play, etc.). This output is data indicating the baby's specific emotional state.

[0639] Step 4:

[0640] The device acquires the parent's voice and analyzes it using an emotion engine. The input is the parent's words and tone of voice, which is then subjected to emotion analysis to output the user's emotional state (e.g., stress, fatigue, feeling of security). This output is data that indicates the user's feelings.

[0641] Step 5:

[0642] The server integrates and analyzes the emotional states of both the baby and the user to determine the optimal course of action. The input is the emotional data obtained in steps 3 and 4, and an AI model is used to design responses that address the baby's needs while supporting the user. The output is a set of specific instructions and recommendations.

[0643] Step 6:

[0644] The server sends the decided course of action to the terminal. The input is the course of action generated on the server side, which is then transmitted to the terminal as a procedure manual. The output is a digital message containing instructions to be followed.

[0645] Step 7:

[0646] The terminal controls connected smart devices to implement the received countermeasures. The input is a command from the server, which executes specific actions such as playing music or adjusting lighting. The output is the optimized state of the environment settings.

[0647] (Application Example 2)

[0648] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0649] There is a need for a system that can detect abnormal situations and safety concerns within the home based on a comprehensive assessment that takes into account the emotional state of each individual and the overall home environment, and to respond appropriately. Conventional technologies only focus on individual emotions and behaviors, making it difficult to comprehensively grasp the safety situation within the home and provide appropriate countermeasures in real time.

[0650] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0651] In this invention, the server includes means for receiving audio and video data from an acquisition device, means for analyzing the audio and video data and inferring the subject's behavior or emotional state, and means for evaluating the safety status within the home based on the audio characteristics of the subject and neighbors. This makes it possible to ensure safety and provide prompt countermeasures tailored to the situation and individual emotions within the home.

[0652] An "acquisition device" is a device used to receive audio and video data, and is responsible for acquiring data within a home or specific environment.

[0653] "Audio data" refers to audio or sound information represented in digital format, and is used for analysis and evaluation.

[0654] "Video data" refers to visual information represented in digital format, and is used for analysis and evaluation.

[0655] A "subject" is an individual observed through audio and video data, whose emotional state and behavior are inferred from it.

[0656] "Inference" is the act of analyzing acquired data to determine the emotional state and behavior of a subject.

[0657] "Vocal features" are specific elements extracted from voice data and serve as indicators of an individual's emotional state and environmental characteristics.

[0658] A "means for evaluating the safety status within the home" is a system that analyzes the overall safety of the environment based on acquired data and provides an appropriate assessment.

[0659] In the system implementing this invention, a server and a terminal play important roles. First, the terminal collects audio and video data through an acquisition device installed in the home. The acquired data is then transmitted to the server. The server uses audio analysis software (e.g., speech recognition API) to analyze the audio data and image analysis software (e.g., image recognition API) to analyze the video data to infer the behavior and emotional state of the subject.

[0660] The server assesses the safety status of the home based on the inferred emotional state of the subject, while also considering voice characteristics. For example, an AI-powered analysis platform learns voices and sounds within the home to detect anomalies. Next, a generative AI model is used to determine appropriate countermeasures based on the safety status assessment. This makes it possible to present appropriate countermeasures in real time. For example, if someone in the family is experiencing stress, it will suggest relaxing content or activities.

[0661] These countermeasures are communicated to the user via the device, and automatically controlled devices are appropriately managed. For example, actions such as adjusting lighting or playing music can be performed. This gives users a sense of security within their homes and enables proactive safety management.

[0662] As a concrete example, when a baby starts crying, the device collects the audio data, and the server analyzes it. If it determines that the baby is hungry, the server notifies the user to prepare milk, and the system then starts playing relaxing music. This entire process is achieved by utilizing IoT devices and cloud computing technology.

[0663] Example of a prompt:

[0664] "To ensure safety within the home, analyze audio and video in real time and suggest appropriate responses. Explain how to control smart devices, taking into account the emotions of both the baby and the caregiver."

[0665] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0666] Step 1:

[0667] The terminal collects audio and video data using acquisition equipment installed in the home. This input data consists of audio and video captures from within the home and is necessary for subsequent analysis. The collected data is transmitted to the server in real time.

[0668] Step 2:

[0669] The server analyzes the received audio data using speech analysis software. This extracts speech features and generates basic information for determining the subject's emotions and the situation within the home. The output of this speech analysis includes feature quantities, such as crying patterns and voice tone.

[0670] Step 3:

[0671] The server analyzes the video data using image analysis software. It analyzes the subject's actions in detail from the video data and saves the results to the server's database. The analysis results output indicators that show abnormalities or warning signs of behavior.

[0672] Step 4:

[0673] The server integrates features derived from audio and video data and uses a generative AI model to infer the subject's emotional state. In this step, prompt sentences are used to enable the AI ​​model to infer emotion, providing the basis for appropriate responses.

[0674] Step 5:

[0675] The server assesses the safety status and determines appropriate countermeasures based on inferred emotional states and voice characteristics within the home. Using a generative AI model, it generates instructions to quickly provide responses in the event of an anomaly.

[0676] Step 6:

[0677] The device notifies the user of the countermeasures output from the server and executes control of the smart device based on those countermeasures. This includes specific actions such as playing music or adjusting lighting. The aim is to give the user a sense of security within their home.

[0678] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0679] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0680] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0681] [Fourth Embodiment]

[0682] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0683] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0684] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0685] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0686] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0687] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0688] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0689] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0690] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0691] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0692] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0693] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0694] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0695] The childcare support system of the present invention analyzes the baby's behavior and emotional state using audio and video data acquired around the baby, and proposes appropriate responses. An embodiment of this system is described below.

[0696] System Overview

[0697] 1. Data Acquisition Method

[0698] A device (such as a smartphone) is placed near the baby and uses a microphone and camera to acquire audio and video data. The device then sends the acquired data to a cloud server. At this stage, the data is processed in real time, so technologies are applied to minimize communication delays.

[0699] 2. Data Analysis Process

[0700] The server analyzes the received data. For audio data, it analyzes the tone, volume, and sound patterns of the voice to infer emotional states such as hunger or discomfort from the crying. For video data, it uses computer vision technology to analyze the baby's movements and facial expressions to identify behavioral patterns.

[0701] 3. Making predictions and deciding on countermeasures

[0702] Based on the analysis results, the server estimates the baby's current condition and determines the optimal course of action. This is done using a dedicated machine learning algorithm that provides suggestions tailored to the individual characteristics of each baby.

[0703] 4. Integration with smart devices

[0704] The device then implements the countermeasures received from the server. This includes steps such as coordinating with smart devices like smart speakers and smart lights to create an appropriate environment. For example, it might play soft music to calm a baby or adjust the room lighting to create a sleep-friendly environment.

[0705] Specific example

[0706] Example 1: Analysis and response to crying sounds

[0707] The device detects the baby's crying and immediately sends the signal to the server. The server analyzes the crying pattern and determines that the baby is likely hungry. Based on this, the action "notify the user to feed the baby milk" is selected. The device then sends a notification to the user saying, "The baby seems hungry. Please prepare some milk."

[0708] Example 2: Environmental adjustments to promote sleep

[0709] If the video data detects that the baby is frequently yawning or rubbing its eyes, the server infers that the baby is sleepy. Based on this, the device instructs the smart light to dim the room's brightness and play relaxing music.

[0710] In this way, the present invention can provide detailed childcare support tailored to the baby's condition while working in conjunction with a variety of devices.

[0711] The following describes the processing flow.

[0712] Step 1:

[0713] The device is placed in the baby's environment and acquires audio and video data in real time via a microphone and camera. This includes the baby's cries, ambient sounds, and video of the baby's movements and facial expressions.

[0714] Step 2:

[0715] The device temporarily stores the acquired data, performs preprocessing such as noise reduction and data compression, and then sends the data to a server in the cloud via the internet.

[0716] Step 3:

[0717] When the server receives audio data, it begins analyzing the crying patterns using speech recognition technology. Here, it analyzes the volume, rhythm, and pitch of the sounds to infer the baby's emotional state (e.g., hunger, sleepiness, discomfort).

[0718] Step 4:

[0719] Simultaneously, the server analyzes the video data and uses computer vision algorithms to identify the baby's movements and posture. For example, it can detect actions such as rubbing the eyes or flailing the arms and legs.

[0720] Step 5:

[0721] The server analyzes the audio and video results to estimate the baby's current condition. This involves applying machine learning models and combining them with predictive models based on historical data to achieve highly accurate predictions.

[0722] Step 6:

[0723] The server determines the appropriate course of action based on the prediction results. For each situation, it formulates a specific action plan, such as "give milk" or "prepare for a nap."

[0724] Step 7:

[0725] The device executes instructions received from the server and sends a notification to the user. This notification includes recommended actions and provides guidance to facilitate childcare.

[0726] Step 8:

[0727] The device connects with smart devices and, when necessary, plays music or adjusts lighting to create a comfortable environment for the baby. The goal is to ensure that the proposed countermeasures are effectively implemented.

[0728] (Example 1)

[0729] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0730] In modern society, the burden of childcare is significant, and there is a particular need to quickly and accurately understand the physical and emotional state of babies. However, current technology makes it difficult to analyze a baby's condition in real time and automatically implement appropriate countermeasures tailored to their individual characteristics. Furthermore, there is the challenge that relatives and caregivers often have difficulty receiving immediate support regarding childcare while going about their daily lives.

[0731] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0732] In this invention, the server includes means for receiving acoustic and image information from an acquisition device, means for analyzing the acoustic and image information to infer the subject's behavior or emotional state, and means for determining and displaying appropriate action based on the inference. This enables real-time analysis of the baby's condition and highly accurate childcare support tailored to individual characteristics.

[0733] An "acquisition device" is a device used to receive acoustic and image information, and is mainly equipped with a microphone and a camera.

[0734] "Acoustic information" refers to the sound emitted from an object, acquired as digital data, and is characterized by its tone, volume, and pattern.

[0735] "Image information" refers to still images or video data captured by a camera or similar device, which visually records the actions and facial expressions of a subject.

[0736] The "subject" is the person being monitored for childcare support purposes, usually a baby, but can also include other individuals designated by the user.

[0737] "Inference" is the process of analyzing acquired acoustic and visual information to predict the state or emotions of a subject.

[0738] "Appropriate action" refers to suggestions for actions or environmental adjustments that should be taken for the subject based on their presumed condition.

[0739] "Intelligent devices" are devices that are connected to a network and have the ability to automatically switch their operation based on external commands. Examples include smart speakers and smart lights.

[0740] "Real-time" refers to a process where data acquisition, analysis, and result delivery occur immediately, with processing completed within a timeframe that minimizes delays.

[0741] A "generative AI model" is a form of artificial intelligence that learns from past data and has the ability to make predictions and suggestions based on new data.

[0742] This invention is a childcare support system that reduces the burden of childcare and analyzes the baby's condition in real time. This system uses a terminal (e.g., a smartphone) as an acquisition device to collect acoustic and image information. The terminal is placed near the baby and acquires data using a microphone and camera. The acquired data is transmitted in real time to a server in the cloud. The server performs high-speed data processing and uses efficient data compression technology to minimize communication delays.

[0743] The server uses a speech analysis engine to analyze acoustic information. Specifically, it analyzes the tone, volume, and sound patterns of the voice to infer the baby's emotional state. Image information is analyzed using computer vision technology through a video analysis engine to identify the baby's movements and facial expressions.

[0744] The server inputs the analysis results into a generating AI model to generate countermeasures tailored to the individual characteristics of each baby. These countermeasures are optimized based on the child's individual condition and may include, for example, environmental adjustments to promote sleep or offering milk when the baby is hungry. The generated countermeasures are notified to the user via the terminal.

[0745] Furthermore, the device can automatically create an appropriate environment by linking with intelligent devices. For example, it can adjust the brightness of smart lights and instruct smart speakers to play relaxation music.

[0746] As a concrete example, a device that detects a baby crying sends the acoustic information to a server, and a generating AI model infers that the crying is due to hunger. In this case, the device notifies the user, "Please prepare to feed the baby milk." An example of a prompt message is, "Analyze the baby's condition from the audio and video data when the baby is crying and suggest appropriate childcare support."

[0747] In this way, the invention provides support tailored to the baby's condition and reduces the burden on caregivers.

[0748] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0749] Step 1:

[0750] The device is placed near the baby and uses a microphone to acquire ambient acoustic information and a camera to collect image information. The input to this process is the baby's cries and movements, and the output is digitized acoustic and image information. The device transmits this data to a cloud server in real time.

[0751] Step 2:

[0752] The server receives acoustic information transmitted from the terminal. Using the acquired acoustic data as input, the audio analysis engine analyzes the data. Specifically, it analyzes the tone, volume, and pattern of the sound to infer the baby's emotional state. The output is a possible state indicated by the crying (e.g., hunger, discomfort).

[0753] Step 3:

[0754] The server also receives image information and performs analysis using computer vision technology. Based on the image data obtained as input, the video analysis engine analyzes the baby's movements (e.g., yawning, rubbing eyes) and facial expressions. This identifies the baby's state from its behavioral patterns and facial expressions, and provides insights into that state as output.

[0755] Step 4:

[0756] The server inputs the results of the analysis of acoustic and image information into a generating AI model. This model accurately predicts the baby's current state from this data and determines the appropriate course of action based on that prediction. The output is the optimal course of action tailored to the baby's characteristics (e.g., a suggestion to change the diaper, instructions to prepare milk).

[0757] Step 5:

[0758] The device receives a notification of a course of action from the server. Based on this notification, it displays a message on the device's screen prompting the user to take specific action and can, if necessary, link with intelligent devices. For example, the device might send a notification to the user saying, "Please feed your baby milk," and instruct them to adjust the brightness of a smart light or play relaxing music.

[0759] Step 6:

[0760] After receiving a notification from the device, the user takes action on the suggested course of action. Specifically, this might involve preparing milk or changing diapers. They can also observe the baby's condition within the environment set up by the smart device and take additional action as needed.

[0761] Through these steps, the system provides users with immediate support tailored to the baby's various needs.

[0762] (Application Example 1)

[0763] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0764] Traditional product inspection methods have limitations when it comes to manual quality control. In particular, detecting defects using sound and video is labor-intensive and time-consuming, and its accuracy is also limited. There is a need to provide a method that effectively solves these problems and enables rapid and precise product quality control.

[0765] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0766] In this invention, the server includes means for receiving audio and video data from an acquisition device, means for analyzing the audio and video data to infer the behavior or emotional state of the object, and means for detecting specific disorder in a product and proposing countermeasures. This enables automated, highly accurate audio and video-based product inspection and rapid notification to managers.

[0767] An "acquisition device" is a device used to acquire audio and video data from a target in real time.

[0768] "Audio data" refers to sound information obtained through an acquisition device, and is data for analysis, including volume and sound quality.

[0769] "Video data" refers to visual information obtained through an acquisition device, and is data used for analyzing movement and appearance.

[0770] "Object" refers to an item or subject that is monitored or analyzed by the system.

[0771] "Intelligent devices" refer to devices that have the ability to take appropriate actions or give instructions, and include smart devices.

[0772] "Disorder" refers to abnormalities or defects in the state of a product or object that deviate from the normal pattern.

[0773] "Countermeasures" refer to specific actions or measures that are determined to be taken based on the results that the system has inferred or detected.

[0774] "Administrator" refers to the person or department responsible for the operation of the system and the implementation of countermeasures.

[0775] The system implementing this application primarily aims to automatically manage the quality of objects by analyzing audio and video data. The server receives audio and video data from acquisition devices in real time and analyzes this data. For audio data, an audio analysis algorithm is used to detect abnormal or unusual sounds. For video data, computer vision technology is used to identify abnormalities in the surface condition and shape of the product.

[0776] This process utilizes cloud servers and intelligent devices to streamline data analysis. The calculations employ programming languages ​​such as Python and image processing techniques using libraries like OpenCV. Based on the analysis results, the server controls the intelligent devices and proposes appropriate countermeasures.

[0777] As a concrete example, in product inspection at a factory, an acquisition device moves along the production line, acquiring audio and video data of the products. The acquired data is sent to a server, and if any abnormalities in the sound or image are automatically detected, the system notifies the administrator and instructs them on how to respond.

[0778] An example of a prompt is, "Please tell me about the most important anomaly detection system for product inspection." This prompt is used to provide specific information about anomaly detection using a generative AI model.

[0779] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0780] Step 1:

[0781] The terminal collects audio and video data of the target object through an acquisition device. The audio data, as input, is the waveform data of sound detected by the sensor, and the video data includes frame images from the camera. This data is then prepared to be transmitted to the server in real time.

[0782] Step 2:

[0783] The server receives audio data transmitted from the terminal and applies an audio analysis algorithm to analyze any unusual sounds or patterns. If the sound pattern differs from the normal pattern, this analysis detects the abnormal pattern as an output.

[0784] Step 3:

[0785] The server receives video data transmitted from the terminal and inspects the surface condition of the object using image analysis. Computer vision technology is used to process the image data and identify abnormal shapes, scratches, and other defects. The results of this analysis are then generated as output data.

[0786] Step 4:

[0787] The server uses a generative AI model to determine the necessary countermeasures based on the results of audio and video analysis. At this stage, the analysis results are converted into prompts, and the generative AI model generates the optimal countermeasure as output.

[0788] Step 5:

[0789] The server controls the intelligent device and executes the determined response. This control includes sending notifications and issuing instructions for the device's operation. The user is also notified of the anomaly and the recommended course of action.

[0790] Step 6:

[0791] The user receives notifications from the server and takes actual corrective actions based on the proposed solutions. Efficient responses are confirmed through the system's operation and user interface.

[0792] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0793] This invention provides a system for childcare support that takes into account not only the emotional state and behavior of the baby, but also the emotions of the user, the caregiver. The embodiments of this system are described in detail below.

[0794] Overall system configuration

[0795] 1. Data Acquisition and Analysis

[0796] The device acquires audio and video data about the baby. This includes crying, ambient sounds, movements, and facial expressions. This data is sent to a server in the cloud and analyzed in real time by a dedicated analysis engine.

[0797] 2. Predicting the baby's emotional state

[0798] The server infers the baby's emotions and behavior based on the analyzed audio and video. This uses an emotion analysis algorithm, which makes it possible to understand what needs the baby has.

[0799] 3. User emotion recognition

[0800] The device further uses an emotion engine to recognize the user's emotional state through the user's voice input and actions (e.g., tone of voice and word choice). This is done to identify emotions such as stress, fatigue, or relief.

[0801] 4. Decision and presentation of countermeasures

[0802] The server determines a course of action that takes into account the emotional states of both the baby and the user. This course of action is tailored to the baby's needs while also including measures to soothe and support the user's emotions.

[0803] 5. Smart device control

[0804] The device controls connected smart devices based on the countermeasures taken. For example, if the user is feeling stressed, relaxing music may be played or the lighting may be adjusted.

[0805] Specific example

[0806] Example 1: Baby's hunger and user fatigue

[0807] The server detects a baby crying and infers that the baby is hungry. At the same time, the emotion engine recognizes signs of fatigue in the user's voice. In this case, the server decides on a course of action: "gently notify the user to prepare the baby's milk and play relaxing music in the meantime." The device follows these instructions, notifying the user and playing the music.

[0808] Example 2: A baby who wants to play and a user who feels safe.

[0809] When video analysis recognizes the baby's desire to play and the user's voice indicates they feel at ease, the server suggests a solution such as extending playtime. The device then plays upbeat music and automatically adjusts the room temperature to a suitable level for playtime.

[0810] Thus, this invention grasps the emotional states of both the baby and the user, providing new value to conventional childcare support by taking the user's emotions into consideration.

[0811] The following describes the processing flow.

[0812] Step 1:

[0813] The device is placed near the baby and uses its built-in microphone and camera to capture audio and video data. This includes the baby's cries, facial expressions, and body movements. The data is converted into a digital format and prepared for analysis.

[0814] Step 2:

[0815] The device transmits the acquired audio and video data to a cloud server. This communication uses a secure protocol to minimize the risk of data leakage. The transmitted data is processed in real time, taking into account any temporal delays.

[0816] Step 3:

[0817] When the server receives audio data, it uses a speech recognition system to analyze the crying patterns. From the identified audio features, it infers the baby's emotional state (e.g., hunger, sleepiness). A pre-trained emotion model is used for this analysis.

[0818] Step 4:

[0819] The server processes the video data and uses computer vision algorithms to analyze the baby's facial expressions and movements. For example, it can determine whether the baby wants to play or is sleepy based on their facial expressions and limb movements.

[0820] Step 5:

[0821] The server recognizes the user's emotions based on voice input and behavioral logs, in addition to the baby's emotional state. The emotion engine identifies stress and feelings of security from the user's voice and sends the resulting data to the emotion analysis system.

[0822] Step 6:

[0823] By combining this data, the server generates a response plan that takes into account the emotional state of both the baby and the user. For example, specific actions such as "feed the baby milk" or "recommend playing relaxation music" are determined.

[0824] Step 7:

[0825] The device sends a notification to the user based on the countermeasures received from the server. The notification presents an action plan tailored to the baby's condition and the user's emotions.

[0826] Step 8:

[0827] The device connects with smart devices and automatically adjusts the environment as needed. For example, it optimizes the childcare environment by lowering the brightness of smart lights or playing music.

[0828] Through this series of processes, the system provides childcare support that meets the needs of both the baby and the user.

[0829] (Example 2)

[0830] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0831] Conventional childcare support systems have struggled to provide comprehensive solutions that consider not only the baby's emotions and needs, but also the parents' emotions and stress levels. Furthermore, providing optimal solutions in real time, based on the conditions of both the baby and the parents, has been a technical challenge.

[0832] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0833] In this invention, the server includes means for receiving audio and video data from an acquisition device, means for analyzing the audio and video data to estimate the emotional states of the baby and the caregiver, and means for determining appropriate countermeasures that take into account the emotional states of the baby and the caregiver based on the estimation results. This enables comprehensive childcare support that simultaneously satisfies the needs of the baby and the emotions of the caregiver.

[0834] An "acquisition device" is a device for receiving audio and video data.

[0835] "Analyzing audio and video data" means processing the received data to determine the subject's behavior and emotional state.

[0836] "Inferring behavioral and emotional states" means estimating the subject's current psychological or physical state based on analyzed data.

[0837] "Determining appropriate countermeasures" means using the prediction results to select the optimal actions and instructions for the target and users.

[0838] "Controlling smart devices" means operating or managing connected electronic devices based on predetermined countermeasures.

[0839] A "generative AI model" refers to an algorithm that learns from data and accurately predicts the state of a situation.

[0840] "Generating optimal action plans" means creating specific action guidelines tailored to the target audience and the user's situation.

[0841] This invention is a system aimed at supporting childcare, and it provides functions that take into account not only the emotional state and behavior of the baby, but also the emotions of the caregiver who uses the system. This system is implemented as follows.

[0842] About the configuration and equipment

[0843] The device is equipped with a camera and microphone to acquire audio and video data about the baby. The data acquired from these sensors includes crying, ambient sounds, movements, and facial expressions. This collected data is transmitted to a server via the internet.

[0844] About data analysis

[0845] The server analyzes data in real time using an analysis engine built on the cloud. This analysis uses a generative AI model to infer the baby's emotional state and behavior from audio and video. The analyzed information is processed by an emotion analysis algorithm within the system to specifically identify the baby's needs. Similarly, the user's emotional state is also analyzed from the voice input.

[0846] Proposal of countermeasures

[0847] The server takes into account the emotional states of both the baby and the user to determine the appropriate course of action. For example, if the server detects both the baby's hunger and the user's fatigue simultaneously, it will prompt the user to prepare milk and send instructions to the device recommending relaxing music.

[0848] Specific examples and prompt statements

[0849] For example, if a baby cries, and the server determines that the baby is hungry, and at the same time the user's calm tone indicates low stress levels, the server will notify the user, "Please enjoy some soothing music while you prepare your baby's milk."

[0850] Example prompt: "The baby is crying continuously. It may need milk. Gently notify the user and instruct them to play relaxing music."

[0851] In this way, the system aims to provide environmentally conscious childcare support and create a comfortable environment for both the baby and the user.

[0852] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0853] Step 1:

[0854] The device uses a camera and microphone to collect audio and video data of the baby's surroundings. Inputs include ambient sounds, the baby's movements, and cries, which are captured as digital data. The collected data is temporarily stored within the device and prepared for analysis.

[0855] Step 2:

[0856] The terminal encrypts the acquired audio and video data and transmits it to the server via the internet. The input is the raw data acquired in step 1, which is delivered to the server while maintaining security during transmission. The output is an encrypted data flow.

[0857] Step 3:

[0858] The server inputs the received audio and video data into its analysis engine, where a generative AI model performs real-time analysis. The input consists of digital audio and video, which the AI ​​model processes and outputs the baby's emotional state (e.g., hunger, sleepiness, desire to play, etc.). This output is data indicating the baby's specific emotional state.

[0859] Step 4:

[0860] The device acquires the parent's voice and analyzes it using an emotion engine. The input is the parent's words and tone of voice, which is then subjected to emotion analysis to output the user's emotional state (e.g., stress, fatigue, feeling of security). This output is data that indicates the user's feelings.

[0861] Step 5:

[0862] The server integrates and analyzes the emotional states of both the baby and the user to determine the optimal course of action. The input is the emotional data obtained in steps 3 and 4, and an AI model is used to design responses that address the baby's needs while supporting the user. The output is a set of specific instructions and recommendations.

[0863] Step 6:

[0864] The server sends the decided course of action to the terminal. The input is the course of action generated on the server side, which is then transmitted to the terminal as a procedure manual. The output is a digital message containing instructions to be followed.

[0865] Step 7:

[0866] The terminal controls connected smart devices to implement the received countermeasures. The input is a command from the server, which executes specific actions such as playing music or adjusting lighting. The output is the optimized state of the environment settings.

[0867] (Application Example 2)

[0868] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0869] There is a need for a system that can detect abnormal situations and safety concerns within the home based on a comprehensive assessment that takes into account the emotional state of each individual and the overall home environment, and to respond appropriately. Conventional technologies only focus on individual emotions and behaviors, making it difficult to comprehensively grasp the safety situation within the home and provide appropriate countermeasures in real time.

[0870] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0871] In this invention, the server includes means for receiving audio and video data from an acquisition device, means for analyzing the audio and video data and inferring the subject's behavior or emotional state, and means for evaluating the safety status within the home based on the audio characteristics of the subject and neighbors. This makes it possible to ensure safety and provide prompt countermeasures tailored to the situation and individual emotions within the home.

[0872] An "acquisition device" is a device used to receive audio and video data, and is responsible for acquiring data within a home or specific environment.

[0873] "Audio data" refers to audio or sound information represented in digital format, and is used for analysis and evaluation.

[0874] "Video data" refers to visual information represented in digital format, and is used for analysis and evaluation.

[0875] A "subject" is an individual observed through audio and video data, whose emotional state and behavior are inferred from it.

[0876] "Inference" is the act of analyzing acquired data to determine the emotional state and behavior of a subject.

[0877] "Vocal features" are specific elements extracted from voice data and serve as indicators of an individual's emotional state and environmental characteristics.

[0878] A "means for evaluating the safety status within the home" is a system that analyzes the overall safety of the environment based on acquired data and provides an appropriate assessment.

[0879] In the system implementing this invention, a server and a terminal play important roles. First, the terminal collects audio and video data through an acquisition device installed in the home. The acquired data is then transmitted to the server. The server uses audio analysis software (e.g., speech recognition API) to analyze the audio data and image analysis software (e.g., image recognition API) to analyze the video data to infer the behavior and emotional state of the subject.

[0880] The server assesses the safety status of the home based on the inferred emotional state of the subject, while also considering voice characteristics. For example, an AI-powered analysis platform learns voices and sounds within the home to detect anomalies. Next, a generative AI model is used to determine appropriate countermeasures based on the safety status assessment. This makes it possible to present appropriate countermeasures in real time. For example, if someone in the family is experiencing stress, it will suggest relaxing content or activities.

[0881] These countermeasures are communicated to the user via the device, and automatically controlled devices are appropriately managed. For example, actions such as adjusting lighting or playing music can be performed. This gives users a sense of security within their homes and enables proactive safety management.

[0882] As a concrete example, when a baby starts crying, the device collects the audio data, and the server analyzes it. If it determines that the baby is hungry, the server notifies the user to prepare milk, and the system then starts playing relaxing music. This entire process is achieved by utilizing IoT devices and cloud computing technology.

[0883] Example of a prompt:

[0884] "To ensure safety within the home, analyze audio and video in real time and suggest appropriate responses. Explain how to control smart devices, taking into account the emotions of both the baby and the caregiver."

[0885] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0886] Step 1:

[0887] The terminal collects audio and video data using acquisition equipment installed in the home. This input data consists of audio and video captures from within the home and is necessary for subsequent analysis. The collected data is transmitted to the server in real time.

[0888] Step 2:

[0889] The server analyzes the received audio data using speech analysis software. This extracts speech features and generates basic information for determining the subject's emotions and the situation within the home. The output of this speech analysis includes feature quantities, such as crying patterns and voice tone.

[0890] Step 3:

[0891] The server analyzes the video data using image analysis software. It analyzes the subject's actions in detail from the video data and saves the results to the server's database. The analysis results output indicators that show abnormalities or warning signs of behavior.

[0892] Step 4:

[0893] The server integrates features derived from audio and video data and uses a generative AI model to infer the subject's emotional state. In this step, prompt sentences are used to enable the AI ​​model to infer emotion, providing the basis for appropriate responses.

[0894] Step 5:

[0895] The server assesses the safety status and determines appropriate countermeasures based on inferred emotional states and voice characteristics within the home. Using a generative AI model, it generates instructions to quickly provide responses in the event of an anomaly.

[0896] Step 6:

[0897] The device notifies the user of the countermeasures output from the server and executes control of the smart device based on those countermeasures. This includes specific actions such as playing music or adjusting lighting. The aim is to give the user a sense of security within their home.

[0898] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0899] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0900] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0901] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0902] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0903] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0904] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0905] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0906] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0907] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0908] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0909] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0910] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0911] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0912] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0913] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0914] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0915] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0916] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0917] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0918] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0919] The following is further disclosed regarding the embodiments described above.

[0920] (Claim 1)

[0921] Means for receiving audio data and video data from an acquisition device,

[0922] A means for analyzing the aforementioned audio and video data to infer the subject's behavior or emotional state,

[0923] A means of determining and proposing appropriate countermeasures based on the aforementioned inferences,

[0924] A means for predicting the future state of the subject based on past data,

[0925] Means for controlling a smart device that automatically executes the aforementioned appropriate countermeasures,

[0926] A system that includes this.

[0927] (Claim 2)

[0928] The system according to claim 1, further comprising means for improving accuracy in the state estimation and prediction of the subject using a machine learning algorithm.

[0929] (Claim 3)

[0930] The system according to claim 1, further comprising means for generating and providing a manual describing the optimal way for relatives to interact with the subject.

[0931] "Example 1"

[0932] (Claim 1)

[0933] Means for receiving acoustic and image information from an acquisition device,

[0934] A means for analyzing the aforementioned acoustic and image information and inferring the subject's actions or emotional state,

[0935] Means for determining and indicating appropriate measures based on the aforementioned inference,

[0936] A means for predicting the future state of the aforementioned target based on past information of the target,

[0937] Means for controlling an intelligent device that automatically performs the aforementioned appropriate measures,

[0938] A means of transmitting information in real time and minimizing communication delays,

[0939] A means of using a generative AI model that generates treatments tailored to the individual characteristics of the target,

[0940] A system that includes this.

[0941] (Claim 2)

[0942] The system according to claim 1, further comprising means for improving accuracy using a machine learning algorithm in the state estimation and prediction of the aforementioned target, and including means for enhancing real-time processing capability.

[0943] (Claim 3)

[0944] The system according to claim 1, further comprising means for describing the optimal way for relatives to interact with the subject, and for generating and providing a manual that is individually customized by a generative AI model.

[0945] "Application Example 1"

[0946] (Claim 1)

[0947] Means for receiving audio data and video data from an acquisition device,

[0948] A means for analyzing the aforementioned audio and video data to infer the behavior or emotional state of the object,

[0949] A means of determining and proposing appropriate countermeasures based on the aforementioned inferences,

[0950] A means for predicting the future state of the aforementioned object based on past data,

[0951] Means for controlling an intelligent device that automatically executes the aforementioned appropriate countermeasures,

[0952] A means to detect specific disorder in a product and propose countermeasures,

[0953] A system that includes this.

[0954] (Claim 2)

[0955] The system according to claim 1, further comprising means for improving accuracy in inferring and predicting the state of the object using a machine learning algorithm.

[0956] (Claim 3)

[0957] The system according to claim 1, further comprising means for generating and providing instructions describing the optimal method for an administrator to interact with the object.

[0958] "Example 2 of combining an emotion engine"

[0959] (Claim 1)

[0960] Means for receiving audio data and video data from an acquisition device,

[0961] A means for analyzing the aforementioned audio and video data and inferring the subject's behavior and emotional state,

[0962] A means of determining and presenting appropriate countermeasures, taking into consideration both the emotional state of the subject and the user,

[0963] A means for recognizing the emotional state using the voice input of the user,

[0964] A means for controlling a smart device in accordance with the countermeasures based on the above speculation,

[0965] A system that includes this.

[0966] (Claim 2)

[0967] The system according to claim 1, further comprising means for improving accuracy using a generative AI model in predicting the state of the target and the user.

[0968] (Claim 3)

[0969] The system according to claim 1, further comprising means for generating and providing optimal behavioral guidance based on the emotional state of the subject and the user.

[0970] "Application example 2 when combining with an emotional engine"

[0971] (Claim 1)

[0972] Means for receiving audio data and video data from an acquisition device,

[0973] A means for analyzing the aforementioned audio and video data and inferring the subject's behavior or emotional state,

[0974] A means of determining and proposing appropriate countermeasures based on the aforementioned inferences,

[0975] A means for evaluating the safety status within the home based on the voice characteristics of the subject and neighbors,

[0976] A means for predicting the future state of the subject based on past data,

[0977] Means for controlling equipment that automatically executes the aforementioned appropriate countermeasures,

[0978] A system that includes this.

[0979] (Claim 2)

[0980] The system according to claim 1, further comprising means for improving accuracy in the state estimation and prediction of the subject using a machine learning algorithm.

[0981] (Claim 3)

[0982] The system according to claim 1, further comprising means for detecting anomalies related to household safety and suggesting appropriate responses according to the urgency. [Explanation of Symbols]

[0983] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. Means for receiving audio data and video data from an acquisition device, A means for analyzing the aforementioned audio and video data to infer the subject's behavior or emotional state, A means of determining and proposing appropriate countermeasures based on the aforementioned inferences, A means for predicting the future state of the subject based on past data, Means for controlling a smart device that automatically executes the aforementioned appropriate countermeasures, A system that includes this.

2. The system according to claim 1, further comprising means for improving accuracy in the state estimation and prediction of the subject using a machine learning algorithm.

3. The system according to claim 1, further comprising means for generating and providing a manual describing the optimal method for relatives to interact with the subject.

Images