System

Abstract

A system is provided.SOLUTION: A system including camera means for acquiring image data of a pet, microphone means for acquiring sound data of the pet, sensor means for acquiring biological data of the pet, emotion analysis means for analyzing the acquired image data, sound data, and biological data and estimating an emotion of the pet, data transmission means for transmitting the acquired data to a server, data analysis means for analyzing an emotion and a health condition of the pet on the server, abnormality detection means for detecting an abnormality of the pet based on an analysis result and generating an alert, and notification means for notifying a user of the generated alert.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Pets are cherished as family members, but it is difficult for owners to accurately grasp their pets' emotions and health status. In particular, pets often do not show signs of illness or injury, which typically delays early detection. As a result, appropriate treatment is delayed, increasing the risk of the pet's health deteriorating. Furthermore, inability to communicate appropriately with pets can increase their stress and anxiety. Therefore, there is a need for a system that can accurately grasp a pet's emotions and health status and immediately notify owners when an abnormality occurs. [Means for solving the problem]

[0005] The present invention provides a system that comprehensively monitors a pet's emotions and health status and promotes communication between owners and their pets. Specifically, the system includes a camera for acquiring image data of the pet, a microphone for acquiring audio data of the pet, and a sensor for acquiring biometric data of the pet. The system also includes an emotion analysis unit that analyzes the acquired data and estimates the pet's emotions, a data transmission unit that transmits the data to a server, a data analysis unit that performs analysis on the server, an anomaly detection unit that detects abnormalities based on the analysis results and generates an alert, and a notification unit that notifies the user of the generated alert. Furthermore, the data analysis unit continuously monitors the pet's health status and provides real-time audio notifications to the user. This reduces the burden on owners in terms of both pet health and communication, promoting the longevity and happiness of their pets.

[0006] Below are definitions of important words.

[0007] The "camera means" is a device for acquiring image data of a pet.

[0008] The "microphone means" is a device for acquiring voice data of a pet.

[0009] "Sensor means" refers to a device for obtaining a pet's body temperature, heart rate, and other vital data.

[0010] The "emotion analysis means" is a device or software that analyzes the acquired image data, audio data, and biometric data to estimate the pet's emotions.

[0011] "Data transmission means" refers to a device or software for transmitting acquired data to a server via the Internet or other communication means.

[0012] The "data analysis means" is a device or software that analyzes the data received on the server and estimates the pet's emotions and health condition.

[0013] The "abnormality detection means" is a device or software that detects abnormalities in the pet based on the analyzed results and generates an alert.

[0014] The "notification means" is a device or software for notifying the user of the generated alert or information.

[0015] An "alert" is a notification that warns the owner when an abnormality is detected in the pet's health or emotional state.

[0016] A "server" is a computer system for analyzing, storing, and managing data.

[0017] "User" refers to owners and other interested parties who use the system to monitor their pets' emotions and health. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8]FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0019] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0022] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0023] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0024] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0026] [First embodiment]

[0027] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0028] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0031] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0034] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0038] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0039] MODE FOR CARRYING OUT THE INVENTION

[0040] This invention provides a system for comprehensively monitoring pets' emotions and health conditions, allowing owners to quickly and accurately grasp their pets' conditions and take appropriate action. This system is realized by the following components and their operations:

[0041] Components

[0042] 1. Terminal: This terminal is equipped with a camera means for acquiring image data of the pet, a microphone means for acquiring audio data, and a sensor means for acquiring biometric data.

[0043] 2. Server: Has a data analysis means for receiving and analyzing the acquired data, and an anomaly detection means for detecting anomalies and generating alerts.

[0044] 3. User: The owner or pet caregiver will use the system to monitor the pet's condition and receive notifications.

[0045] Program processing

[0046] Below, we will explain in natural language how each element of this system works together.

[0047] Data collection

[0048] The device activates cameras, microphones, and sensors to continuously collect data to monitor your pet's condition in real time.

[0049] Example: The device's camera captures video of your pet, the microphone records its sounds, and sensors measure its temperature and heart rate.

[0050] Data transmission

[0051] The terminal transmits the collected data to a server via a network.

[0052] Example: A device sends a pet's heart rate data to a server every 30 seconds.

[0053] Data analysis

[0054] The server converts the received data into an analyzable format and uses a generative AI model to estimate the pet's emotional state.

[0055] Example: The server analyzes the eye movements of a pet from video data and infers that the pet is excited.

[0056] Anomaly detection

[0057] The server monitors the pet's health based on the analysis results, and immediately generates an alert if an abnormality is detected.

[0058] Example: If the server analyzes body temperature data and detects a temperature outside the normal range, it generates an alert saying, "Your pet may have a fever."

[0059] notification

[0060] The server sends the generated alerts and the results of emotion analysis to the user's device.

[0061] The terminal notifies the user visually or audibly.

[0062] Example: The device displays the message "Pet has a high temperature. Please take him to the vet."

[0063] Interface and voice assistant integration

[0064] The terminal provides a user interface that displays the results of a pet's emotional analysis and health status data in graphs and text.

[0065] The device works in conjunction with a voice assistant to provide real-time notifications to the user via voice.

[0066] Example: A voice assistant informs the user that "your pet is hungry."

[0067] summary

[0068] This system allows owners to closely monitor their pets' emotions and health. By analyzing the collected data and providing appropriate alerts and notifications based on the results, owners can detect and respond to abnormalities in their pets early, making it a useful tool for promoting the longevity and happiness of pets.

[0069] The processing flow will be explained below.

[0070] Step 1:

[0071] The device enters data collection mode upon user operation, activating the camera, microphone, and sensors to continuously capture video, audio, and biometric data of the pet.

[0072] Step 2:

[0073] The device saves the captured video data at regular intervals (e.g., 30 frames per second), records audio data (e.g., saves audio data every 3 seconds), and periodically records biometric data such as body temperature and heart rate (e.g., measures every 30 seconds).

[0074] Step 3:

[0075] The terminal transmits the collected data to a server via the Internet. The transmission is performed in real time as the data is collected, but in the event of a network failure, the data is temporarily stored in the terminal.

[0076] Step 4:

[0077] The server converts the received data into an analyzable format, extracting characteristic frames from the video data and analyzing the volume and tone of the bird's calls from the audio data. The biometric data is then organized into an easily readable format.

[0078] Step 5:

[0079] The server uses a generative AI model to analyze the pet's emotions from the pre-processed data. For example, it can infer that the pet is excited based on the presence of a wagging tail in the video data or a high-pitched cry in the audio data.

[0080] Step 6:

[0081] The server monitors the pet's health based on the results of data analysis. It continuously analyzes data such as food intake, water intake, feces, body temperature, and heart rate, and generates an abnormality alert when values ​​outside the normal health range are detected.

[0082] Step 7:

[0083] The server then sends the generated alerts and the results of the emotion analysis to the owner's device, which includes specific instructions such as "The pet has a high temperature. Please take it to the vet."

[0084] Step 8:

[0085] The device notifies the user of alerts received from the server and the results of sentiment analysis. Notifications are displayed visually and, if necessary, audio notifications.

[0086] Step 9:

[0087] The user can check the information provided and take appropriate action depending on the pet's condition. For example, if they receive an alert about a fever, they can immediately take their pet to a veterinarian.

[0088] Step 10:

[0089] The device displays the pet's emotional analysis and health data in graphs and text, and allows users to check past data, allowing them to manage their pet's health over the long term.

[0090] Step 11:

[0091] The device will provide real-time notifications through the voice assistant, for example, letting users know when their pet is hungry, allowing them to take prompt action.

[0092] Example 1

[0093] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0094] It is difficult for pet owners to quickly and accurately grasp their pets' emotional and health states. Furthermore, there is a lack of effective systems for early detection of abnormalities in pets and appropriate response. Therefore, in order to maintain the happiness and health of pets, there is a need for a system that can comprehensively monitor the emotions and health states of pets and immediately notify of abnormalities.

[0095] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0096] In this invention, the server includes a data analysis means that standardizes collected data and analyzes it using a generative AI model, an anomaly detection means that analyzes the pet's emotions and health condition on the server and detects abnormalities, and a notification means that generates an alert and notifies the user when an abnormality is detected. This makes it possible to analyze various pet data and realize anomaly detection and notification in real time.

[0097] The "camera means" is a device for acquiring image data of a pet.

[0098] The "microphone means" is a device for acquiring voice data of a pet.

[0099] The "sensor means" is a device for acquiring biological data of a pet.

[0100] The "emotion analysis means" is an algorithm or software for analyzing the acquired image data, audio data, and biometric data to estimate the pet's emotions.

[0101] "Data analysis means" means algorithms or software for normalizing collected data and analyzing it using generative AI models.

[0102] The "data transmission means" is a communication module or software for transmitting acquired data to a server.

[0103] The "abnormality detection means" is an algorithm or software that analyzes the pet's emotions and health condition on the server and detects abnormalities in the pet based on the analysis results.

[0104] The "notification means" is a device or software for notifying the user of the generated alerts and analysis results.

[0105] A "user interface" is a display device or software for visually displaying generated alerts and analysis results.

[0106] "Audio notification means" means a device or software that uses a voice assistant to provide real-time notifications to a user by voice.

[0107] MODE FOR CARRYING OUT THE INVENTION

[0108] This invention provides a system for comprehensively monitoring pets' emotions and health conditions, enabling owners to quickly and accurately grasp their pets' conditions and take appropriate action. This system is realized through cooperation between terminals, a server, and users.

[0109] Components

[0110] 1. Device:

[0111] The terminal is equipped with a camera means for acquiring image data of the pet, a microphone means for acquiring audio data, and a sensor means for acquiring biological data.

[0112] Specifically, the camera captures real-time footage of your pet, the microphone records barks and environmental sounds, and the sensors measure your pet's temperature and heart rate.

[0113] 2. Server:

[0114] The server has a data analysis means for receiving data sent from the terminal and converting it into an analyzable format.

[0115] This data analysis method uses a generative AI model to estimate the emotional state of your pet.

[0116] In addition, it also has an abnormality detection means that detects abnormalities in pets based on the analysis results and generates an alert.

[0117] 3. User:

[0118] Users (owners or pet caregivers) monitor the condition of their pets through this system.

[0119] The health and emotional state of the pet can be understood through a means of displaying alerts and analysis results generated from the server on a user interface.

[0120] You can also use the voice assistant to receive real-time notifications via voice.

[0121] Data collection:

[0122] The device monitors your pet's condition in real time and continuously collects data, including video, audio, body temperature, and heart rate.

[0123] Example: A device camera takes a video of your pet every minute, a microphone records its sounds, and sensors measure your pet's temperature and heart rate every minute.

[0124] Data transmission:

[0125] The terminal compresses the collected data and transmits it to the server via the network.

[0126] Example: A device sends compressed heart rate data and video data to a server every five minutes.

[0127] Data Analysis:

[0128] The server normalizes the received data and analyzes it using a generative AI model to estimate the pet's emotional state and monitor its health.

[0129] Example: Analyzing a pet's facial expressions from video data to estimate its state of excitement or relaxation.

[0130] Anomaly detection:

[0131] The server constantly monitors the pet's health based on the analysis results, and immediately generates an alert if an abnormality is detected.

[0132] Example: If the body temperature exceeds the normal range, generate an alert saying "Suspected Fever."

[0133] notification:

[0134] The server sends the generated alerts and analysis results to the user's device.

[0135] The terminal displays alerts and analysis results on the user interface and notifies the user visually or audibly.

[0136] Example: The device displays a message on the screen saying, "Your pet has a high temperature. Please take him to the vet." The voice assistant also informs the user that "Your pet's temperature is above normal."

[0137] Examples of prompts:

[0138] "Tell me your pet's latest emotional state"

[0139] Check your pet's temperature

[0140] "I want to know if my pet is stressed."

[0141] This system allows owners to closely monitor their pets' emotions and health. By analyzing the collected data and providing appropriate alerts and notifications based on the results, owners can detect abnormalities in their pets early and respond quickly, making it a useful tool for promoting the longevity and happiness of pets.

[0142] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0143] Step 1:

[0144] Data collection:

[0145] The device activates cameras, microphones, and sensors to collect data to monitor your pet's condition in real time.

[0146] Input: Pet video, audio, temperature, heart rate

[0147] Output: Collected image data, audio data, and biometric data

[0148] How it works: The device's camera captures real-time video of your pet, the microphone records your pet's cries and environmental sounds, and the sensors measure your pet's temperature and heart rate every minute.

[0149] Step 2:

[0150] Data transmission:

[0151] The terminal transmits the collected data to a server via a network.

[0152] Input: Collected image data, audio data, and biometric data

[0153] Output: Data sent to the server

[0154] How it works: At a predefined interval (e.g., every 5 minutes), the device compresses and transmits data, including heart rate and temperature measurements, as well as video data, to a server using the secure HTTP protocol.

[0155] Step 3:

[0156] Data Analysis:

[0157] The server converts the received data into an analyzable format and uses a generative AI model to infer the emotional state.

[0158] Input: Data sent to the server (image data, audio data, biometric data)

[0159] Output: Estimated pet's emotional state

[0160] How it works: The server first standardizes the format of the received data, then divides the video data into frames, analyzes the audio data, and inputs it into a generative AI model. The server then analyzes the pet's facial expressions and vocal characteristics to estimate its emotional state.

[0161] Step 4:

[0162] Anomaly detection:

[0163] The server monitors the pet's health based on the analysis results and generates an alert if an abnormality is detected.

[0164] Input: Pet emotional state estimation results, biometric data

[0165] Output: Alert message

[0166] Specific operation: The server compares the analyzed data with normal health data, and if an abnormality is detected, it generates an alert such as "body temperature is outside the normal range."

[0167] Step 5:

[0168] notification:

[0169] The server sends the generated alerts and the results of emotion analysis to the user's device, and the device notifies the user visually or audibly.

[0170] Input: Alert message, sentiment analysis results

[0171] Output: User notification

[0172] What happens: The server immediately sends the generated alert to the user's device, which then warns the user via a pop-up notification or voice assistant. For example, a message might appear on the screen saying, "Your pet has a high temperature. Please take it to the vet."

[0173] Step 6:

[0174] User Interface and Voice Assistant Integration:

[0175] The device provides a user interface that displays the results of a pet's emotional analysis and health data in graphs and text, and works in conjunction with a voice assistant to provide real-time notifications to the user via voice.

[0176] Input: Emotion analysis results, biometric data

[0177] Output: Display on the user interface, audio notification

[0178] Specific operation: The user interface displays graphs of heart rate and body temperature fluctuations over the past 24 hours, and the voice assistant notifies the user with notifications such as "Your pet's stress level is increasing."

[0179] (Application example 1)

[0180] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0181] Conventional health and emotional monitoring systems for pets and workers only collect individual data, which means they lack comprehensive analysis of their condition. Furthermore, they lack real-time alert notifications when abnormalities occur, making it difficult to respond quickly. Therefore, there is a need for a system that can comprehensively monitor the health and emotional state of pets and workers in real time and respond quickly when abnormalities are detected.

[0182] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0183] In this invention, the server includes a data analysis means, an emotion analysis means, and an anomaly detection means, which makes it possible to comprehensively analyze the health and emotional states of pets and workers, and to quickly generate and notify an alert when an anomaly is detected.

[0184] The "data analysis means" is a device or program that analyzes the acquired image data, audio data, and biometric data on the server to analyze the emotions and health status of the pet or worker.

[0185] The "emotion analysis means" is a device or program for estimating the emotions of a pet or worker based on acquired image data, audio data, and biometric data.

[0186] The "abnormality detection means" is a device or program for detecting abnormalities in a pet or worker based on the analysis results obtained by the data analysis means and the emotion analysis means, and generating an alert.

[0187] The "notification means" is a device or program for notifying the user of the generated alert.

[0188] "Camera means" is a device for acquiring image data of a pet or working environment.

[0189] The "microphone means" is a device for acquiring sound data of a pet or the working environment.

[0190] The "sensor means" is a device for acquiring biometric data of a pet or worker.

[0191] The "data transmission means" is a device or program for transmitting the acquired data to the server.

[0192] This invention is a system that monitors the emotions and health of pets by comprehensively acquiring and analyzing images, sounds, and biometric data of pets or their working environment. This allows for rapid response when an abnormality is detected. The following explains how each element of this system works together.

[0193] First, the terminal is equipped with a camera, microphone, and sensor means, and these devices are used to collect image data, audio data, and biometric data of pets or workers in real time. As a specific example, the terminal's camera captures images of pets, the microphone records sounds and the workplace, and the sensors measure body temperature and heart rate.

[0194] Next, the collected data is transmitted to the server via the data transmission means. For example, the terminal transmits the heart rate data of the pet or the worker to the server every 30 seconds.

[0195] On the server, the data analysis means converts the received data into an analyzable format and uses a generative AI model to estimate emotions and health conditions. For example, the server analyzes the eye movements of pets and workers from video data and estimates that they are "excited."

[0196] Furthermore, the server detects anomalies based on the analysis results. The anomaly detection means generates an alert if it detects data that exceeds the normal range. For example, if the server analyzes body temperature data and detects a temperature that exceeds the normal range, it generates an alert saying, "There is a possibility of fever."

[0197] The generated alert is sent to the user's device via a notification means. The device notifies the user visually or audibly. For example, the device may display a message on the screen saying, "The pet's temperature is high. Please take it to a veterinarian."

[0198] The device also provides a user interface that displays emotion analysis results and health status data in graphs and text. It also works with a voice assistant to provide real-time notifications to the user. For example, the voice assistant might notify the user, "Your pet is hungry."

[0199] Examples of prompts for a generative AI model include: "Try to determine if a robot working in a factory is experiencing stress" and "Analyze current health scan data to see if there are any abnormalities."

[0200] This system allows for detailed monitoring of the emotions and health of pets and workers, enabling early detection and response to abnormalities. Specific hardware includes cameras (e.g., Logitech C920) and sensors, and software includes OpenCV and TensorFlow.

[0201] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0202] Step 1:

[0203] The device uses a camera to capture image data of the pet or working environment. Specifically, a camera (e.g., Logitech C920) captures images in real time and stores the image data in its internal memory. The input is the real-time image, and the output is the stored image data.

[0204] Step 2:

[0205] The terminal uses a microphone means to capture audio data of the pet or working environment. The microphone records the audio in real time and stores the audio data in its internal memory. The input is the real-time audio, and the output is the stored audio data.

[0206] Step 3:

[0207] The terminal acquires the biometric data of the pet or worker using a sensor means. The sensor measures the body temperature and heart rate and stores the data in the internal memory. The input is the real-time biometric data, and the output is the stored biometric data.

[0208] Step 4:

[0209] The terminal transmits the collected image data, voice data, and biometric data to the server using a data transmission means. Specifically, each piece of data is compressed and packaged at regular intervals and sent to the server via the Internet. The input is the collected data, and the output is the transmitted data.

[0210] Step 5:

[0211] The server uses data analysis means to convert the received data into an analyzable format. Specifically, image data is converted into grayscale using OpenCV, and audio data is converted into a spectrogram using an audio analysis program. Biometric data is normalized. The input is the transmitted data, and the output is the data converted into an analyzable format.

[0212] Step 6:

[0213] The server performs emotion analysis using a generative AI model. Specifically, the converted data is input into a TensorFlow model to estimate the emotional state of the pet or worker. The main output is the estimated emotional state.

[0214] Step 7:

[0215] The server uses anomaly detection means to detect anomalies based on the analysis results. Specifically, it compares the estimated emotional and health states with predetermined standards and generates an alert if an abnormality is detected. The input is the analysis results, and the output is an alert.

[0216] Step 8:

[0217] The server sends the generated alert to the user's terminal via a notification means. Specifically, the alert content is displayed as a text message on the user's terminal screen. The input is the generated alert, and the output is a notification to the user's terminal.

[0218] Step 9:

[0219] The notification means of the device notifies the user of the generated alert visually or audibly. Specifically, the device displays an alert message on the screen and the voice assistant plays a corresponding voice message. The input is the sent alert, and the output is the notification to the user.

[0220] The above steps realize comprehensive monitoring of the emotions and health status of pets and the working environment.

[0221] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0222] MODE FOR CARRYING OUT THE INVENTION

[0223] This invention provides a system that comprehensively monitors the emotions and health status of pets and also recognizes the emotions of the owner (user) to optimize the response. This allows the owner to quickly and accurately grasp the condition of their pet and take the optimal response. This system is realized by the following components and their operations:

[0224] Components

[0225] 1. Terminal: This terminal is equipped with a camera means for acquiring image data of the pet, a microphone means for acquiring voice data, and a sensor means for acquiring biometric data. It also includes a camera means for acquiring image data of the user and a microphone means for acquiring voice data.

[0226] 2. Server: Equipped with a data analysis means for receiving and analyzing acquired data, an emotion analysis means for recognizing and estimating the emotions of pets and users, and an anomaly detection means for detecting abnormalities and generating alerts.

[0227] 3. User: Owners and pet caregivers use the system to monitor and be notified of their pets' and their own emotional states.

[0228] Program processing

[0229] Below, we will explain in natural language how each element of this system works together.

[0230] Data collection

[0231] The device enters data collection mode upon user operation, activating the camera, microphone, and sensors to collect images, audio, and biometric data of the pet, as well as the user's image and audio data.

[0232] For example: The device's camera simultaneously captures images of both the pet and the user, the microphone records audio from both, and sensors measure the pet's temperature and heart rate.

[0233] Data transmission

[0234] The terminals transmit the collected data to a server via the network. The transmission is done in real time, but in the event of a network failure, the data is temporarily stored in the terminal.

[0235] Example: The device sends audio and video data of the pet and the user, as well as the pet's heart rate data, to the server every 30 seconds.

[0236] Data analysis

[0237] The server converts the received data into an analyzable format, extracting facial and movement features from the video data and analyzing the tone and strength of the voice from the audio data. The biometric data is then organized into an easily readable format.

[0238] Example: The server analyzes facial expressions from video data of a pet and simultaneously estimates emotions from video data of the user.

[0239] Emotion analysis

[0240] The server analyzes the emotions of the pet and the user from the preprocessed data using a generative AI model. Based on the movement of the pet's tail and ears, and the tone of its cries, it infers that the pet is happy, and based on the user's facial expression and tone of voice, it infers that the owner is at ease.

[0241] Example: Estimating the emotions of a pet wagging its tail and a user smiling and talking in video data.

[0242] Anomaly detection

[0243] The server monitors the pet's health based on the analysis of emotional and biological data, and generates alerts for abnormalities taking into account the emotional state of the pet and the user.

[0244] Example: If the user looks worried about a pet with an abnormally high body temperature, an alert will be generated saying, "Take your pet to the vet immediately."

[0245] notification

[0246] The server sends the generated alerts and the results of emotion analysis to the user's device.

[0247] The terminal notifies the user visually or audibly.

[0248] Example: The device displays the message "Your pet has a high temperature. Please take immediate action" on the screen and simultaneously announces "Your pet is anxious" through voice.

[0249] Interface and voice assistant integration

[0250] The terminal provides a user interface that displays the emotional analysis results and health data of the pet and the user in graphs and text.

[0251] The device will provide real-time notifications through the voice assistant, for example, letting users know when their pet is hungry, allowing them to take prompt action.

[0252] Example: A voice assistant might inform the user that "your pet is hungry" and simultaneously advise that "your smiling face will make your pet feel at ease."

[0253] summary

[0254] This system allows for detailed monitoring of the emotions and health of both owners and their pets. By analyzing the collected data and issuing appropriate alerts and notifications based on the results, it is possible to detect and respond to abnormalities in pets and owners early on. This makes it a useful tool for both pets and owners to live happy and healthy lives.

[0255] The processing flow will be explained below.

[0256] Step 1:

[0257] The terminal starts a data collection mode by user operation. Here, the camera means for acquiring image data of the pet, the microphone means for acquiring audio data, and the sensor means for acquiring biometric data are activated. At the same time, the camera means for acquiring image data of the user and the microphone means for acquiring audio data are also activated.

[0258] Step 2:

[0259] The device collects and stores the collected video data of the pet and the user in real time. For example, the pet's video is recorded at 30 frames per second, and the user's video is recorded at the same time. Audio data is saved every three seconds. The sensor also measures the pet's temperature and heart rate every 30 seconds.

[0260] Step 3:

[0261] The device sends the collected data to a server via the Internet. The data is sent in real time, but if there are network problems, it is temporarily stored in the device.

[0262] Step 4:

[0263] The server converts the received data into an analyzable format, extracting specific frames from the video data and analyzing the tone and strength of the voice from the audio data. Biometric data is standardized and converted into an easily analyzable format.

[0264] Step 5:

[0265] The server uses a generative AI model to analyze the emotions of the pet and the user from pre-processed data, for example, by using the pet's tail movements and the tone of its cries, as well as the user's facial expressions and tone of voice.

[0266] Step 6:

[0267] The server comprehensively evaluates the emotional states of the pet and the user based on the results of emotion analysis. For example, if the pet is anxious and the user is also showing signs of impatience or anxiety, the server determines that special attention is required.

[0268] Step 7:

[0269] The server monitors your pet's health in real time, continuously analyzing food intake, water intake, waste data, temperature data, and heart rate data to identify any data points that are outside of normal ranges.

[0270] Step 8:

[0271] The server generates an alert when an abnormality is detected, and determines the specific content of the alert based on the pet's biological and emotional data, as well as the user's emotional data.

[0272] Step 9:

[0273] The server then sends the generated alert and the emotion analysis results to the user's device, including specific instructions such as "Your pet has a high temperature. Please take it to the vet immediately."

[0274] Step 10:

[0275] The device notifies the user of alerts and emotion analysis results received from the server either visually or audibly. If the notification is visual, it will be displayed on the screen, and if it is audible, it will be conveyed to the user through the voice assistant.

[0276] Step 11:

[0277] The user can check the notified information and take appropriate action depending on the pet's condition. For example, if they receive a fever alert, they can immediately take their pet to a veterinarian.

[0278] Step 12:

[0279] The device displays the pet and user's emotional analysis results and health data in graphs and text, allowing users to check past data and manage their pet's long-term health.

[0280] Step 13:

[0281] The device will provide real-time notifications through the voice assistant, for example, telling the user, "Your pet is hungry," and offering advice such as, "Your smile will make your pet feel at ease."

[0282] Example 2

[0283] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0284] Today, there is a need for a system that comprehensively monitors the health and emotional state of pets and their owners. However, existing systems only handle pet health and emotional data and are unable to perform comprehensive analysis that includes the owner's emotional state. Furthermore, abnormality detection and real-time notification tend to be delayed, making prediction and early response difficult. Therefore, there is a need for a systematic system that can more precisely monitor the health and emotional state of pets and their owners and respond quickly and appropriately when abnormalities occur.

[0285] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0286] In this invention, the server includes means for converting received data into an analyzable format and extracting facial and movement features, means for analyzing the tone and strength of voice from the audio data, and means for analyzing and estimating the emotions of the pet and the user from the preprocessed data using a generative AI model. This enables comprehensive analysis of the emotional and health states of the pet and the owner, and if an abnormality is detected, it is possible to quickly and accurately generate an alert and notify the user in real time.

[0287] A "terminal" is a device that starts a data collection mode by user operation, collects images, sounds, and biometric data of the pet and the user, and transmits the collected data to a server via the Internet.

[0288] "Camera means" refers to an optical sensor device for capturing image data of the pet and the user.

[0289] The "microphone means" is an acoustic sensor device for acquiring voice data of the pet and the user.

[0290] The "sensor means" is a biometric device for measuring the pet's biological data, specifically, body temperature, heart rate, etc.

[0291] The "data transmission means" is a communication function for transmitting data collected by the terminal to a server via a network.

[0292] "Data analysis means" refers to software and algorithms that convert data received on the server into an analyzable format, extract facial and movement features, and analyze voice tone and strength from audio data.

[0293] "Emotion Analysis Means" means software and algorithms for analyzing and inferring pet and user emotions from data preprocessed using a generative AI model.

[0294] The "abnormality detection means" is software and a system for monitoring the health of the pet and the user based on the analyzed emotional data and biometric data, and generating an alert if an abnormality is detected.

[0295] The "notification means" is a function for visually or audibly notifying the user of the generated alert or the results of emotion analysis.

[0296] The "user interface" is a display and operating environment that displays analysis results in text and graph format, and works in conjunction with a voice assistant to provide real-time notifications.

[0297] A "generative AI model" is an artificial intelligence model used to analyze pet and user emotions from preprocessed data, and includes, for example, GPT-3, image recognition models, and speech recognition models.

[0298] This invention is a system that comprehensively monitors the emotions and health status of pets, and also recognizes the emotions of the owner (user) to optimize the response. This allows the user to quickly and accurately grasp the condition of their pet and take the optimal response. This system is realized by the following components and their operations.

[0299] Components

[0300] 1. Terminal: This terminal is equipped with a camera means for acquiring image data of the pet, a microphone means for acquiring audio data, and a sensor means for acquiring biometric data. It also includes a camera means for acquiring image data of the user and a microphone means for acquiring audio data.

[0301] 2. Server: The server has a data analysis means for receiving and analyzing the acquired data, an emotion analysis means for recognizing and estimating the emotions of the pet and the user, and an anomaly detection means for detecting anomalies and generating alerts.

[0302] 3. User: Owners and pet caregivers use the system to monitor and be notified of their pets' and their own emotional states.

[0303] Program processing

[0304] Data collection

[0305] The device enters data collection mode when the user operates it, activating the camera, microphone, and sensors to collect images, audio, and biometric data of the pet, as well as the user's image and audio data.

[0306] Example: The device's camera captures video of the pet and the user at 30 frames per second, the microphone records audio from both parties at a sampling rate of 44.1 kHz, and the sensor measures the pet's temperature once per second and its heart rate in real time.

[0307] Data transmission

[0308] The device sends the collected data to a server via the Internet. The data is sent in real time, but if a network failure occurs, the data is temporarily stored in the device's internal storage and will be sent again once the network is restored.

[0309] Example: The terminal sends data to the server through the REST API using the HTTP protocol. For example, it sends a JSON formatted data packet every 30 seconds. The data sent also includes a timestamp to ensure data integrity.

[0310] Data analysis

[0311] The server converts the received data into an analyzable format, extracting facial and movement features from the video data and analyzing the tone and strength of the voice from the audio data. Biometric data is also converted into easy-to-read graphs and numbers.

[0312] Example: The server uses Python's OpenCV library to detect facial landmarks from video data. For audio data, the Librosa library is used to extract audio features and analyze the frequency spectrum. Biometric data is organized into a data frame using the Pandas library and visualized as time-series data.

[0313] Emotion analysis

[0314] The server uses a generative AI model to analyze the emotions of the pet and the user from preprocessed data. It analyzes the movements of the pet's tail and ears, the tone of its cries, and other factors to determine whether the pet is happy, and the user's facial expression and tone of voice to determine whether the owner is at ease.

[0315] Example: Using the generative AI model GPT-3, emotions are expressed in text format based on data obtained from image recognition models (e.g., ResNet) and speech recognition models (e.g., DeepSpeech). For example, the most appropriate emotion for a given situation can be estimated based on the width and frequency of a pet's tail wagging or the degree of a user's smile.

[0316] Anomaly detection

[0317] The server monitors the pet's health based on the analysis of its emotional and biological data, and generates an alert if an abnormality is detected. The alert is generated taking into account the pet's emotional state.

[0318] Example: The server sets a threshold based on the analysis results and detects abnormalities. For example, if the pet's body temperature exceeds 39 degrees or the heart rate exceeds 200 bpm, an alert will be generated stating "possible high fever." It will also alert the user if the emotional state of the pet and the user do not match.

[0319] notification

[0320] The server sends the generated alerts and the results of emotion analysis to the user's device.

[0321] The terminal notifies the user visually or audibly.

[0322] Example: The server uses WebSocket or Push Notification services to send notifications to the device in real time. The device displays a message on the screen saying "Your pet's temperature is high. Please take action immediately" and uses a Text-to-Speech (TTS) engine to notify the user by voice, "Your pet is feeling anxious."

[0323] Interface and voice assistant integration

[0324] The device provides a user interface that displays the pet and user's emotional analysis and health data in graphs and text, and also provides real-time notifications via a voice assistant.

[0325] Example: The device uses front-end frameworks such as React and Vue.js to visualize the analysis results in the browser. It also works with a voice assistant to notify users in real time that their pet is hungry, while simultaneously providing advice such as, "Your smile will make your pet feel at ease."

[0326] Prompt Sentence Examples

[0327] Below are some examples of prompt sentences to input to the generative AI model.

[0328] Prompt Sentence Examples

[0329] Analyze the health and emotions of your pet, as well as your emotions. The following data will be collected: pet video, audio, and biometric data, and user video and audio data.

[0330] This system allows users to closely monitor their pets' emotions and health. By analyzing the collected data and providing appropriate alerts and notifications based on the results, it is possible to detect and respond to abnormalities in both pets and users early on. This makes it a useful tool for helping both pets and users live happy and healthy lives.

[0331] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0332] Step 1: Data collection

[0333] The device starts data collection mode when the user operates it. The camera (image data), microphone (audio data), and sensor (biometric data) are activated to collect images, audio, and biometric data of the pet. At the same time, the user's image and audio data are also collected.

[0334] Input: The user operates the terminal to start data collection mode.

[0335] Output: A set of video data, audio data, and biometric data collected by the device.

[0336] How it works: The device's camera captures video of the pet and the user at 30 frames per second, the microphone records audio from both parties at 44.1kHz, and the sensor measures the pet's temperature every second and heart rate in real time.

[0337] Step 2: Send data

[0338] The terminal transmits the collected data to a server via the Internet in real time, but in the event of a network failure, the data is temporarily stored in the terminal.

[0339] Input: A set of collected image, audio, and biometric data.

[0340] Output: A set of image data, audio data, and biometric data sent to the server.

[0341] Specific operation: The terminal uses the HTTP protocol to send data to the server via the REST API every 30 seconds. The data is in JSON format and each data item is time-stamped.

[0342] Step 3: Data analysis

[0343] The server converts the received data into an analyzable format, extracting facial and movement features from the video data and analyzing the tone and strength of the voice from the audio data. The biometric data is then organized into graphs and numerical values.

[0344] Input: A set of image data, audio data, and biometric data sent to the server.

[0345] Output: Analyzed facial and movement characteristics, voice tone and strength, and organized biometric data.

[0346] Specific operation: The server uses Python's OpenCV library to detect facial landmarks from video data, and the Librosa library to extract audio features. The biometric data is organized into a data frame using the Pandas library and visualized.

[0347] Step 4: Sentiment Analysis

[0348] The server uses a generative AI model to analyze the emotions of the pet and the user from the pre-processed data.

[0349] Input: Analyzed facial and movement characteristics, voice tone and volume, and organized biometric data.

[0350] Output: Estimated pet and user emotions.

[0351] Specific operation: The server uses image recognition models (e.g., ResNet) and speech recognition models (e.g., DeepSpeech) as generative AI models to express emotions from data in text format. For example, by linking the behavior of a pet wagging its tail with the user's smile, it can infer that "the pet is happy" and "the owner is at ease."

[0352] Step 5: Anomaly detection

[0353] The server monitors the pet's health based on the analysis of emotional and biometric data, and generates an alert if an abnormality is detected.

[0354] Input: Estimated pet and user emotions, and organised biometric data.

[0355] Output: The generated alert.

[0356] Specific operation: The server sets a threshold and generates an alert saying "possible high fever" if, for example, the pet's body temperature exceeds 39 degrees or the heart rate exceeds 200 bpm. If an anxious expression on the user's face is also detected at the same time, the server issues a warning.

[0357] Step 6: Notification

[0358] The server sends the generated alerts and the results of emotion analysis to the user's device.

[0359] The terminal notifies the user visually or audibly.

[0360] Input: Generated alerts, sentiment analysis results.

[0361] Output: A visual or audio notification.

[0362] Specific operation: The server sends a real-time notification to the device using WebSocket or Push Notification services. The device displays the message "Your pet's temperature is high. Please take action immediately" on the screen and uses a Text-to-Speech (TTS) engine to notify the user by voice, "Your pet is anxious."

[0363] Step 7: Interface and Voice Assistant Integration

[0364] The terminal provides a user interface that displays the emotional analysis results and health data of the pet and the user in graphs and text.

[0365] The device will provide real-time notifications via the voice assistant.

[0366] Input: Sentiment analysis results, health data, generated alerts.

[0367] Output: Data displayed in graphs and text, notifications via voice assistant.

[0368] How it works: The device visualizes the analysis results in a browser using front-end frameworks such as React and Vue.js. It also works with a voice assistant to notify users in real time that their pet is hungry, while simultaneously providing advice such as, "Your smile will make your pet feel at ease."

[0369] (Application example 2)

[0370] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0371] Conventional systems exist that comprehensively monitor a pet's emotions and health status, and also recognize the owner's emotions to optimize responses. However, a system that utilizes such a system in an autonomous vehicle to monitor the status of the pet and owner in real time and automatically optimize the in-vehicle environment has not yet been realized. This has made it difficult for owners to grasp the pet's condition in detail and respond appropriately while on the move. The present invention aims to solve this problem.

[0372] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a data analysis means for analyzing acquired data, an emotion analysis means for estimating the pet's emotion and health state, an abnormality detection means for detecting an abnormality based on the analysis result and generating an alarm, and an environment adjustment means for optimizing the in-vehicle environment. This makes it possible to monitor the status of the pet and its owner in real time inside the autonomously driving vehicle and automatically adjust the in-vehicle environment based on the analysis result.

[0373] The "photography means" is a device for acquiring image data of a pet.

[0374] The "recording means" is a device for acquiring voice data of a pet.

[0375] "Measuring means" refers to a device for acquiring biological data of a pet.

[0376] The "emotion analysis means" is a device that analyzes the acquired image data, audio data, and biometric data to estimate the pet's emotions.

[0377] The "data transmission means" is a device for transmitting the acquired data to the server.

[0378] The "data analysis means" is a device for analyzing the emotions and health condition of pets on the server.

[0379] The "abnormality detection means" is a device that detects abnormalities in the pet based on the analysis results and generates an alarm.

[0380] The "notification means" is a device for notifying the user of the generated alert.

[0381] The "environment adjustment means" is a device that monitors the condition of pets and their owners inside an autonomous vehicle and optimizes the in-vehicle environment based on the analysis results.

[0382] This invention relates to a system that monitors the status of pets and their owners in an autonomous vehicle and optimizes the in-vehicle environment. This system is realized by a number of components and their cooperation.

[0383] Components

[0384] 1. Photography method: Image data of the pet is obtained using a camera installed in the vehicle.

[0385] Specifically, a general webcam (e.g., Logitech C920) is used.

[0386] 2. Recording method: Record your pet's voice using a microphone in the vehicle.

[0387] Specifically, a high-performance microphone (e.g., Blue Yeti) is used.

[0388] 3. Measurement methods: Use sensors to obtain your pet's biological data.

[0389] Specifically, biosensors for pets (e.g., Fitbit pet sensors) are used.

[0390] 4. Emotion analysis method: Analyze the acquired data to estimate the pet's emotions.

[0391] Specifically, video and audio data are analyzed using an AI model (e.g., a generative AI model using Keras).

[0392] 5. Data transmission means: Transmits the acquired data to the server.

[0393] In particular, a communication device is utilized to transmit data over a network connection.

[0394] 6. Data analysis means: The data received on the server is analyzed to determine the pet's emotions and health condition.

[0395] Specifically, data analysis software (e.g., Python scripts) that runs on the server side is used.

[0396] 7. Anomaly detection means: Detects anomalies based on the analysis results and generates an alert.

[0397] The server generates an alert if it detects an abnormality in the health status.

[0398] 8. Notification means: Notifies the user of the generated alert.

[0399] Specifically, audio notifications and screen displays are used.

[0400] 9. Environmental control means: A device for optimizing the in-vehicle environment.

[0401] This includes devices that automatically control temperature, lighting, music, etc. inside the vehicle.

[0402] Explaining program processing in natural language

[0403] The server first collects data through photography, audio recording, and measurement. It then transmits this data in real time to the server. The server processes the data using a generative AI model using Keras to analyze the emotions and health of the pet and owner. Based on the analysis results, it then generates an alert if necessary and notifies the user. At the same time as the notification, the vehicle's environmental control measures are used to optimize the temperature, lighting, music, and other aspects of the vehicle interior.

[0404] Specific examples

[0405] For example, a camera inside the car captures video of the pet and its owner, and a microphone records audio. Measurements are made to measure the pet's heart rate and body temperature. This data is sent to a server in real time, and a generative AI model using Keras analyzes the data. If the server detects that the pet is in an unstable state, it generates an alarm and notifies the user with an audio message saying, "Your pet is in an unstable state," while simultaneously adjusting the in-car environment to make it more comfortable for the pet.

[0406] Prompt Sentence Examples

[0407] "Measure the pet's temperature and analyze the video data captured by the camera. Also, analyze the owner's voice data and infer the emotions of both parties. Based on the analysis results, play appropriate music."

[0408] In this way, an environment can be provided in which both pets and owners can move around with peace of mind.

[0409] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0410] Step 1:

[0411] The terminal activates the photographing means, recording means, and measuring means to collect data on the pet and its owner. Image data acquired by the photographing means, audio data acquired by the recording means, and biometric data acquired by the measuring means (e.g., the pet's body temperature and heart rate) are collected in real time. Input data includes image data, audio data, and biometric data.

[0412] Step 2:

[0413] The terminal temporarily stores the collected data and then transmits it to the server via the network. At this time, transmission may be temporarily delayed depending on the network status, but this data transmission method ensures stable transmission. The input is the image data, voice data, and biometric data collected in step 1, and the output is that these data are transmitted to the server.

[0414] Step 3:

[0415] The server converts the received image data into a processable format. Specifically, this involves extracting facial and movement features of the pet and owner from the image data. This process uses an image processing library such as OpenCV. The input is the transmitted image data, and the output is the data with the extracted features.

[0416] Step 4:

[0417] The server analyzes the received voice data and analyzes the tone and stress of the voice. This process uses a voice analysis library such as Librosa. The input is the transmitted voice data, and the output is the analyzed voice feature data.

[0418] Step 5:

[0419] The server organizes and analyzes the received biometric data, including converting the pet's heart rate and temperature data into an easily readable format. The input is the transmitted biometric data, and the output is the analyzed biometric data.

[0420] Step 6:

[0421] The server uses emotion analysis means to analyze the emotions of the pet and owner from the preprocessed image data, audio data, and biometric data based on the generative AI model. For example, it analyzes the pet's tail movements and the tone of its cries, as well as the owner's facial expressions and tone of voice. The input is the feature-extracted data and analyzed audio and biometric data, and the output is an estimation of the emotional state of the pet and owner.

[0422] Step 7:

[0423] The server monitors the pet's health using anomaly detection means based on the analysis results. If an abnormality is detected, an emergency alert is generated. For example, if the pet's body temperature is high and the user looks anxious, an alert is generated. The input is the emotion analysis result and biometric data, and the output is the alert generation.

[0424] Step 8:

[0425] The device notifies the user of the generated alert and the results of the emotion analysis by audio or visual notification. For example, the device may display "Your pet's temperature is high. Please take action immediately" on the screen and simultaneously notify the user by audio notification, "Your pet is feeling anxious." The input is the alert and the results of the emotion analysis, and the output is the notification to the user.

[0426] Step 9:

[0427] The device adjusts the in-car environment in real time, optimizing the temperature, lighting, music, etc. based on the analysis results. For example, if the pet is relaxed and the owner feels at ease, relaxing music will be played and the lighting will be softened. The input is the results of emotion analysis and health status, and the output is the adjustment of the in-car environment.

[0428] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0429] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0430] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0431] [Second embodiment]

[0432] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0433] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0434] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0435] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0436] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0437] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0438] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0439] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0440] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0441] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0442] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0443] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0444] MODE FOR CARRYING OUT THE INVENTION

[0445] This invention provides a system for comprehensively monitoring pets' emotions and health conditions, allowing owners to quickly and accurately grasp their pets' conditions and take appropriate action. This system is realized by the following components and their operations:

[0446] Components

[0447] 1. Terminal: This terminal is equipped with a camera means for acquiring image data of the pet, a microphone means for acquiring audio data, and a sensor means for acquiring biometric data.

[0448] 2. Server: Has a data analysis means for receiving and analyzing the acquired data, and an anomaly detection means for detecting anomalies and generating alerts.

[0449] 3. User: The owner or pet caregiver will use the system to monitor the pet's condition and receive notifications.

[0450] Program processing

[0451] Below, we will explain in natural language how each element of this system works together.

[0452] Data collection

[0453] The device activates cameras, microphones, and sensors to continuously collect data to monitor your pet's condition in real time.

[0454] Example: The device's camera captures video of your pet, the microphone records its sounds, and sensors measure its temperature and heart rate.

[0455] Data transmission

[0456] The terminal transmits the collected data to a server via a network.

[0457] Example: A device sends a pet's heart rate data to a server every 30 seconds.

[0458] Data analysis

[0459] The server converts the received data into an analyzable format and uses a generative AI model to estimate the pet's emotional state.

[0460] Example: The server analyzes the eye movements of a pet from video data and infers that the pet is excited.

[0461] Anomaly detection

[0462] The server monitors the pet's health based on the analysis results, and immediately generates an alert if an abnormality is detected.

[0463] Example: If the server analyzes body temperature data and detects a temperature outside the normal range, it generates an alert saying, "Your pet may have a fever."

[0464] notification

[0465] The server sends the generated alerts and the results of emotion analysis to the user's device.

[0466] The terminal notifies the user visually or audibly.

[0467] Example: The device displays the message "Pet has a high temperature. Please take him to the vet."

[0468] Interface and voice assistant integration

[0469] The terminal provides a user interface that displays the results of a pet's emotional analysis and health status data in graphs and text.

[0470] The device works in conjunction with a voice assistant to provide real-time notifications to the user via voice.

[0471] Example: A voice assistant informs the user that "your pet is hungry."

[0472] summary

[0473] This system allows owners to closely monitor their pets' emotions and health. By analyzing the collected data and providing appropriate alerts and notifications based on the results, owners can detect and respond to abnormalities in their pets early, making it a useful tool for promoting the longevity and happiness of pets.

[0474] The processing flow will be explained below.

[0475] Step 1:

[0476] The device enters data collection mode upon user operation, activating the camera, microphone, and sensors to continuously capture video, audio, and biometric data of the pet.

[0477] Step 2:

[0478] The device saves the captured video data at regular intervals (e.g., 30 frames per second), records audio data (e.g., saves audio data every 3 seconds), and periodically records biometric data such as body temperature and heart rate (e.g., measures every 30 seconds).

[0479] Step 3:

[0480] The terminal transmits the collected data to a server via the Internet. The transmission is performed in real time as the data is collected, but in the event of a network failure, the data is temporarily stored in the terminal.

[0481] Step 4:

[0482] The server converts the received data into an analyzable format, extracting characteristic frames from the video data and analyzing the volume and tone of the bird's calls from the audio data. The biometric data is then organized into an easily readable format.

[0483] Step 5:

[0484] The server uses a generative AI model to analyze the pet's emotions from the pre-processed data. For example, it can infer that the pet is excited based on the presence of a wagging tail in the video data or a high-pitched cry in the audio data.

[0485] Step 6:

[0486] The server monitors the pet's health based on the results of data analysis. It continuously analyzes data such as food intake, water intake, feces, body temperature, and heart rate, and generates an abnormality alert when values ​​outside the normal health range are detected.

[0487] Step 7:

[0488] The server then sends the generated alerts and the results of the emotion analysis to the owner's device, which includes specific instructions such as "The pet has a high temperature. Please take it to the vet."

[0489] Step 8:

[0490] The device notifies the user of alerts received from the server and the results of sentiment analysis. Notifications are displayed visually and, if necessary, audio notifications.

[0491] Step 9:

[0492] The user can check the information provided and take appropriate action depending on the pet's condition. For example, if they receive an alert about a fever, they can immediately take their pet to a veterinarian.

[0493] Step 10:

[0494] The device displays the pet's emotional analysis and health data in graphs and text, and allows users to check past data, allowing them to manage their pet's health over the long term.

[0495] Step 11:

[0496] The device will provide real-time notifications through the voice assistant, for example, letting users know when their pet is hungry, allowing them to take prompt action.

[0497] Example 1

[0498] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0499] It is difficult for pet owners to quickly and accurately grasp their pets' emotional and health states. Furthermore, there is a lack of effective systems for early detection of abnormalities in pets and appropriate response. Therefore, in order to maintain the happiness and health of pets, there is a need for a system that can comprehensively monitor the emotions and health states of pets and immediately notify of abnormalities.

[0500] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0501] In this invention, the server includes a data analysis means that standardizes collected data and analyzes it using a generative AI model, an anomaly detection means that analyzes the pet's emotions and health condition on the server and detects abnormalities, and a notification means that generates an alert and notifies the user when an abnormality is detected. This makes it possible to analyze various pet data and realize anomaly detection and notification in real time.

[0502] The "camera means" is a device for acquiring image data of a pet.

[0503] The "microphone means" is a device for acquiring voice data of a pet.

[0504] The "sensor means" is a device for acquiring biological data of a pet.

[0505] The "emotion analysis means" is an algorithm or software for analyzing the acquired image data, audio data, and biometric data to estimate the pet's emotions.

[0506] "Data analysis means" means algorithms or software for normalizing collected data and analyzing it using generative AI models.

[0507] The "data transmission means" is a communication module or software for transmitting acquired data to a server.

[0508] The "abnormality detection means" is an algorithm or software that analyzes the pet's emotions and health condition on the server and detects abnormalities in the pet based on the analysis results.

[0509] The "notification means" is a device or software for notifying the user of the generated alerts and analysis results.

[0510] A "user interface" is a display device or software for visually displaying generated alerts and analysis results.

[0511] "Audio notification means" means a device or software that uses a voice assistant to provide real-time notifications to a user by voice.

[0512] MODE FOR CARRYING OUT THE INVENTION

[0513] This invention provides a system for comprehensively monitoring pets' emotions and health conditions, enabling owners to quickly and accurately grasp their pets' conditions and take appropriate action. This system is realized through cooperation between terminals, a server, and users.

[0514] Components

[0515] 1. Device:

[0516] The terminal is equipped with a camera means for acquiring image data of the pet, a microphone means for acquiring audio data, and a sensor means for acquiring biological data.

[0517] Specifically, the camera captures real-time footage of your pet, the microphone records barks and environmental sounds, and the sensors measure your pet's temperature and heart rate.

[0518] 2. Server:

[0519] The server has a data analysis means for receiving data sent from the terminal and converting it into an analyzable format.

[0520] This data analysis method uses a generative AI model to estimate the emotional state of your pet.

[0521] In addition, it also has an abnormality detection means that detects abnormalities in pets based on the analysis results and generates an alert.

[0522] 3. User:

[0523] Users (owners or pet caregivers) monitor the condition of their pets through this system.

[0524] The health and emotional state of the pet can be understood through a means of displaying alerts and analysis results generated from the server on a user interface.

[0525] You can also use the voice assistant to receive real-time notifications via voice.

[0526] Data collection:

[0527] The device monitors your pet's condition in real time and continuously collects data, including video, audio, body temperature, and heart rate.

[0528] Example: A device camera takes a video of your pet every minute, a microphone records its sounds, and sensors measure your pet's temperature and heart rate every minute.

[0529] Data transmission:

[0530] The terminal compresses the collected data and transmits it to the server via the network.

[0531] Example: A device sends compressed heart rate data and video data to a server every five minutes.

[0532] Data Analysis:

[0533] The server normalizes the received data and analyzes it using a generative AI model to estimate the pet's emotional state and monitor its health.

[0534] Example: Analyzing a pet's facial expressions from video data to estimate its state of excitement or relaxation.

[0535] Anomaly detection:

[0536] The server constantly monitors the pet's health based on the analysis results, and immediately generates an alert if an abnormality is detected.

[0537] Example: If the body temperature exceeds the normal range, generate an alert saying "Suspected Fever."

[0538] notification:

[0539] The server sends the generated alerts and analysis results to the user's device.

[0540] The terminal displays alerts and analysis results on the user interface and notifies the user visually or audibly.

[0541] Example: The device displays a message on the screen saying, "Your pet has a high temperature. Please take him to the vet." The voice assistant also informs the user that "Your pet's temperature is above normal."

[0542] Examples of prompts:

[0543] "Tell me your pet's latest emotional state"

[0544] Check your pet's temperature

[0545] "I want to know if my pet is stressed."

[0546] This system allows owners to closely monitor their pets' emotions and health. By analyzing the collected data and providing appropriate alerts and notifications based on the results, owners can detect abnormalities in their pets early and respond quickly, making it a useful tool for promoting the longevity and happiness of pets.

[0547] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0548] Step 1:

[0549] Data collection:

[0550] The device activates cameras, microphones, and sensors to collect data to monitor your pet's condition in real time.

[0551] Input: Pet video, audio, temperature, heart rate

[0552] Output: Collected image data, audio data, and biometric data

[0553] How it works: The device's camera captures real-time video of your pet, the microphone records your pet's cries and environmental sounds, and the sensors measure your pet's temperature and heart rate every minute.

[0554] Step 2:

[0555] Data transmission:

[0556] The terminal transmits the collected data to a server via a network.

[0557] Input: Collected image data, audio data, and biometric data

[0558] Output: Data sent to the server

[0559] How it works: At a predefined interval (e.g., every 5 minutes), the device compresses and transmits data, including heart rate and temperature measurements, as well as video data, to a server using the secure HTTP protocol.

[0560] Step 3:

[0561] Data Analysis:

[0562] The server converts the received data into an analyzable format and uses a generative AI model to infer the emotional state.

[0563] Input: Data sent to the server (image data, audio data, biometric data)

[0564] Output: Estimated pet's emotional state

[0565] How it works: The server first standardizes the format of the received data, then divides the video data into frames, analyzes the audio data, and inputs it into a generative AI model. The server then analyzes the pet's facial expressions and vocal characteristics to estimate its emotional state.

[0566] Step 4:

[0567] Anomaly detection:

[0568] The server monitors the pet's health based on the analysis results and generates an alert if an abnormality is detected.

[0569] Input: Pet emotional state estimation results, biometric data

[0570] Output: Alert message

[0571] Specific operation: The server compares the analyzed data with normal health data, and if an abnormality is detected, it generates an alert such as "body temperature is outside the normal range."

[0572] Step 5:

[0573] notification:

[0574] The server sends the generated alerts and the results of emotion analysis to the user's device, and the device notifies the user visually or audibly.

[0575] Input: Alert message, sentiment analysis results

[0576] Output: User notification

[0577] What happens: The server immediately sends the generated alert to the user's device, which then warns the user via a pop-up notification or voice assistant. For example, a message might appear on the screen saying, "Your pet has a high temperature. Please take it to the vet."

[0578] Step 6:

[0579] User Interface and Voice Assistant Integration:

[0580] The device provides a user interface that displays the results of a pet's emotional analysis and health data in graphs and text, and works in conjunction with a voice assistant to provide real-time notifications to the user via voice.

[0581] Input: Emotion analysis results, biometric data

[0582] Output: Display on the user interface, audio notification

[0583] Specific operation: The user interface displays graphs of heart rate and body temperature fluctuations over the past 24 hours, and the voice assistant notifies the user with notifications such as "Your pet's stress level is increasing."

[0584] (Application example 1)

[0585] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0586] Conventional health and emotional monitoring systems for pets and workers only collect individual data, which means they lack comprehensive analysis of their condition. Furthermore, they lack real-time alert notifications when abnormalities occur, making it difficult to respond quickly. Therefore, there is a need for a system that can comprehensively monitor the health and emotional state of pets and workers in real time and respond quickly when abnormalities are detected.

[0587] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0588] In this invention, the server includes a data analysis means, an emotion analysis means, and an anomaly detection means, which makes it possible to comprehensively analyze the health and emotional states of pets and workers, and to quickly generate and notify an alert when an anomaly is detected.

[0589] The "data analysis means" is a device or program that analyzes the acquired image data, audio data, and biometric data on the server to analyze the emotions and health status of the pet or worker.

[0590] The "emotion analysis means" is a device or program for estimating the emotions of a pet or worker based on acquired image data, audio data, and biometric data.

[0591] The "abnormality detection means" is a device or program for detecting abnormalities in a pet or worker based on the analysis results obtained by the data analysis means and the emotion analysis means, and generating an alert.

[0592] The "notification means" is a device or program for notifying the user of the generated alert.

[0593] "Camera means" is a device for acquiring image data of a pet or working environment.

[0594] The "microphone means" is a device for acquiring sound data of a pet or the working environment.

[0595] The "sensor means" is a device for acquiring biometric data of a pet or worker.

[0596] The "data transmission means" is a device or program for transmitting the acquired data to the server.

[0597] This invention is a system that monitors the emotions and health of pets by comprehensively acquiring and analyzing images, sounds, and biometric data of pets or their working environment. This allows for rapid response when an abnormality is detected. The following explains how each element of this system works together.

[0598] First, the terminal is equipped with a camera, microphone, and sensor means, and these devices are used to collect image data, audio data, and biometric data of pets or workers in real time. As a specific example, the terminal's camera captures images of pets, the microphone records sounds and the workplace, and the sensors measure body temperature and heart rate.

[0599] Next, the collected data is transmitted to the server via the data transmission means. For example, the terminal transmits the heart rate data of the pet or the worker to the server every 30 seconds.

[0600] On the server, the data analysis means converts the received data into an analyzable format and uses a generative AI model to estimate emotions and health conditions. For example, the server analyzes the eye movements of pets and workers from video data and estimates that they are "excited."

[0601] Furthermore, the server detects anomalies based on the analysis results. The anomaly detection means generates an alert if it detects data that exceeds the normal range. For example, if the server analyzes body temperature data and detects a temperature that exceeds the normal range, it generates an alert saying, "There is a possibility of fever."

[0602] The generated alert is sent to the user's device via a notification means. The device notifies the user visually or audibly. For example, the device may display a message on the screen saying, "The pet's temperature is high. Please take it to a veterinarian."

[0603] The device also provides a user interface that displays emotion analysis results and health status data in graphs and text. It also works with a voice assistant to provide real-time notifications to the user. For example, the voice assistant might notify the user, "Your pet is hungry."

[0604] Examples of prompts for a generative AI model include: "Try to determine if a robot working in a factory is experiencing stress" and "Analyze current health scan data to see if there are any abnormalities."

[0605] This system allows for detailed monitoring of the emotions and health of pets and workers, enabling early detection and response to abnormalities. Specific hardware includes cameras (e.g., Logitech C920) and sensors, and software includes OpenCV and TensorFlow.

[0606] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0607] Step 1:

[0608] The device uses a camera to capture image data of the pet or working environment. Specifically, a camera (e.g., Logitech C920) captures images in real time and stores the image data in its internal memory. The input is the real-time image, and the output is the stored image data.

[0609] Step 2:

[0610] The terminal uses a microphone means to capture audio data of the pet or working environment. The microphone records the audio in real time and stores the audio data in its internal memory. The input is the real-time audio, and the output is the stored audio data.

[0611] Step 3:

[0612] The terminal acquires the biometric data of the pet or worker using a sensor means. The sensor measures the body temperature and heart rate and stores the data in the internal memory. The input is the real-time biometric data, and the output is the stored biometric data.

[0613] Step 4:

[0614] The terminal transmits the collected image data, voice data, and biometric data to the server using a data transmission means. Specifically, each piece of data is compressed and packaged at regular intervals and sent to the server via the Internet. The input is the collected data, and the output is the transmitted data.

[0615] Step 5:

[0616] The server uses data analysis means to convert the received data into an analyzable format. Specifically, image data is converted into grayscale using OpenCV, and audio data is converted into a spectrogram using an audio analysis program. Biometric data is normalized. The input is the transmitted data, and the output is the data converted into an analyzable format.

[0617] Step 6:

[0618] The server performs emotion analysis using a generative AI model. Specifically, the converted data is input into a TensorFlow model to estimate the emotional state of the pet or worker. The main output is the estimated emotional state.

[0619] Step 7:

[0620] The server uses anomaly detection means to detect anomalies based on the analysis results. Specifically, it compares the estimated emotional and health states with predetermined standards and generates an alert if an abnormality is detected. The input is the analysis results, and the output is an alert.

[0621] Step 8:

[0622] The server sends the generated alert to the user's terminal via a notification means. Specifically, the alert content is displayed as a text message on the user's terminal screen. The input is the generated alert, and the output is a notification to the user's terminal.

[0623] Step 9:

[0624] The notification means of the device notifies the user of the generated alert visually or audibly. Specifically, the device displays an alert message on the screen and the voice assistant plays a corresponding voice message. The input is the sent alert, and the output is the notification to the user.

[0625] The above steps realize comprehensive monitoring of the emotions and health status of pets and the working environment.

[0626] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0627] MODE FOR CARRYING OUT THE INVENTION

[0628] This invention provides a system that comprehensively monitors the emotions and health status of pets and also recognizes the emotions of the owner (user) to optimize the response. This allows the owner to quickly and accurately grasp the condition of their pet and take the optimal response. This system is realized by the following components and their operations:

[0629] Components

[0630] 1. Terminal: This terminal is equipped with a camera means for acquiring image data of the pet, a microphone means for acquiring voice data, and a sensor means for acquiring biometric data. It also includes a camera means for acquiring image data of the user and a microphone means for acquiring voice data.

[0631] 2. Server: Equipped with a data analysis means for receiving and analyzing acquired data, an emotion analysis means for recognizing and estimating the emotions of pets and users, and an anomaly detection means for detecting abnormalities and generating alerts.

[0632] 3. User: Owners and pet caregivers use the system to monitor and be notified of their pets' and their own emotional states.

[0633] Program processing

[0634] Below, we will explain in natural language how each element of this system works together.

[0635] Data collection

[0636] The device enters data collection mode upon user operation, activating the camera, microphone, and sensors to collect images, audio, and biometric data of the pet, as well as the user's image and audio data.

[0637] For example: The device's camera simultaneously captures images of both the pet and the user, the microphone records audio from both, and sensors measure the pet's temperature and heart rate.

[0638] Data transmission

[0639] The terminals transmit the collected data to a server via the network. The transmission is done in real time, but in the event of a network failure, the data is temporarily stored in the terminal.

[0640] Example: The device sends audio and video data of the pet and the user, as well as the pet's heart rate data, to the server every 30 seconds.

[0641] Data analysis

[0642] The server converts the received data into an analyzable format, extracting facial and movement features from the video data and analyzing the tone and strength of the voice from the audio data. The biometric data is then organized into an easily readable format.

[0643] Example: The server analyzes facial expressions from video data of a pet and simultaneously estimates emotions from video data of the user.

[0644] Emotion analysis

[0645] The server analyzes the emotions of the pet and the user from the preprocessed data using a generative AI model. Based on the movement of the pet's tail and ears, and the tone of its cries, it infers that the pet is happy, and based on the user's facial expression and tone of voice, it infers that the owner is at ease.

[0646] Example: Estimating the emotions of a pet wagging its tail and a user smiling and talking in video data.

[0647] Anomaly detection

[0648] The server monitors the pet's health based on the analysis of emotional and biological data, and generates alerts for abnormalities taking into account the emotional state of the pet and the user.

[0649] Example: If the user looks worried about a pet with an abnormally high body temperature, an alert will be generated saying, "Take your pet to the vet immediately."

[0650] notification

[0651] The server sends the generated alerts and the results of emotion analysis to the user's device.

[0652] The terminal notifies the user visually or audibly.

[0653] Example: The device displays the message "Your pet has a high temperature. Please take immediate action" on the screen and simultaneously announces "Your pet is anxious" through voice.

[0654] Interface and voice assistant integration

[0655] The terminal provides a user interface that displays the emotional analysis results and health data of the pet and the user in graphs and text.

[0656] The device will provide real-time notifications through the voice assistant, for example, letting users know when their pet is hungry, allowing them to take prompt action.

[0657] Example: A voice assistant might inform the user that "your pet is hungry" and simultaneously advise that "your smiling face will make your pet feel at ease."

[0658] summary

[0659] This system allows for detailed monitoring of the emotions and health of both owners and their pets. By analyzing the collected data and issuing appropriate alerts and notifications based on the results, it is possible to detect and respond to abnormalities in pets and owners early on. This makes it a useful tool for both pets and owners to live happy and healthy lives.

[0660] The processing flow will be explained below.

[0661] Step 1:

[0662] The terminal starts a data collection mode by user operation. Here, the camera means for acquiring image data of the pet, the microphone means for acquiring audio data, and the sensor means for acquiring biometric data are activated. At the same time, the camera means for acquiring image data of the user and the microphone means for acquiring audio data are also activated.

[0663] Step 2:

[0664] The device collects and stores the collected video data of the pet and the user in real time. For example, the pet's video is recorded at 30 frames per second, and the user's video is recorded at the same time. Audio data is saved every three seconds. The sensor also measures the pet's temperature and heart rate every 30 seconds.

[0665] Step 3:

[0666] The device sends the collected data to a server via the Internet. The data is sent in real time, but if there are network problems, it is temporarily stored in the device.

[0667] Step 4:

[0668] The server converts the received data into an analyzable format, extracting specific frames from the video data and analyzing the tone and strength of the voice from the audio data. Biometric data is standardized and converted into an easily analyzable format.

[0669] Step 5:

[0670] The server uses a generative AI model to analyze the emotions of the pet and the user from pre-processed data, for example, by using the pet's tail movements and the tone of its cries, as well as the user's facial expressions and tone of voice.

[0671] Step 6:

[0672] The server comprehensively evaluates the emotional states of the pet and the user based on the results of emotion analysis. For example, if the pet is anxious and the user is also showing signs of impatience or anxiety, the server determines that special attention is required.

[0673] Step 7:

[0674] The server monitors your pet's health in real time, continuously analyzing food intake, water intake, waste data, temperature data, and heart rate data to identify any data points that are outside of normal ranges.

[0675] Step 8:

[0676] The server generates an alert when an abnormality is detected, and determines the specific content of the alert based on the pet's biological and emotional data, as well as the user's emotional data.

[0677] Step 9:

[0678] The server then sends the generated alert and the emotion analysis results to the user's device, including specific instructions such as "Your pet has a high temperature. Please take it to the vet immediately."

[0679] Step 10:

[0680] The device notifies the user of alerts and emotion analysis results received from the server either visually or audibly. If the notification is visual, it will be displayed on the screen, and if it is audible, it will be conveyed to the user through the voice assistant.

[0681] Step 11:

[0682] The user can check the notified information and take appropriate action depending on the pet's condition. For example, if they receive a fever alert, they can immediately take their pet to a veterinarian.

[0683] Step 12:

[0684] The device displays the pet and user's emotional analysis results and health data in graphs and text, allowing users to check past data and manage their pet's long-term health.

[0685] Step 13:

[0686] The device will provide real-time notifications through the voice assistant, for example, telling the user, "Your pet is hungry," and offering advice such as, "Your smile will make your pet feel at ease."

[0687] Example 2

[0688] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0689] Today, there is a need for a system that comprehensively monitors the health and emotional state of pets and their owners. However, existing systems only handle pet health and emotional data and are unable to perform comprehensive analysis that includes the owner's emotional state. Furthermore, abnormality detection and real-time notification tend to be delayed, making prediction and early response difficult. Therefore, there is a need for a systematic system that can more precisely monitor the health and emotional state of pets and their owners and respond quickly and appropriately when abnormalities occur.

[0690] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0691] In this invention, the server includes means for converting received data into an analyzable format and extracting facial and movement features, means for analyzing the tone and strength of voice from the audio data, and means for analyzing and estimating the emotions of the pet and the user from the preprocessed data using a generative AI model. This enables comprehensive analysis of the emotional and health states of the pet and the owner, and if an abnormality is detected, it is possible to quickly and accurately generate an alert and notify the user in real time.

[0692] A "terminal" is a device that starts a data collection mode by user operation, collects images, sounds, and biometric data of the pet and the user, and transmits the collected data to a server via the Internet.

[0693] "Camera means" refers to an optical sensor device for capturing image data of the pet and the user.

[0694] The "microphone means" is an acoustic sensor device for acquiring voice data of the pet and the user.

[0695] The "sensor means" is a biometric device for measuring the pet's biological data, specifically, body temperature, heart rate, etc.

[0696] The "data transmission means" is a communication function for transmitting data collected by the terminal to a server via a network.

[0697] "Data analysis means" refers to software and algorithms that convert data received on the server into an analyzable format, extract facial and movement features, and analyze voice tone and strength from audio data.

[0698] "Emotion Analysis Means" means software and algorithms for analyzing and inferring pet and user emotions from data preprocessed using a generative AI model.

[0699] The "abnormality detection means" is software and a system for monitoring the health of the pet and the user based on the analyzed emotional data and biometric data, and generating an alert if an abnormality is detected.

[0700] The "notification means" is a function for visually or audibly notifying the user of the generated alert or the results of emotion analysis.

[0701] The "user interface" is a display and operating environment that displays analysis results in text and graph format, and works in conjunction with a voice assistant to provide real-time notifications.

[0702] A "generative AI model" is an artificial intelligence model used to analyze pet and user emotions from preprocessed data, and includes, for example, GPT-3, image recognition models, and speech recognition models.

[0703] This invention is a system that comprehensively monitors the emotions and health status of pets, and also recognizes the emotions of the owner (user) to optimize the response. This allows the user to quickly and accurately grasp the condition of their pet and take the optimal response. This system is realized by the following components and their operations.

[0704] Components

[0705] 1. Terminal: This terminal is equipped with a camera means for acquiring image data of the pet, a microphone means for acquiring audio data, and a sensor means for acquiring biometric data. It also includes a camera means for acquiring image data of the user and a microphone means for acquiring audio data.

[0706] 2. Server: The server has a data analysis means for receiving and analyzing the acquired data, an emotion analysis means for recognizing and estimating the emotions of the pet and the user, and an anomaly detection means for detecting anomalies and generating alerts.

[0707] 3. User: Owners and pet caregivers use the system to monitor and be notified of their pets' and their own emotional states.

[0708] Program processing

[0709] Data collection

[0710] The device enters data collection mode when the user operates it, activating the camera, microphone, and sensors to collect images, audio, and biometric data of the pet, as well as the user's image and audio data.

[0711] Example: The device's camera captures video of the pet and the user at 30 frames per second, the microphone records audio from both parties at a sampling rate of 44.1 kHz, and the sensor measures the pet's temperature once per second and its heart rate in real time.

[0712] Data transmission

[0713] The device sends the collected data to a server via the Internet. The data is sent in real time, but if a network failure occurs, the data is temporarily stored in the device's internal storage and will be sent again once the network is restored.

[0714] Example: The terminal sends data to the server through the REST API using the HTTP protocol. For example, it sends a JSON formatted data packet every 30 seconds. The data sent also includes a timestamp to ensure data integrity.

[0715] Data analysis

[0716] The server converts the received data into an analyzable format, extracting facial and movement features from the video data and analyzing the tone and strength of the voice from the audio data. Biometric data is also converted into easy-to-read graphs and numbers.

[0717] Example: The server uses Python's OpenCV library to detect facial landmarks from video data. For audio data, the Librosa library is used to extract audio features and analyze the frequency spectrum. Biometric data is organized into a data frame using the Pandas library and visualized as time-series data.

[0718] Emotion analysis

[0719] The server uses a generative AI model to analyze the emotions of the pet and the user from preprocessed data. It analyzes the movements of the pet's tail and ears, the tone of its cries, and other factors to determine whether the pet is happy, and the user's facial expression and tone of voice to determine whether the owner is at ease.

[0720] Example: Using the generative AI model GPT-3, emotions are expressed in text format based on data obtained from image recognition models (e.g., ResNet) and speech recognition models (e.g., DeepSpeech). For example, the most appropriate emotion for a given situation can be estimated based on the width and frequency of a pet's tail wagging or the degree of a user's smile.

[0721] Anomaly detection

[0722] The server monitors the pet's health based on the analysis of its emotional and biological data, and generates an alert if an abnormality is detected. The alert is generated taking into account the pet's emotional state.

[0723] Example: The server sets a threshold based on the analysis results and detects abnormalities. For example, if the pet's body temperature exceeds 39 degrees or the heart rate exceeds 200 bpm, an alert will be generated stating "possible high fever." It will also alert the user if the emotional state of the pet and the user do not match.

[0724] notification

[0725] The server sends the generated alerts and the results of emotion analysis to the user's device.

[0726] The terminal notifies the user visually or audibly.

[0727] Example: The server uses WebSocket or Push Notification services to send notifications to the device in real time. The device displays a message on the screen saying "Your pet's temperature is high. Please take action immediately" and uses a Text-to-Speech (TTS) engine to notify the user by voice, "Your pet is feeling anxious."

[0728] Interface and voice assistant integration

[0729] The device provides a user interface that displays the pet and user's emotional analysis and health data in graphs and text, and also provides real-time notifications via a voice assistant.

[0730] Example: The device uses front-end frameworks such as React and Vue.js to visualize the analysis results in the browser. It also works with a voice assistant to notify users in real time that their pet is hungry, while simultaneously providing advice such as, "Your smile will make your pet feel at ease."

[0731] Prompt Sentence Examples

[0732] Below are some examples of prompt sentences to input to the generative AI model.

[0733] Prompt Sentence Examples

[0734] Analyze the health and emotions of your pet, as well as your emotions. The following data will be collected: pet video, audio, and biometric data, and user video and audio data.

[0735] This system allows users to closely monitor their pets' emotions and health. By analyzing the collected data and providing appropriate alerts and notifications based on the results, it is possible to detect and respond to abnormalities in both pets and users early on. This makes it a useful tool for helping both pets and users live happy and healthy lives.

[0736] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0737] Step 1: Data collection

[0738] The device starts data collection mode when the user operates it. The camera (image data), microphone (audio data), and sensor (biometric data) are activated to collect images, audio, and biometric data of the pet. At the same time, the user's image and audio data are also collected.

[0739] Input: The user operates the terminal to start data collection mode.

[0740] Output: A set of video data, audio data, and biometric data collected by the device.

[0741] How it works: The device's camera captures video of the pet and the user at 30 frames per second, the microphone records audio from both parties at 44.1kHz, and the sensor measures the pet's temperature every second and heart rate in real time.

[0742] Step 2: Send data

[0743] The terminal transmits the collected data to a server via the Internet in real time, but in the event of a network failure, the data is temporarily stored in the terminal.

[0744] Input: A set of collected image, audio, and biometric data.

[0745] Output: A set of image data, audio data, and biometric data sent to the server.

[0746] Specific operation: The terminal uses the HTTP protocol to send data to the server via the REST API every 30 seconds. The data is in JSON format and each data item is time-stamped.

[0747] Step 3: Data analysis

[0748] The server converts the received data into an analyzable format, extracting facial and movement features from the video data and analyzing the tone and strength of the voice from the audio data. The biometric data is then organized into graphs and numerical values.

[0749] Input: A set of image data, audio data, and biometric data sent to the server.

[0750] Output: Analyzed facial and movement characteristics, voice tone and strength, and organized biometric data.

[0751] Specific operation: The server uses Python's OpenCV library to detect facial landmarks from video data, and the Librosa library to extract audio features. The biometric data is organized into a data frame using the Pandas library and visualized.

[0752] Step 4: Sentiment Analysis

[0753] The server uses a generative AI model to analyze the emotions of the pet and the user from the pre-processed data.

[0754] Input: Analyzed facial and movement characteristics, voice tone and volume, and organized biometric data.

[0755] Output: Estimated pet and user emotions.

[0756] Specific operation: The server uses image recognition models (e.g., ResNet) and speech recognition models (e.g., DeepSpeech) as generative AI models to express emotions from data in text format. For example, by linking the behavior of a pet wagging its tail with the user's smile, it can infer that "the pet is happy" and "the owner is at ease."

[0757] Step 5: Anomaly detection

[0758] The server monitors the pet's health based on the analysis of emotional and biometric data, and generates an alert if an abnormality is detected.

[0759] Input: Estimated pet and user emotions, and organised biometric data.

[0760] Output: The generated alert.

[0761] Specific operation: The server sets a threshold and generates an alert saying "possible high fever" if, for example, the pet's body temperature exceeds 39 degrees or the heart rate exceeds 200 bpm. If an anxious expression on the user's face is also detected at the same time, the server issues a warning.

[0762] Step 6: Notification

[0763] The server sends the generated alerts and the results of emotion analysis to the user's device.

[0764] The terminal notifies the user visually or audibly.

[0765] Input: Generated alerts, sentiment analysis results.

[0766] Output: A visual or audio notification.

[0767] Specific operation: The server sends a real-time notification to the device using WebSocket or Push Notification services. The device displays the message "Your pet's temperature is high. Please take action immediately" on the screen and uses a Text-to-Speech (TTS) engine to notify the user by voice, "Your pet is anxious."

[0768] Step 7: Interface and Voice Assistant Integration

[0769] The terminal provides a user interface that displays the emotional analysis results and health data of the pet and the user in graphs and text.

[0770] The device will provide real-time notifications via the voice assistant.

[0771] Input: Sentiment analysis results, health data, generated alerts.

[0772] Output: Data displayed in graphs and text, notifications via voice assistant.

[0773] How it works: The device visualizes the analysis results in a browser using front-end frameworks such as React and Vue.js. It also works with a voice assistant to notify users in real time that their pet is hungry, while simultaneously providing advice such as, "Your smile will make your pet feel at ease."

[0774] (Application example 2)

[0775] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0776] Conventional systems exist that comprehensively monitor a pet's emotions and health status, and also recognize the owner's emotions to optimize responses. However, a system that utilizes such a system in an autonomous vehicle to monitor the status of the pet and owner in real time and automatically optimize the in-vehicle environment has not yet been realized. This has made it difficult for owners to grasp the pet's condition in detail and respond appropriately while on the move. The present invention aims to solve this problem.

[0777] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a data analysis means for analyzing acquired data, an emotion analysis means for estimating the pet's emotion and health state, an abnormality detection means for detecting an abnormality based on the analysis result and generating an alarm, and an environment adjustment means for optimizing the in-vehicle environment. This makes it possible to monitor the status of the pet and its owner in real time inside the autonomously driving vehicle and automatically adjust the in-vehicle environment based on the analysis result.

[0778] The "photography means" is a device for acquiring image data of a pet.

[0779] The "recording means" is a device for acquiring voice data of a pet.

[0780] "Measuring means" refers to a device for acquiring biological data of a pet.

[0781] The "emotion analysis means" is a device that analyzes the acquired image data, audio data, and biometric data to estimate the pet's emotions.

[0782] The "data transmission means" is a device for transmitting the acquired data to the server.

[0783] The "data analysis means" is a device for analyzing the emotions and health condition of pets on the server.

[0784] The "abnormality detection means" is a device that detects abnormalities in the pet based on the analysis results and generates an alarm.

[0785] The "notification means" is a device for notifying the user of the generated alert.

[0786] The "environment adjustment means" is a device that monitors the condition of pets and their owners inside an autonomous vehicle and optimizes the in-vehicle environment based on the analysis results.

[0787] This invention relates to a system that monitors the status of pets and their owners in an autonomous vehicle and optimizes the in-vehicle environment. This system is realized by a number of components and their cooperation.

[0788] Components

[0789] 1. Photography method: Image data of the pet is obtained using a camera installed in the vehicle.

[0790] Specifically, a general webcam (e.g., Logitech C920) is used.

[0791] 2. Recording method: Record your pet's voice using a microphone in the vehicle.

[0792] Specifically, a high-performance microphone (e.g., Blue Yeti) is used.

[0793] 3. Measurement methods: Use sensors to obtain your pet's biological data.

[0794] Specifically, biosensors for pets (e.g., Fitbit pet sensors) are used.

[0795] 4. Emotion analysis method: Analyze the acquired data to estimate the pet's emotions.

[0796] Specifically, video and audio data are analyzed using an AI model (e.g., a generative AI model using Keras).

[0797] 5. Data transmission means: Transmits the acquired data to the server.

[0798] In particular, a communication device is utilized to transmit data over a network connection.

[0799] 6. Data analysis means: The data received on the server is analyzed to determine the pet's emotions and health condition.

[0800] Specifically, data analysis software (e.g., Python scripts) that runs on the server side is used.

[0801] 7. Anomaly detection means: Detects anomalies based on the analysis results and generates an alert.

[0802] The server generates an alert if it detects an abnormality in the health status.

[0803] 8. Notification means: Notifies the user of the generated alert.

[0804] Specifically, audio notifications and screen displays are used.

[0805] 9. Environmental control means: A device for optimizing the in-vehicle environment.

[0806] This includes devices that automatically control temperature, lighting, music, etc. inside the vehicle.

[0807] Explaining program processing in natural language

[0808] The server first collects data through photography, audio recording, and measurement. It then transmits this data in real time to the server. The server processes the data using a generative AI model using Keras to analyze the emotions and health of the pet and owner. Based on the analysis results, it then generates an alert if necessary and notifies the user. At the same time as the notification, the vehicle's environmental control measures are used to optimize the temperature, lighting, music, and other aspects of the vehicle interior.

[0809] Specific examples

[0810] For example, a camera inside the car captures video of the pet and its owner, and a microphone records audio. Measurements are made to measure the pet's heart rate and body temperature. This data is sent to a server in real time, and a generative AI model using Keras analyzes the data. If the server detects that the pet is in an unstable state, it generates an alarm and notifies the user with an audio message saying, "Your pet is in an unstable state," while simultaneously adjusting the in-car environment to make it more comfortable for the pet.

[0811] Prompt Sentence Examples

[0812] "Measure the pet's temperature and analyze the video data captured by the camera. Also, analyze the owner's voice data and infer the emotions of both parties. Based on the analysis results, play appropriate music."

[0813] In this way, an environment can be provided in which both pets and owners can move around with peace of mind.

[0814] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0815] Step 1:

[0816] The terminal activates the photographing means, recording means, and measuring means to collect data on the pet and its owner. Image data acquired by the photographing means, audio data acquired by the recording means, and biometric data acquired by the measuring means (e.g., the pet's body temperature and heart rate) are collected in real time. Input data includes image data, audio data, and biometric data.

[0817] Step 2:

[0818] The terminal temporarily stores the collected data and then transmits it to the server via the network. At this time, transmission may be temporarily delayed depending on the network status, but this data transmission method ensures stable transmission. The input is the image data, voice data, and biometric data collected in step 1, and the output is that these data are transmitted to the server.

[0819] Step 3:

[0820] The server converts the received image data into a processable format. Specifically, this involves extracting facial and movement features of the pet and owner from the image data. This process uses an image processing library such as OpenCV. The input is the transmitted image data, and the output is the data with the extracted features.

[0821] Step 4:

[0822] The server analyzes the received voice data and analyzes the tone and stress of the voice. This process uses a voice analysis library such as Librosa. The input is the transmitted voice data, and the output is the analyzed voice feature data.

[0823] Step 5:

[0824] The server organizes and analyzes the received biometric data, including converting the pet's heart rate and temperature data into an easily readable format. The input is the transmitted biometric data, and the output is the analyzed biometric data.

[0825] Step 6:

[0826] The server uses emotion analysis means to analyze the emotions of the pet and owner from the preprocessed image data, audio data, and biometric data based on the generative AI model. For example, it analyzes the pet's tail movements and the tone of its cries, as well as the owner's facial expressions and tone of voice. The input is the feature-extracted data and analyzed audio and biometric data, and the output is an estimation of the emotional state of the pet and owner.

[0827] Step 7:

[0828] The server monitors the pet's health using anomaly detection means based on the analysis results. If an abnormality is detected, an emergency alert is generated. For example, if the pet's body temperature is high and the user looks anxious, an alert is generated. The input is the emotion analysis result and biometric data, and the output is the alert generation.

[0829] Step 8:

[0830] The device notifies the user of the generated alert and the results of the emotion analysis by audio or visual notification. For example, the device may display "Your pet's temperature is high. Please take action immediately" on the screen and simultaneously notify the user by audio notification, "Your pet is feeling anxious." The input is the alert and the results of the emotion analysis, and the output is the notification to the user.

[0831] Step 9:

[0832] The device adjusts the in-car environment in real time, optimizing the temperature, lighting, music, etc. based on the analysis results. For example, if the pet is relaxed and the owner feels at ease, relaxing music will be played and the lighting will be softened. The input is the results of emotion analysis and health status, and the output is the adjustment of the in-car environment.

[0833] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0834] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0835] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0836] [Third embodiment]

[0837] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0838] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0839] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0840] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0841] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0842] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0843] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0844] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0845] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0846] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0847] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0848] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0849] MODE FOR CARRYING OUT THE INVENTION

[0850] This invention provides a system for comprehensively monitoring pets' emotions and health conditions, allowing owners to quickly and accurately grasp their pets' conditions and take appropriate action. This system is realized by the following components and their operations:

[0851] Components

[0852] 1. Terminal: This terminal is equipped with a camera means for acquiring image data of the pet, a microphone means for acquiring audio data, and a sensor means for acquiring biometric data.

[0853] 2. Server: Has a data analysis means for receiving and analyzing the acquired data, and an anomaly detection means for detecting anomalies and generating alerts.

[0854] 3. User: The owner or pet caregiver will use the system to monitor the pet's condition and receive notifications.

[0855] Program processing

[0856] Below, we will explain in natural language how each element of this system works together.

[0857] Data collection

[0858] The device activates cameras, microphones, and sensors to continuously collect data to monitor your pet's condition in real time.

[0859] Example: The device's camera captures video of your pet, the microphone records its sounds, and sensors measure its temperature and heart rate.

[0860] Data transmission

[0861] The terminal transmits the collected data to a server via a network.

[0862] Example: A device sends a pet's heart rate data to a server every 30 seconds.

[0863] Data analysis

[0864] The server converts the received data into an analyzable format and uses a generative AI model to estimate the pet's emotional state.

[0865] Example: The server analyzes the eye movements of a pet from video data and infers that the pet is excited.

[0866] Anomaly detection

[0867] The server monitors the pet's health based on the analysis results, and immediately generates an alert if an abnormality is detected.

[0868] Example: If the server analyzes body temperature data and detects a temperature outside the normal range, it generates an alert saying, "Your pet may have a fever."

[0869] notification

[0870] The server sends the generated alerts and the results of emotion analysis to the user's device.

[0871] The terminal notifies the user visually or audibly.

[0872] Example: The device displays the message "Pet has a high temperature. Please take him to the vet."

[0873] Interface and voice assistant integration

[0874] The terminal provides a user interface that displays the results of a pet's emotional analysis and health status data in graphs and text.

[0875] The device works in conjunction with a voice assistant to provide real-time notifications to the user via voice.

[0876] Example: A voice assistant informs the user that "your pet is hungry."

[0877] summary

[0878] This system allows owners to closely monitor their pets' emotions and health. By analyzing the collected data and providing appropriate alerts and notifications based on the results, owners can detect and respond to abnormalities in their pets early, making it a useful tool for promoting the longevity and happiness of pets.

[0879] The processing flow will be explained below.

[0880] Step 1:

[0881] The device enters data collection mode upon user operation, activating the camera, microphone, and sensors to continuously capture video, audio, and biometric data of the pet.

[0882] Step 2:

[0883] The device saves the captured video data at regular intervals (e.g., 30 frames per second), records audio data (e.g., saves audio data every 3 seconds), and periodically records biometric data such as body temperature and heart rate (e.g., measures every 30 seconds).

[0884] Step 3:

[0885] The terminal transmits the collected data to a server via the Internet. The transmission is performed in real time as the data is collected, but in the event of a network failure, the data is temporarily stored in the terminal.

[0886] Step 4:

[0887] The server converts the received data into an analyzable format, extracting characteristic frames from the video data and analyzing the volume and tone of the bird's calls from the audio data. The biometric data is then organized into an easily readable format.

[0888] Step 5:

[0889] The server uses a generative AI model to analyze the pet's emotions from the pre-processed data. For example, it can infer that the pet is excited based on the presence of a wagging tail in the video data or a high-pitched cry in the audio data.

[0890] Step 6:

[0891] The server monitors the pet's health based on the results of data analysis. It continuously analyzes data such as food intake, water intake, feces, body temperature, and heart rate, and generates an abnormality alert when values ​​outside the normal health range are detected.

[0892] Step 7:

[0893] The server then sends the generated alerts and the results of the emotion analysis to the owner's device, which includes specific instructions such as "The pet has a high temperature. Please take it to the vet."

[0894] Step 8:

[0895] The device notifies the user of alerts received from the server and the results of sentiment analysis. Notifications are displayed visually and, if necessary, audio notifications.

[0896] Step 9:

[0897] The user can check the information provided and take appropriate action depending on the pet's condition. For example, if they receive an alert about a fever, they can immediately take their pet to a veterinarian.

[0898] Step 10:

[0899] The device displays the pet's emotional analysis and health data in graphs and text, and allows users to check past data, allowing them to manage their pet's health over the long term.

[0900] Step 11:

[0901] The device will provide real-time notifications through the voice assistant, for example, letting users know when their pet is hungry, allowing them to take prompt action.

[0902] Example 1

[0903] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0904] It is difficult for pet owners to quickly and accurately grasp their pets' emotional and health states. Furthermore, there is a lack of effective systems for early detection of abnormalities in pets and appropriate response. Therefore, in order to maintain the happiness and health of pets, there is a need for a system that can comprehensively monitor the emotions and health states of pets and immediately notify of abnormalities.

[0905] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0906] In this invention, the server includes a data analysis means that standardizes collected data and analyzes it using a generative AI model, an anomaly detection means that analyzes the pet's emotions and health condition on the server and detects abnormalities, and a notification means that generates an alert and notifies the user when an abnormality is detected. This makes it possible to analyze various pet data and realize anomaly detection and notification in real time.

[0907] The "camera means" is a device for acquiring image data of a pet.

[0908] The "microphone means" is a device for acquiring voice data of a pet.

[0909] The "sensor means" is a device for acquiring biological data of a pet.

[0910] The "emotion analysis means" is an algorithm or software for analyzing the acquired image data, audio data, and biometric data to estimate the pet's emotions.

[0911] "Data analysis means" means algorithms or software for normalizing collected data and analyzing it using generative AI models.

[0912] The "data transmission means" is a communication module or software for transmitting acquired data to a server.

[0913] The "abnormality detection means" is an algorithm or software that analyzes the pet's emotions and health condition on the server and detects abnormalities in the pet based on the analysis results.

[0914] The "notification means" is a device or software for notifying the user of the generated alerts and analysis results.

[0915] A "user interface" is a display device or software for visually displaying generated alerts and analysis results.

[0916] "Audio notification means" means a device or software that uses a voice assistant to provide real-time notifications to a user by voice.

[0917] MODE FOR CARRYING OUT THE INVENTION

[0918] This invention provides a system for comprehensively monitoring pets' emotions and health conditions, enabling owners to quickly and accurately grasp their pets' conditions and take appropriate action. This system is realized through cooperation between terminals, a server, and users.

[0919] Components

[0920] 1. Device:

[0921] The terminal is equipped with a camera means for acquiring image data of the pet, a microphone means for acquiring audio data, and a sensor means for acquiring biological data.

[0922] Specifically, the camera captures real-time footage of your pet, the microphone records barks and environmental sounds, and the sensors measure your pet's temperature and heart rate.

[0923] 2. Server:

[0924] The server has a data analysis means for receiving data sent from the terminal and converting it into an analyzable format.

[0925] This data analysis method uses a generative AI model to estimate the emotional state of your pet.

[0926] In addition, it also has an abnormality detection means that detects abnormalities in pets based on the analysis results and generates an alert.

[0927] 3. User:

[0928] Users (owners or pet caregivers) monitor the condition of their pets through this system.

[0929] The health and emotional state of the pet can be understood through a means of displaying alerts and analysis results generated from the server on a user interface.

[0930] You can also use the voice assistant to receive real-time notifications via voice.

[0931] Data collection:

[0932] The device monitors your pet's condition in real time and continuously collects data, including video, audio, body temperature, and heart rate.

[0933] Example: A device camera takes a video of your pet every minute, a microphone records its sounds, and sensors measure your pet's temperature and heart rate every minute.

[0934] Data transmission:

[0935] The terminal compresses the collected data and transmits it to the server via the network.

[0936] Example: A device sends compressed heart rate data and video data to a server every five minutes.

[0937] Data Analysis:

[0938] The server normalizes the received data and analyzes it using a generative AI model to estimate the pet's emotional state and monitor its health.

[0939] Example: Analyzing a pet's facial expressions from video data to estimate its state of excitement or relaxation.

[0940] Anomaly detection:

[0941] The server constantly monitors the pet's health based on the analysis results, and immediately generates an alert if an abnormality is detected.

[0942] Example: If the body temperature exceeds the normal range, generate an alert saying "Suspected Fever."

[0943] notification:

[0944] The server sends the generated alerts and analysis results to the user's device.

[0945] The terminal displays alerts and analysis results on the user interface and notifies the user visually or audibly.

[0946] Example: The device displays a message on the screen saying, "Your pet has a high temperature. Please take him to the vet." The voice assistant also informs the user that "Your pet's temperature is above normal."

[0947] Examples of prompts:

[0948] "Tell me your pet's latest emotional state"

[0949] Check your pet's temperature

[0950] "I want to know if my pet is stressed."

[0951] This system allows owners to closely monitor their pets' emotions and health. By analyzing the collected data and providing appropriate alerts and notifications based on the results, owners can detect abnormalities in their pets early and respond quickly, making it a useful tool for promoting the longevity and happiness of pets.

[0952] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0953] Step 1:

[0954] Data collection:

[0955] The device activates cameras, microphones, and sensors to collect data to monitor your pet's condition in real time.

[0956] Input: Pet video, audio, temperature, heart rate

[0957] Output: Collected image data, audio data, and biometric data

[0958] How it works: The device's camera captures real-time video of your pet, the microphone records your pet's cries and environmental sounds, and the sensors measure your pet's temperature and heart rate every minute.

[0959] Step 2:

[0960] Data transmission:

[0961] The terminal transmits the collected data to a server via a network.

[0962] Input: Collected image data, audio data, and biometric data

[0963] Output: Data sent to the server

[0964] How it works: At a predefined interval (e.g., every 5 minutes), the device compresses and transmits data, including heart rate and temperature measurements, as well as video data, to a server using the secure HTTP protocol.

[0965] Step 3:

[0966] Data Analysis:

[0967] The server converts the received data into an analyzable format and uses a generative AI model to infer the emotional state.

[0968] Input: Data sent to the server (image data, audio data, biometric data)

[0969] Output: Estimated pet's emotional state

[0970] How it works: The server first standardizes the format of the received data, then divides the video data into frames, analyzes the audio data, and inputs it into a generative AI model. The server then analyzes the pet's facial expressions and vocal characteristics to estimate its emotional state.

[0971] Step 4:

[0972] Anomaly detection:

[0973] The server monitors the pet's health based on the analysis results and generates an alert if an abnormality is detected.

[0974] Input: Pet emotional state estimation results, biometric data

[0975] Output: Alert message

[0976] Specific operation: The server compares the analyzed data with normal health data, and if an abnormality is detected, it generates an alert such as "body temperature is outside the normal range."

[0977] Step 5:

[0978] notification:

[0979] The server sends the generated alerts and the results of emotion analysis to the user's device, and the device notifies the user visually or audibly.

[0980] Input: Alert message, sentiment analysis results

[0981] Output: User notification

[0982] What happens: The server immediately sends the generated alert to the user's device, which then warns the user via a pop-up notification or voice assistant. For example, a message might appear on the screen saying, "Your pet has a high temperature. Please take it to the vet."

[0983] Step 6:

[0984] User Interface and Voice Assistant Integration:

[0985] The device provides a user interface that displays the results of a pet's emotional analysis and health data in graphs and text, and works in conjunction with a voice assistant to provide real-time notifications to the user via voice.

[0986] Input: Emotion analysis results, biometric data

[0987] Output: Display on the user interface, audio notification

[0988] Specific operation: The user interface displays graphs of heart rate and body temperature fluctuations over the past 24 hours, and the voice assistant notifies the user with notifications such as "Your pet's stress level is increasing."

[0989] (Application example 1)

[0990] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0991] Conventional health and emotional monitoring systems for pets and workers only collect individual data, which means they lack comprehensive analysis of their condition. Furthermore, they lack real-time alert notifications when abnormalities occur, making it difficult to respond quickly. Therefore, there is a need for a system that can comprehensively monitor the health and emotional state of pets and workers in real time and respond quickly when abnormalities are detected.

[0992] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0993] In this invention, the server includes a data analysis means, an emotion analysis means, and an anomaly detection means, which makes it possible to comprehensively analyze the health and emotional states of pets and workers, and to quickly generate and notify an alert when an anomaly is detected.

[0994] The "data analysis means" is a device or program that analyzes the acquired image data, audio data, and biometric data on the server to analyze the emotions and health status of the pet or worker.

[0995] The "emotion analysis means" is a device or program for estimating the emotions of a pet or worker based on acquired image data, audio data, and biometric data.

[0996] The "abnormality detection means" is a device or program for detecting abnormalities in a pet or worker based on the analysis results obtained by the data analysis means and the emotion analysis means, and generating an alert.

[0997] The "notification means" is a device or program for notifying the user of the generated alert.

[0998] "Camera means" is a device for acquiring image data of a pet or working environment.

[0999] The "microphone means" is a device for acquiring sound data of a pet or the working environment.

[1000] The "sensor means" is a device for acquiring biometric data of a pet or worker.

[1001] The "data transmission means" is a device or program for transmitting the acquired data to the server.

[1002] This invention is a system that monitors the emotions and health of pets by comprehensively acquiring and analyzing images, sounds, and biometric data of pets or their working environment. This allows for rapid response when an abnormality is detected. The following explains how each element of this system works together.

[1003] First, the terminal is equipped with a camera, microphone, and sensor means, and these devices are used to collect image data, audio data, and biometric data of pets or workers in real time. As a specific example, the terminal's camera captures images of pets, the microphone records sounds and the workplace, and the sensors measure body temperature and heart rate.

[1004] Next, the collected data is transmitted to the server via the data transmission means. For example, the terminal transmits the heart rate data of the pet or the worker to the server every 30 seconds.

[1005] On the server, the data analysis means converts the received data into an analyzable format and uses a generative AI model to estimate emotions and health conditions. For example, the server analyzes the eye movements of pets and workers from video data and estimates that they are "excited."

[1006] Furthermore, the server detects anomalies based on the analysis results. The anomaly detection means generates an alert if it detects data that exceeds the normal range. For example, if the server analyzes body temperature data and detects a temperature that exceeds the normal range, it generates an alert saying, "There is a possibility of fever."

[1007] The generated alert is sent to the user's device via a notification means. The device notifies the user visually or audibly. For example, the device may display a message on the screen saying, "The pet's temperature is high. Please take it to a veterinarian."

[1008] The device also provides a user interface that displays emotion analysis results and health status data in graphs and text. It also works with a voice assistant to provide real-time notifications to the user. For example, the voice assistant might notify the user, "Your pet is hungry."

[1009] Examples of prompts for a generative AI model include: "Try to determine if a robot working in a factory is experiencing stress" and "Analyze current health scan data to see if there are any abnormalities."

[1010] This system allows for detailed monitoring of the emotions and health of pets and workers, enabling early detection and response to abnormalities. Specific hardware includes cameras (e.g., Logitech C920) and sensors, and software includes OpenCV and TensorFlow.

[1011] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1012] Step 1:

[1013] The device uses a camera to capture image data of the pet or working environment. Specifically, a camera (e.g., Logitech C920) captures images in real time and stores the image data in its internal memory. The input is the real-time image, and the output is the stored image data.

[1014] Step 2:

[1015] The terminal uses a microphone means to capture audio data of the pet or working environment. The microphone records the audio in real time and stores the audio data in its internal memory. The input is the real-time audio, and the output is the stored audio data.

[1016] Step 3:

[1017] The terminal acquires the biometric data of the pet or worker using a sensor means. The sensor measures the body temperature and heart rate and stores the data in the internal memory. The input is the real-time biometric data, and the output is the stored biometric data.

[1018] Step 4:

[1019] The terminal transmits the collected image data, voice data, and biometric data to the server using a data transmission means. Specifically, each piece of data is compressed and packaged at regular intervals and sent to the server via the Internet. The input is the collected data, and the output is the transmitted data.

[1020] Step 5:

[1021] The server uses data analysis means to convert the received data into an analyzable format. Specifically, image data is converted into grayscale using OpenCV, and audio data is converted into a spectrogram using an audio analysis program. Biometric data is normalized. The input is the transmitted data, and the output is the data converted into an analyzable format.

[1022] Step 6:

[1023] The server performs emotion analysis using a generative AI model. Specifically, the converted data is input into a TensorFlow model to estimate the emotional state of the pet or worker. The main output is the estimated emotional state.

[1024] Step 7:

[1025] The server uses anomaly detection means to detect anomalies based on the analysis results. Specifically, it compares the estimated emotional and health states with predetermined standards and generates an alert if an abnormality is detected. The input is the analysis results, and the output is an alert.

[1026] Step 8:

[1027] The server sends the generated alert to the user's terminal via a notification means. Specifically, the alert content is displayed as a text message on the user's terminal screen. The input is the generated alert, and the output is a notification to the user's terminal.

[1028] Step 9:

[1029] The notification means of the device notifies the user of the generated alert visually or audibly. Specifically, the device displays an alert message on the screen and the voice assistant plays a corresponding voice message. The input is the sent alert, and the output is the notification to the user.

[1030] The above steps realize comprehensive monitoring of the emotions and health status of pets and the working environment.

[1031] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1032] MODE FOR CARRYING OUT THE INVENTION

[1033] This invention provides a system that comprehensively monitors the emotions and health status of pets and also recognizes the emotions of the owner (user) to optimize the response. This allows the owner to quickly and accurately grasp the condition of their pet and take the optimal response. This system is realized by the following components and their operations:

[1034] Components

[1035] 1. Terminal: This terminal is equipped with a camera means for acquiring image data of the pet, a microphone means for acquiring voice data, and a sensor means for acquiring biometric data. It also includes a camera means for acquiring image data of the user and a microphone means for acquiring voice data.

[1036] 2. Server: Equipped with a data analysis means for receiving and analyzing acquired data, an emotion analysis means for recognizing and estimating the emotions of pets and users, and an anomaly detection means for detecting abnormalities and generating alerts.

[1037] 3. User: Owners and pet caregivers use the system to monitor and be notified of their pets' and their own emotional states.

[1038] Program processing

[1039] Below, we will explain in natural language how each element of this system works together.

[1040] Data collection

[1041] The device enters data collection mode upon user operation, activating the camera, microphone, and sensors to collect images, audio, and biometric data of the pet, as well as the user's image and audio data.

[1042] For example: The device's camera simultaneously captures images of both the pet and the user, the microphone records audio from both, and sensors measure the pet's temperature and heart rate.

[1043] Data transmission

[1044] The terminals transmit the collected data to a server via the network. The transmission is done in real time, but in the event of a network failure, the data is temporarily stored in the terminal.

[1045] Example: The device sends audio and video data of the pet and the user, as well as the pet's heart rate data, to the server every 30 seconds.

[1046] Data analysis

[1047] The server converts the received data into an analyzable format, extracting facial and movement features from the video data and analyzing the tone and strength of the voice from the audio data. The biometric data is then organized into an easily readable format.

[1048] Example: The server analyzes facial expressions from video data of a pet and simultaneously estimates emotions from video data of the user.

[1049] Emotion analysis

[1050] The server analyzes the emotions of the pet and the user from the preprocessed data using a generative AI model. Based on the movement of the pet's tail and ears, and the tone of its cries, it infers that the pet is happy, and based on the user's facial expression and tone of voice, it infers that the owner is at ease.

[1051] Example: Estimating the emotions of a pet wagging its tail and a user smiling and talking in video data.

[1052] Anomaly detection

[1053] The server monitors the pet's health based on the analysis of emotional and biological data, and generates alerts for abnormalities taking into account the emotional state of the pet and the user.

[1054] Example: If the user looks worried about a pet with an abnormally high body temperature, an alert will be generated saying, "Take your pet to the vet immediately."

[1055] notification

[1056] The server sends the generated alerts and the results of emotion analysis to the user's device.

[1057] The terminal notifies the user visually or audibly.

[1058] Example: The device displays the message "Your pet has a high temperature. Please take immediate action" on the screen and simultaneously announces "Your pet is anxious" through voice.

[1059] Interface and voice assistant integration

[1060] The terminal provides a user interface that displays the emotional analysis results and health data of the pet and the user in graphs and text.

[1061] The device will provide real-time notifications through the voice assistant, for example, letting users know when their pet is hungry, allowing them to take prompt action.

[1062] Example: A voice assistant might inform the user that "your pet is hungry" and simultaneously advise that "your smiling face will make your pet feel at ease."

[1063] summary

[1064] This system allows for detailed monitoring of the emotions and health of both owners and their pets. By analyzing the collected data and issuing appropriate alerts and notifications based on the results, it is possible to detect and respond to abnormalities in pets and owners early on. This makes it a useful tool for both pets and owners to live happy and healthy lives.

[1065] The processing flow will be explained below.

[1066] Step 1:

[1067] The terminal starts a data collection mode by user operation. Here, the camera means for acquiring image data of the pet, the microphone means for acquiring audio data, and the sensor means for acquiring biometric data are activated. At the same time, the camera means for acquiring image data of the user and the microphone means for acquiring audio data are also activated.

[1068] Step 2:

[1069] The device collects and stores the collected video data of the pet and the user in real time. For example, the pet's video is recorded at 30 frames per second, and the user's video is recorded at the same time. Audio data is saved every three seconds. The sensor also measures the pet's temperature and heart rate every 30 seconds.

[1070] Step 3:

[1071] The device sends the collected data to a server via the Internet. The data is sent in real time, but if there are network problems, it is temporarily stored in the device.

[1072] Step 4:

[1073] The server converts the received data into an analyzable format, extracting specific frames from the video data and analyzing the tone and strength of the voice from the audio data. Biometric data is standardized and converted into an easily analyzable format.

[1074] Step 5:

[1075] The server uses a generative AI model to analyze the emotions of the pet and the user from pre-processed data, for example, by using the pet's tail movements and the tone of its cries, as well as the user's facial expressions and tone of voice.

[1076] Step 6:

[1077] The server comprehensively evaluates the emotional states of the pet and the user based on the results of emotion analysis. For example, if the pet is anxious and the user is also showing signs of impatience or anxiety, the server determines that special attention is required.

[1078] Step 7:

[1079] The server monitors your pet's health in real time, continuously analyzing food intake, water intake, waste data, temperature data, and heart rate data to identify any data points that are outside of normal ranges.

[1080] Step 8:

[1081] The server generates an alert when an abnormality is detected, and determines the specific content of the alert based on the pet's biological and emotional data, as well as the user's emotional data.

[1082] Step 9:

[1083] The server then sends the generated alert and the emotion analysis results to the user's device, including specific instructions such as "Your pet has a high temperature. Please take it to the vet immediately."

[1084] Step 10:

[1085] The device notifies the user of alerts and emotion analysis results received from the server either visually or audibly. If the notification is visual, it will be displayed on the screen, and if it is audible, it will be conveyed to the user through the voice assistant.

[1086] Step 11:

[1087] The user can check the notified information and take appropriate action depending on the pet's condition. For example, if they receive a fever alert, they can immediately take their pet to a veterinarian.

[1088] Step 12:

[1089] The device displays the pet and user's emotional analysis results and health data in graphs and text, allowing users to check past data and manage their pet's long-term health.

[1090] Step 13:

[1091] The device will provide real-time notifications through the voice assistant, for example, telling the user, "Your pet is hungry," and offering advice such as, "Your smile will make your pet feel at ease."

[1092] Example 2

[1093] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1094] Today, there is a need for a system that comprehensively monitors the health and emotional state of pets and their owners. However, existing systems only handle pet health and emotional data and are unable to perform comprehensive analysis that includes the owner's emotional state. Furthermore, abnormality detection and real-time notification tend to be delayed, making prediction and early response difficult. Therefore, there is a need for a systematic system that can more precisely monitor the health and emotional state of pets and their owners and respond quickly and appropriately when abnormalities occur.

[1095] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1096] In this invention, the server includes means for converting received data into an analyzable format and extracting facial and movement features, means for analyzing the tone and strength of voice from the audio data, and means for analyzing and estimating the emotions of the pet and the user from the preprocessed data using a generative AI model. This enables comprehensive analysis of the emotional and health states of the pet and the owner, and if an abnormality is detected, it is possible to quickly and accurately generate an alert and notify the user in real time.

[1097] A "terminal" is a device that starts a data collection mode by user operation, collects images, sounds, and biometric data of the pet and the user, and transmits the collected data to a server via the Internet.

[1098] "Camera means" refers to an optical sensor device for capturing image data of the pet and the user.

[1099] The "microphone means" is an acoustic sensor device for acquiring voice data of the pet and the user.

[1100] The "sensor means" is a biometric device for measuring the pet's biological data, specifically, body temperature, heart rate, etc.

[1101] The "data transmission means" is a communication function for transmitting data collected by the terminal to a server via a network.

[1102] "Data analysis means" refers to software and algorithms that convert data received on the server into an analyzable format, extract facial and movement features, and analyze voice tone and strength from audio data.

[1103] "Emotion Analysis Means" means software and algorithms for analyzing and inferring pet and user emotions from data preprocessed using a generative AI model.

[1104] The "abnormality detection means" is software and a system for monitoring the health of the pet and the user based on the analyzed emotional data and biometric data, and generating an alert if an abnormality is detected.

[1105] The "notification means" is a function for visually or audibly notifying the user of the generated alert or the results of emotion analysis.

[1106] The "user interface" is a display and operating environment that displays analysis results in text and graph format, and works in conjunction with a voice assistant to provide real-time notifications.

[1107] A "generative AI model" is an artificial intelligence model used to analyze pet and user emotions from preprocessed data, and includes, for example, GPT-3, image recognition models, and speech recognition models.

[1108] This invention is a system that comprehensively monitors the emotions and health status of pets, and also recognizes the emotions of the owner (user) to optimize the response. This allows the user to quickly and accurately grasp the condition of their pet and take the optimal response. This system is realized by the following components and their operations.

[1109] Components

[1110] 1. Terminal: This terminal is equipped with a camera means for acquiring image data of the pet, a microphone means for acquiring audio data, and a sensor means for acquiring biometric data. It also includes a camera means for acquiring image data of the user and a microphone means for acquiring audio data.

[1111] 2. Server: The server has a data analysis means for receiving and analyzing the acquired data, an emotion analysis means for recognizing and estimating the emotions of the pet and the user, and an anomaly detection means for detecting anomalies and generating alerts.

[1112] 3. User: Owners and pet caregivers use the system to monitor and be notified of their pets' and their own emotional states.

[1113] Program processing

[1114] Data collection

[1115] The device enters data collection mode when the user operates it, activating the camera, microphone, and sensors to collect images, audio, and biometric data of the pet, as well as the user's image and audio data.

[1116] Example: The device's camera captures video of the pet and the user at 30 frames per second, the microphone records audio from both parties at a sampling rate of 44.1 kHz, and the sensor measures the pet's temperature once per second and its heart rate in real time.

[1117] Data transmission

[1118] The device sends the collected data to a server via the Internet. The data is sent in real time, but if a network failure occurs, the data is temporarily stored in the device's internal storage and will be sent again once the network is restored.

[1119] Example: The terminal sends data to the server through the REST API using the HTTP protocol. For example, it sends a JSON formatted data packet every 30 seconds. The data sent also includes a timestamp to ensure data integrity.

[1120] Data analysis

[1121] The server converts the received data into an analyzable format, extracting facial and movement features from the video data and analyzing the tone and strength of the voice from the audio data. Biometric data is also converted into easy-to-read graphs and numbers.

[1122] Example: The server uses Python's OpenCV library to detect facial landmarks from video data. For audio data, the Librosa library is used to extract audio features and analyze the frequency spectrum. Biometric data is organized into a data frame using the Pandas library and visualized as time-series data.

[1123] Emotion analysis

[1124] The server uses a generative AI model to analyze the emotions of the pet and the user from preprocessed data. It analyzes the movements of the pet's tail and ears, the tone of its cries, and other factors to determine whether the pet is happy, and the user's facial expression and tone of voice to determine whether the owner is at ease.

[1125] Example: Using the generative AI model GPT-3, emotions are expressed in text format based on data obtained from image recognition models (e.g., ResNet) and speech recognition models (e.g., DeepSpeech). For example, the most appropriate emotion for a given situation can be estimated based on the width and frequency of a pet's tail wagging or the degree of a user's smile.

[1126] Anomaly detection

[1127] The server monitors the pet's health based on the analysis of its emotional and biological data, and generates an alert if an abnormality is detected. The alert is generated taking into account the pet's emotional state.

[1128] Example: The server sets a threshold based on the analysis results and detects abnormalities. For example, if the pet's body temperature exceeds 39 degrees or the heart rate exceeds 200 bpm, an alert will be generated stating "possible high fever." It will also alert the user if the emotional state of the pet and the user do not match.

[1129] notification

[1130] The server sends the generated alerts and the results of emotion analysis to the user's device.

[1131] The terminal notifies the user visually or audibly.

[1132] Example: The server uses WebSocket or Push Notification services to send notifications to the device in real time. The device displays a message on the screen saying "Your pet's temperature is high. Please take action immediately" and uses a Text-to-Speech (TTS) engine to notify the user by voice, "Your pet is feeling anxious."

[1133] Interface and voice assistant integration

[1134] The device provides a user interface that displays the pet and user's emotional analysis and health data in graphs and text, and also provides real-time notifications via a voice assistant.

[1135] Example: The device uses front-end frameworks such as React and Vue.js to visualize the analysis results in the browser. It also works with a voice assistant to notify users in real time that their pet is hungry, while simultaneously providing advice such as, "Your smile will make your pet feel at ease."

[1136] Prompt Sentence Examples

[1137] Below are some examples of prompt sentences to input to the generative AI model.

[1138] Prompt Sentence Examples

[1139] Analyze the health and emotions of your pet, as well as your emotions. The following data will be collected: pet video, audio, and biometric data, and user video and audio data.

[1140] This system allows users to closely monitor their pets' emotions and health. By analyzing the collected data and providing appropriate alerts and notifications based on the results, it is possible to detect and respond to abnormalities in both pets and users early on. This makes it a useful tool for helping both pets and users live happy and healthy lives.

[1141] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1142] Step 1: Data collection

[1143] The device starts data collection mode when the user operates it. The camera (image data), microphone (audio data), and sensor (biometric data) are activated to collect images, audio, and biometric data of the pet. At the same time, the user's image and audio data are also collected.

[1144] Input: The user operates the terminal to start data collection mode.

[1145] Output: A set of video data, audio data, and biometric data collected by the device.

[1146] How it works: The device's camera captures video of the pet and the user at 30 frames per second, the microphone records audio from both parties at 44.1kHz, and the sensor measures the pet's temperature every second and heart rate in real time.

[1147] Step 2: Send data

[1148] The terminal transmits the collected data to a server via the Internet in real time, but in the event of a network failure, the data is temporarily stored in the terminal.

[1149] Input: A set of collected image, audio, and biometric data.

[1150] Output: A set of image data, audio data, and biometric data sent to the server.

[1151] Specific operation: The terminal uses the HTTP protocol to send data to the server via the REST API every 30 seconds. The data is in JSON format and each data item is time-stamped.

[1152] Step 3: Data analysis

[1153] The server converts the received data into an analyzable format, extracting facial and movement features from the video data and analyzing the tone and strength of the voice from the audio data. The biometric data is then organized into graphs and numerical values.

[1154] Input: A set of image data, audio data, and biometric data sent to the server.

[1155] Output: Analyzed facial and movement characteristics, voice tone and strength, and organized biometric data.

[1156] Specific operation: The server uses Python's OpenCV library to detect facial landmarks from video data, and the Librosa library to extract audio features. The biometric data is organized into a data frame using the Pandas library and visualized.

[1157] Step 4: Sentiment Analysis

[1158] The server uses a generative AI model to analyze the emotions of the pet and the user from the pre-processed data.

[1159] Input: Analyzed facial and movement characteristics, voice tone and volume, and organized biometric data.

[1160] Output: Estimated pet and user emotions.

[1161] Specific operation: The server uses image recognition models (e.g., ResNet) and speech recognition models (e.g., DeepSpeech) as generative AI models to express emotions from data in text format. For example, by linking the behavior of a pet wagging its tail with the user's smile, it can infer that "the pet is happy" and "the owner is at ease."

[1162] Step 5: Anomaly detection

[1163] The server monitors the pet's health based on the analysis of emotional and biometric data, and generates an alert if an abnormality is detected.

[1164] Input: Estimated pet and user emotions, and organised biometric data.

[1165] Output: The generated alert.

[1166] Specific operation: The server sets a threshold and generates an alert saying "possible high fever" if, for example, the pet's body temperature exceeds 39 degrees or the heart rate exceeds 200 bpm. If an anxious expression on the user's face is also detected at the same time, the server issues a warning.

[1167] Step 6: Notification

[1168] The server sends the generated alerts and the results of emotion analysis to the user's device.

[1169] The terminal notifies the user visually or audibly.

[1170] Input: Generated alerts, sentiment analysis results.

[1171] Output: A visual or audio notification.

[1172] Specific operation: The server sends a real-time notification to the device using WebSocket or Push Notification services. The device displays the message "Your pet's temperature is high. Please take action immediately" on the screen and uses a Text-to-Speech (TTS) engine to notify the user by voice, "Your pet is anxious."

[1173] Step 7: Interface and Voice Assistant Integration

[1174] The terminal provides a user interface that displays the emotional analysis results and health data of the pet and the user in graphs and text.

[1175] The device will provide real-time notifications via the voice assistant.

[1176] Input: Sentiment analysis results, health data, generated alerts.

[1177] Output: Data displayed in graphs and text, notifications via voice assistant.

[1178] How it works: The device visualizes the analysis results in a browser using front-end frameworks such as React and Vue.js. It also works with a voice assistant to notify users in real time that their pet is hungry, while simultaneously providing advice such as, "Your smile will make your pet feel at ease."

[1179] (Application example 2)

[1180] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1181] Conventional systems exist that comprehensively monitor a pet's emotions and health status, and also recognize the owner's emotions to optimize responses. However, a system that utilizes such a system in an autonomous vehicle to monitor the status of the pet and owner in real time and automatically optimize the in-vehicle environment has not yet been realized. This has made it difficult for owners to grasp the pet's condition in detail and respond appropriately while on the move. The present invention aims to solve this problem.

[1182] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a data analysis means for analyzing acquired data, an emotion analysis means for estimating the pet's emotion and health state, an abnormality detection means for detecting an abnormality based on the analysis result and generating an alarm, and an environment adjustment means for optimizing the in-vehicle environment. This makes it possible to monitor the status of the pet and its owner in real time inside the autonomously driving vehicle and automatically adjust the in-vehicle environment based on the analysis result.

[1183] The "photography means" is a device for acquiring image data of a pet.

[1184] The "recording means" is a device for acquiring voice data of a pet.

[1185] "Measuring means" refers to a device for acquiring biological data of a pet.

[1186] The "emotion analysis means" is a device that analyzes the acquired image data, audio data, and biometric data to estimate the pet's emotions.

[1187] The "data transmission means" is a device for transmitting the acquired data to the server.

[1188] The "data analysis means" is a device for analyzing the emotions and health condition of pets on the server.

[1189] The "abnormality detection means" is a device that detects abnormalities in the pet based on the analysis results and generates an alarm.

[1190] The "notification means" is a device for notifying the user of the generated alert.

[1191] The "environment adjustment means" is a device that monitors the condition of pets and their owners inside an autonomous vehicle and optimizes the in-vehicle environment based on the analysis results.

[1192] This invention relates to a system that monitors the status of pets and their owners in an autonomous vehicle and optimizes the in-vehicle environment. This system is realized by a number of components and their cooperation.

[1193] Components

[1194] 1. Photography method: Image data of the pet is obtained using a camera installed in the vehicle.

[1195] Specifically, a general webcam (e.g., Logitech C920) is used.

[1196] 2. Recording method: Record your pet's voice using a microphone in the vehicle.

[1197] Specifically, a high-performance microphone (e.g., Blue Yeti) is used.

[1198] 3. Measurement methods: Use sensors to obtain your pet's biological data.

[1199] Specifically, biosensors for pets (e.g., Fitbit pet sensors) are used.

[1200] 4. Emotion analysis method: Analyze the acquired data to estimate the pet's emotions.

[1201] Specifically, video and audio data are analyzed using an AI model (e.g., a generative AI model using Keras).

[1202] 5. Data transmission means: Transmits the acquired data to the server.

[1203] In particular, a communication device is utilized to transmit data over a network connection.

[1204] 6. Data analysis means: The data received on the server is analyzed to determine the pet's emotions and health condition.

[1205] Specifically, data analysis software (e.g., Python scripts) that runs on the server side is used.

[1206] 7. Anomaly detection means: Detects anomalies based on the analysis results and generates an alert.

[1207] The server generates an alert if it detects an abnormality in the health status.

[1208] 8. Notification means: Notifies the user of the generated alert.

[1209] Specifically, audio notifications and screen displays are used.

[1210] 9. Environmental control means: A device for optimizing the in-vehicle environment.

[1211] This includes devices that automatically control temperature, lighting, music, etc. inside the vehicle.

[1212] Explaining program processing in natural language

[1213] The server first collects data through photography, audio recording, and measurement. It then transmits this data in real time to the server. The server processes the data using a generative AI model using Keras to analyze the emotions and health of the pet and owner. Based on the analysis results, it then generates an alert if necessary and notifies the user. At the same time as the notification, the vehicle's environmental control measures are used to optimize the temperature, lighting, music, and other aspects of the vehicle interior.

[1214] Specific examples

[1215] For example, a camera inside the car captures video of the pet and its owner, and a microphone records audio. Measurements are made to measure the pet's heart rate and body temperature. This data is sent to a server in real time, and a generative AI model using Keras analyzes the data. If the server detects that the pet is in an unstable state, it generates an alarm and notifies the user with an audio message saying, "Your pet is in an unstable state," while simultaneously adjusting the in-car environment to make it more comfortable for the pet.

[1216] Prompt Sentence Examples

[1217] "Measure the pet's temperature and analyze the video data captured by the camera. Also, analyze the owner's voice data and infer the emotions of both parties. Based on the analysis results, play appropriate music."

[1218] In this way, an environment can be provided in which both pets and owners can move around with peace of mind.

[1219] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1220] Step 1:

[1221] The terminal activates the photographing means, recording means, and measuring means to collect data on the pet and its owner. Image data acquired by the photographing means, audio data acquired by the recording means, and biometric data acquired by the measuring means (e.g., the pet's body temperature and heart rate) are collected in real time. Input data includes image data, audio data, and biometric data.

[1222] Step 2:

[1223] The terminal temporarily stores the collected data and then transmits it to the server via the network. At this time, transmission may be temporarily delayed depending on the network status, but this data transmission method ensures stable transmission. The input is the image data, voice data, and biometric data collected in step 1, and the output is that these data are transmitted to the server.

[1224] Step 3:

[1225] The server converts the received image data into a processable format. Specifically, this involves extracting facial and movement features of the pet and owner from the image data. This process uses an image processing library such as OpenCV. The input is the transmitted image data, and the output is the data with the extracted features.

[1226] Step 4:

[1227] The server analyzes the received voice data and analyzes the tone and stress of the voice. This process uses a voice analysis library such as Librosa. The input is the transmitted voice data, and the output is the analyzed voice feature data.

[1228] Step 5:

[1229] The server organizes and analyzes the received biometric data, including converting the pet's heart rate and temperature data into an easily readable format. The input is the transmitted biometric data, and the output is the analyzed biometric data.

[1230] Step 6:

[1231] The server uses emotion analysis means to analyze the emotions of the pet and owner from the preprocessed image data, audio data, and biometric data based on the generative AI model. For example, it analyzes the pet's tail movements and the tone of its cries, as well as the owner's facial expressions and tone of voice. The input is the feature-extracted data and analyzed audio and biometric data, and the output is an estimation of the emotional state of the pet and owner.

[1232] Step 7:

[1233] The server monitors the pet's health using anomaly detection means based on the analysis results. If an abnormality is detected, an emergency alert is generated. For example, if the pet's body temperature is high and the user looks anxious, an alert is generated. The input is the emotion analysis result and biometric data, and the output is the alert generation.

[1234] Step 8:

[1235] The device notifies the user of the generated alert and the results of the emotion analysis by audio or visual notification. For example, the device may display "Your pet's temperature is high. Please take action immediately" on the screen and simultaneously notify the user by audio notification, "Your pet is feeling anxious." The input is the alert and the results of the emotion analysis, and the output is the notification to the user.

[1236] Step 9:

[1237] The device adjusts the in-car environment in real time, optimizing the temperature, lighting, music, etc. based on the analysis results. For example, if the pet is relaxed and the owner feels at ease, relaxing music will be played and the lighting will be softened. The input is the results of emotion analysis and health status, and the output is the adjustment of the in-car environment.

[1238] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1239] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1240] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1241] [Fourth embodiment]

[1242] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1243] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1244] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1245] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1246] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1247] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1248] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1249] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1250] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1251] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1252] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1253] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1254] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1255] MODE FOR CARRYING OUT THE INVENTION

[1256] This invention provides a system for comprehensively monitoring pets' emotions and health conditions, allowing owners to quickly and accurately grasp their pets' conditions and take appropriate action. This system is realized by the following components and their operations:

[1257] Components

[1258] 1. Terminal: This terminal is equipped with a camera means for acquiring image data of the pet, a microphone means for acquiring audio data, and a sensor means for acquiring biometric data.

[1259] 2. Server: Has a data analysis means for receiving and analyzing the acquired data, and an anomaly detection means for detecting anomalies and generating alerts.

[1260] 3. User: The owner or pet caregiver will use the system to monitor the pet's condition and receive notifications.

[1261] Program processing

[1262] Below, we will explain in natural language how each element of this system works together.

[1263] Data collection

[1264] The device activates cameras, microphones, and sensors to continuously collect data to monitor your pet's condition in real time.

[1265] Example: The device's camera captures video of your pet, the microphone records its sounds, and sensors measure its temperature and heart rate.

[1266] Data transmission

[1267] The terminal transmits the collected data to a server via a network.

[1268] Example: A device sends a pet's heart rate data to a server every 30 seconds.

[1269] Data analysis

[1270] The server converts the received data into an analyzable format and uses a generative AI model to estimate the pet's emotional state.

[1271] Example: The server analyzes the eye movements of a pet from video data and infers that the pet is excited.

[1272] Anomaly detection

[1273] The server monitors the pet's health based on the analysis results, and immediately generates an alert if an abnormality is detected.

[1274] Example: If the server analyzes body temperature data and detects a temperature outside the normal range, it generates an alert saying, "Your pet may have a fever."

[1275] notification

[1276] The server sends the generated alerts and the results of emotion analysis to the user's device.

[1277] The terminal notifies the user visually or audibly.

[1278] Example: The device displays the message "Pet has a high temperature. Please take him to the vet."

[1279] Interface and voice assistant integration

[1280] The terminal provides a user interface that displays the results of a pet's emotional analysis and health status data in graphs and text.

[1281] The device works in conjunction with a voice assistant to provide real-time notifications to the user via voice.

[1282] Example: A voice assistant informs the user that "your pet is hungry."

[1283] summary

[1284] This system allows owners to closely monitor their pets' emotions and health. By analyzing the collected data and providing appropriate alerts and notifications based on the results, owners can detect and respond to abnormalities in their pets early, making it a useful tool for promoting the longevity and happiness of pets.

[1285] The processing flow will be explained below.

[1286] Step 1:

[1287] The device enters data collection mode upon user operation, activating the camera, microphone, and sensors to continuously capture video, audio, and biometric data of the pet.

[1288] Step 2:

[1289] The device saves the captured video data at regular intervals (e.g., 30 frames per second), records audio data (e.g., saves audio data every 3 seconds), and periodically records biometric data such as body temperature and heart rate (e.g., measures every 30 seconds).

[1290] Step 3:

[1291] The terminal transmits the collected data to a server via the Internet. The transmission is performed in real time as the data is collected, but in the event of a network failure, the data is temporarily stored in the terminal.

[1292] Step 4:

[1293] The server converts the received data into an analyzable format, extracting characteristic frames from the video data and analyzing the volume and tone of the bird's calls from the audio data. The biometric data is then organized into an easily readable format.

[1294] Step 5:

[1295] The server uses a generative AI model to analyze the pet's emotions from the pre-processed data. For example, it can infer that the pet is excited based on the presence of a wagging tail in the video data or a high-pitched cry in the audio data.

[1296] Step 6:

[1297] The server monitors the pet's health based on the results of data analysis. It continuously analyzes data such as food intake, water intake, feces, body temperature, and heart rate, and generates an abnormality alert when values ​​outside the normal health range are detected.

[1298] Step 7:

[1299] The server then sends the generated alerts and the results of the emotion analysis to the owner's device, which includes specific instructions such as "The pet has a high temperature. Please take it to the vet."

[1300] Step 8:

[1301] The device notifies the user of alerts received from the server and the results of sentiment analysis. Notifications are displayed visually and, if necessary, audio notifications.

[1302] Step 9:

[1303] The user can check the information provided and take appropriate action depending on the pet's condition. For example, if they receive an alert about a fever, they can immediately take their pet to a veterinarian.

[1304] Step 10:

[1305] The device displays the pet's emotional analysis and health data in graphs and text, and allows users to check past data, allowing them to manage their pet's health over the long term.

[1306] Step 11:

[1307] The device will provide real-time notifications through the voice assistant, for example, letting users know when their pet is hungry, allowing them to take prompt action.

[1308] Example 1

[1309] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1310] It is difficult for pet owners to quickly and accurately grasp their pets' emotional and health states. Furthermore, there is a lack of effective systems for early detection of abnormalities in pets and appropriate response. Therefore, in order to maintain the happiness and health of pets, there is a need for a system that can comprehensively monitor the emotions and health states of pets and immediately notify of abnormalities.

[1311] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1312] In this invention, the server includes a data analysis means that standardizes collected data and analyzes it using a generative AI model, an anomaly detection means that analyzes the pet's emotions and health condition on the server and detects abnormalities, and a notification means that generates an alert and notifies the user when an abnormality is detected. This makes it possible to analyze various pet data and realize anomaly detection and notification in real time.

[1313] The "camera means" is a device for acquiring image data of a pet.

[1314] The "microphone means" is a device for acquiring voice data of a pet.

[1315] The "sensor means" is a device for acquiring biological data of a pet.

[1316] The "emotion analysis means" is an algorithm or software for analyzing the acquired image data, audio data, and biometric data to estimate the pet's emotions.

[1317] "Data analysis means" means algorithms or software for normalizing collected data and analyzing it using generative AI models.

[1318] The "data transmission means" is a communication module or software for transmitting acquired data to a server.

[1319] The "abnormality detection means" is an algorithm or software that analyzes the pet's emotions and health condition on the server and detects abnormalities in the pet based on the analysis results.

[1320] The "notification means" is a device or software for notifying the user of the generated alerts and analysis results.

[1321] A "user interface" is a display device or software for visually displaying generated alerts and analysis results.

[1322] "Audio notification means" means a device or software that uses a voice assistant to provide real-time notifications to a user by voice.

[1323] MODE FOR CARRYING OUT THE INVENTION

[1324] This invention provides a system for comprehensively monitoring pets' emotions and health conditions, enabling owners to quickly and accurately grasp their pets' conditions and take appropriate action. This system is realized through cooperation between terminals, a server, and users.

[1325] Components

[1326] 1. Device:

[1327] The terminal is equipped with a camera means for acquiring image data of the pet, a microphone means for acquiring audio data, and a sensor means for acquiring biological data.

[1328] Specifically, the camera captures real-time footage of your pet, the microphone records barks and environmental sounds, and the sensors measure your pet's temperature and heart rate.

[1329] 2. Server:

[1330] The server has a data analysis means for receiving data sent from the terminal and converting it into an analyzable format.

[1331] This data analysis method uses a generative AI model to estimate the emotional state of your pet.

[1332] In addition, it also has an abnormality detection means that detects abnormalities in pets based on the analysis results and generates an alert.

[1333] 3. User:

[1334] Users (owners or pet caregivers) monitor the condition of their pets through this system.

[1335] The health and emotional state of the pet can be understood through a means of displaying alerts and analysis results generated from the server on a user interface.

[1336] You can also use the voice assistant to receive real-time notifications via voice.

[1337] Data collection:

[1338] The device monitors your pet's condition in real time and continuously collects data, including video, audio, body temperature, and heart rate.

[1339] Example: A device camera takes a video of your pet every minute, a microphone records its sounds, and sensors measure your pet's temperature and heart rate every minute.

[1340] Data transmission:

[1341] The terminal compresses the collected data and transmits it to the server via the network.

[1342] Example: A device sends compressed heart rate data and video data to a server every five minutes.

[1343] Data Analysis:

[1344] The server normalizes the received data and analyzes it using a generative AI model to estimate the pet's emotional state and monitor its health.

[1345] Example: Analyzing a pet's facial expressions from video data to estimate its state of excitement or relaxation.

[1346] Anomaly detection:

[1347] The server constantly monitors the pet's health based on the analysis results, and immediately generates an alert if an abnormality is detected.

[1348] Example: If the body temperature exceeds the normal range, generate an alert saying "Suspected Fever."

[1349] notification:

[1350] The server sends the generated alerts and analysis results to the user's device.

[1351] The terminal displays alerts and analysis results on the user interface and notifies the user visually or audibly.

[1352] Example: The device displays a message on the screen saying, "Your pet has a high temperature. Please take him to the vet." The voice assistant also informs the user that "Your pet's temperature is above normal."

[1353] Examples of prompts:

[1354] "Tell me your pet's latest emotional state"

[1355] Check your pet's temperature

[1356] "I want to know if my pet is stressed."

[1357] This system allows owners to closely monitor their pets' emotions and health. By analyzing the collected data and providing appropriate alerts and notifications based on the results, owners can detect abnormalities in their pets early and respond quickly, making it a useful tool for promoting the longevity and happiness of pets.

[1358] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1359] Step 1:

[1360] Data collection:

[1361] The device activates cameras, microphones, and sensors to collect data to monitor your pet's condition in real time.

[1362] Input: Pet video, audio, temperature, heart rate

[1363] Output: Collected image data, audio data, and biometric data

[1364] How it works: The device's camera captures real-time video of your pet, the microphone records your pet's cries and environmental sounds, and the sensors measure your pet's temperature and heart rate every minute.

[1365] Step 2:

[1366] Data transmission:

[1367] The terminal transmits the collected data to a server via a network.

[1368] Input: Collected image data, audio data, and biometric data

[1369] Output: Data sent to the server

[1370] How it works: At a predefined interval (e.g., every 5 minutes), the device compresses and transmits data, including heart rate and temperature measurements, as well as video data, to a server using the secure HTTP protocol.

[1371] Step 3:

[1372] Data Analysis:

[1373] The server converts the received data into an analyzable format and uses a generative AI model to infer the emotional state.

[1374] Input: Data sent to the server (image data, audio data, biometric data)

[1375] Output: Estimated pet's emotional state

[1376] How it works: The server first standardizes the format of the received data, then divides the video data into frames, analyzes the audio data, and inputs it into a generative AI model. The server then analyzes the pet's facial expressions and vocal characteristics to estimate its emotional state.

[1377] Step 4:

[1378] Anomaly detection:

[1379] The server monitors the pet's health based on the analysis results and generates an alert if an abnormality is detected.

[1380] Input: Pet emotional state estimation results, biometric data

[1381] Output: Alert message

[1382] Specific operation: The server compares the analyzed data with normal health data, and if an abnormality is detected, it generates an alert such as "body temperature is outside the normal range."

[1383] Step 5:

[1384] notification:

[1385] The server sends the generated alerts and the results of emotion analysis to the user's device, and the device notifies the user visually or audibly.

[1386] Input: Alert message, sentiment analysis results

[1387] Output: User notification

[1388] What happens: The server immediately sends the generated alert to the user's device, which then warns the user via a pop-up notification or voice assistant. For example, a message might appear on the screen saying, "Your pet has a high temperature. Please take it to the vet."

[1389] Step 6:

[1390] User Interface and Voice Assistant Integration:

[1391] The device provides a user interface that displays the results of a pet's emotional analysis and health data in graphs and text, and works in conjunction with a voice assistant to provide real-time notifications to the user via voice.

[1392] Input: Emotion analysis results, biometric data

[1393] Output: Display on the user interface, audio notification

[1394] Specific operation: The user interface displays graphs of heart rate and body temperature fluctuations over the past 24 hours, and the voice assistant notifies the user with notifications such as "Your pet's stress level is increasing."

[1395] (Application example 1)

[1396] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1397] Conventional health and emotional monitoring systems for pets and workers only collect individual data, which means they lack comprehensive analysis of their condition. Furthermore, they lack real-time alert notifications when abnormalities occur, making it difficult to respond quickly. Therefore, there is a need for a system that can comprehensively monitor the health and emotional state of pets and workers in real time and respond quickly when abnormalities are detected.

[1398] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1399] In this invention, the server includes a data analysis means, an emotion analysis means, and an anomaly detection means, which makes it possible to comprehensively analyze the health and emotional states of pets and workers, and to quickly generate and notify an alert when an anomaly is detected.

[1400] The "data analysis means" is a device or program that analyzes the acquired image data, audio data, and biometric data on the server to analyze the emotions and health status of the pet or worker.

[1401] The "emotion analysis means" is a device or program for estimating the emotions of a pet or worker based on acquired image data, audio data, and biometric data.

[1402] The "abnormality detection means" is a device or program for detecting abnormalities in a pet or worker based on the analysis results obtained by the data analysis means and the emotion analysis means, and generating an alert.

[1403] The "notification means" is a device or program for notifying the user of the generated alert.

[1404] "Camera means" is a device for acquiring image data of a pet or working environment.

[1405] The "microphone means" is a device for acquiring sound data of a pet or the working environment.

[1406] The "sensor means" is a device for acquiring biometric data of a pet or worker.

[1407] The "data transmission means" is a device or program for transmitting the acquired data to the server.

[1408] This invention is a system that monitors the emotions and health of pets by comprehensively acquiring and analyzing images, sounds, and biometric data of pets or their working environment. This allows for rapid response when an abnormality is detected. The following explains how each element of this system works together.

[1409] First, the terminal is equipped with a camera, microphone, and sensor means, and these devices are used to collect image data, audio data, and biometric data of pets or workers in real time. As a specific example, the terminal's camera captures images of pets, the microphone records sounds and the workplace, and the sensors measure body temperature and heart rate.

[1410] Next, the collected data is transmitted to the server via the data transmission means. For example, the terminal transmits the heart rate data of the pet or the worker to the server every 30 seconds.

[1411] On the server, the data analysis means converts the received data into an analyzable format and uses a generative AI model to estimate emotions and health conditions. For example, the server analyzes the eye movements of pets and workers from video data and estimates that they are "excited."

[1412] Furthermore, the server detects anomalies based on the analysis results. The anomaly detection means generates an alert if it detects data that exceeds the normal range. For example, if the server analyzes body temperature data and detects a temperature that exceeds the normal range, it generates an alert saying, "There is a possibility of fever."

[1413] The generated alert is sent to the user's device via a notification means. The device notifies the user visually or audibly. For example, the device may display a message on the screen saying, "The pet's temperature is high. Please take it to a veterinarian."

[1414] The device also provides a user interface that displays emotion analysis results and health status data in graphs and text. It also works with a voice assistant to provide real-time notifications to the user. For example, the voice assistant might notify the user, "Your pet is hungry."

[1415] Examples of prompts for a generative AI model include: "Try to determine if a robot working in a factory is experiencing stress" and "Analyze current health scan data to see if there are any abnormalities."

[1416] This system allows for detailed monitoring of the emotions and health of pets and workers, enabling early detection and response to abnormalities. Specific hardware includes cameras (e.g., Logitech C920) and sensors, and software includes OpenCV and TensorFlow.

[1417] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1418] Step 1:

[1419] The device uses a camera to capture image data of the pet or working environment. Specifically, a camera (e.g., Logitech C920) captures images in real time and stores the image data in its internal memory. The input is the real-time image, and the output is the stored image data.

[1420] Step 2:

[1421] The terminal uses a microphone means to capture audio data of the pet or working environment. The microphone records the audio in real time and stores the audio data in its internal memory. The input is the real-time audio, and the output is the stored audio data.

[1422] Step 3:

[1423] The terminal acquires the biometric data of the pet or worker using a sensor means. The sensor measures the body temperature and heart rate and stores the data in the internal memory. The input is the real-time biometric data, and the output is the stored biometric data.

[1424] Step 4:

[1425] The terminal transmits the collected image data, voice data, and biometric data to the server using a data transmission means. Specifically, each piece of data is compressed and packaged at regular intervals and sent to the server via the Internet. The input is the collected data, and the output is the transmitted data.

[1426] Step 5:

[1427] The server uses data analysis means to convert the received data into an analyzable format. Specifically, image data is converted into grayscale using OpenCV, and audio data is converted into a spectrogram using an audio analysis program. Biometric data is normalized. The input is the transmitted data, and the output is the data converted into an analyzable format.

[1428] Step 6:

[1429] The server performs emotion analysis using a generative AI model. Specifically, the converted data is input into a TensorFlow model to estimate the emotional state of the pet or worker. The main output is the estimated emotional state.

[1430] Step 7:

[1431] The server uses anomaly detection means to detect anomalies based on the analysis results. Specifically, it compares the estimated emotional and health states with predetermined standards and generates an alert if an abnormality is detected. The input is the analysis results, and the output is an alert.

[1432] Step 8:

[1433] The server sends the generated alert to the user's terminal via a notification means. Specifically, the alert content is displayed as a text message on the user's terminal screen. The input is the generated alert, and the output is a notification to the user's terminal.

[1434] Step 9:

[1435] The notification means of the device notifies the user of the generated alert visually or audibly. Specifically, the device displays an alert message on the screen and the voice assistant plays a corresponding voice message. The input is the sent alert, and the output is the notification to the user.

[1436] The above steps realize comprehensive monitoring of the emotions and health status of pets and the working environment.

[1437] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1438] MODE FOR CARRYING OUT THE INVENTION

[1439] This invention provides a system that comprehensively monitors the emotions and health status of pets and also recognizes the emotions of the owner (user) to optimize the response. This allows the owner to quickly and accurately grasp the condition of their pet and take the optimal response. This system is realized by the following components and their operations:

[1440] Components

[1441] 1. Terminal: This terminal is equipped with a camera means for acquiring image data of the pet, a microphone means for acquiring voice data, and a sensor means for acquiring biometric data. It also includes a camera means for acquiring image data of the user and a microphone means for acquiring voice data.

[1442] 2. Server: Equipped with a data analysis means for receiving and analyzing acquired data, an emotion analysis means for recognizing and estimating the emotions of pets and users, and an anomaly detection means for detecting abnormalities and generating alerts.

[1443] 3. User: Owners and pet caregivers use the system to monitor and be notified of their pets' and their own emotional states.

[1444] Program processing

[1445] Below, we will explain in natural language how each element of this system works together.

[1446] Data collection

[1447] The device enters data collection mode upon user operation, activating the camera, microphone, and sensors to collect images, audio, and biometric data of the pet, as well as the user's image and audio data.

[1448] For example: The device's camera simultaneously captures images of both the pet and the user, the microphone records audio from both, and sensors measure the pet's temperature and heart rate.

[1449] Data transmission

[1450] The terminals transmit the collected data to a server via the network. The transmission is done in real time, but in the event of a network failure, the data is temporarily stored in the terminal.

[1451] Example: The device sends audio and video data of the pet and the user, as well as the pet's heart rate data, to the server every 30 seconds.

[1452] Data analysis

[1453] The server converts the received data into an analyzable format, extracting facial and movement features from the video data and analyzing the tone and strength of the voice from the audio data. The biometric data is then organized into an easily readable format.

[1454] Example: The server analyzes facial expressions from video data of a pet and simultaneously estimates emotions from video data of the user.

[1455] Emotion analysis

[1456] The server analyzes the emotions of the pet and the user from the preprocessed data using a generative AI model. Based on the movement of the pet's tail and ears, and the tone of its cries, it infers that the pet is happy, and based on the user's facial expression and tone of voice, it infers that the owner is at ease.

[1457] Example: Estimating the emotions of a pet wagging its tail and a user smiling and talking in video data.

[1458] Anomaly detection

[1459] The server monitors the pet's health based on the analysis of emotional and biological data, and generates alerts for abnormalities taking into account the emotional state of the pet and the user.

[1460] Example: If the user looks worried about a pet with an abnormally high body temperature, an alert will be generated saying, "Take your pet to the vet immediately."

[1461] notification

[1462] The server sends the generated alerts and the results of emotion analysis to the user's device.

[1463] The terminal notifies the user visually or audibly.

[1464] Example: The device displays the message "Your pet has a high temperature. Please take immediate action" on the screen and simultaneously announces "Your pet is anxious" through voice.

[1465] Interface and voice assistant integration

[1466] The terminal provides a user interface that displays the emotional analysis results and health data of the pet and the user in graphs and text.

[1467] The device will provide real-time notifications through the voice assistant, for example, letting users know when their pet is hungry, allowing them to take prompt action.

[1468] Example: A voice assistant might inform the user that "your pet is hungry" and simultaneously advise that "your smiling face will make your pet feel at ease."

[1469] summary

[1470] This system allows for detailed monitoring of the emotions and health of both owners and their pets. By analyzing the collected data and issuing appropriate alerts and notifications based on the results, it is possible to detect and respond to abnormalities in pets and owners early on. This makes it a useful tool for both pets and owners to live happy and healthy lives.

[1471] The processing flow will be explained below.

[1472] Step 1:

[1473] The terminal starts a data collection mode by user operation. Here, the camera means for acquiring image data of the pet, the microphone means for acquiring audio data, and the sensor means for acquiring biometric data are activated. At the same time, the camera means for acquiring image data of the user and the microphone means for acquiring audio data are also activated.

[1474] Step 2:

[1475] The device collects and stores the collected video data of the pet and the user in real time. For example, the pet's video is recorded at 30 frames per second, and the user's video is recorded at the same time. Audio data is saved every three seconds. The sensor also measures the pet's temperature and heart rate every 30 seconds.

[1476] Step 3:

[1477] The device sends the collected data to a server via the Internet. The data is sent in real time, but if there are network problems, it is temporarily stored in the device.

[1478] Step 4:

[1479] The server converts the received data into an analyzable format, extracting specific frames from the video data and analyzing the tone and strength of the voice from the audio data. Biometric data is standardized and converted into an easily analyzable format.

[1480] Step 5:

[1481] The server uses a generative AI model to analyze the emotions of the pet and the user from pre-processed data, for example, by using the pet's tail movements and the tone of its cries, as well as the user's facial expressions and tone of voice.

[1482] Step 6:

[1483] The server comprehensively evaluates the emotional states of the pet and the user based on the results of emotion analysis. For example, if the pet is anxious and the user is also showing signs of impatience or anxiety, the server determines that special attention is required.

[1484] Step 7:

[1485] The server monitors your pet's health in real time, continuously analyzing food intake, water intake, waste data, temperature data, and heart rate data to identify any data points that are outside of normal ranges.

[1486] Step 8:

[1487] The server generates an alert when an abnormality is detected, and determines the specific content of the alert based on the pet's biological and emotional data, as well as the user's emotional data.

[1488] Step 9:

[1489] The server then sends the generated alert and the emotion analysis results to the user's device, including specific instructions such as "Your pet has a high temperature. Please take it to the vet immediately."

[1490] Step 10:

[1491] The device notifies the user of alerts and emotion analysis results received from the server either visually or audibly. If the notification is visual, it will be displayed on the screen, and if it is audible, it will be conveyed to the user through the voice assistant.

[1492] Step 11:

[1493] The user can check the notified information and take appropriate action depending on the pet's condition. For example, if they receive a fever alert, they can immediately take their pet to a veterinarian.

[1494] Step 12:

[1495] The device displays the pet and user's emotional analysis results and health data in graphs and text, allowing users to check past data and manage their pet's long-term health.

[1496] Step 13:

[1497] The device will provide real-time notifications through the voice assistant, for example, telling the user, "Your pet is hungry," and offering advice such as, "Your smile will make your pet feel at ease."

[1498] Example 2

[1499] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1500] Today, there is a need for a system that comprehensively monitors the health and emotional state of pets and their owners. However, existing systems only handle pet health and emotional data and are unable to perform comprehensive analysis that includes the owner's emotional state. Furthermore, abnormality detection and real-time notification tend to be delayed, making prediction and early response difficult. Therefore, there is a need for a systematic system that can more precisely monitor the health and emotional state of pets and their owners and respond quickly and appropriately when abnormalities occur.

[1501] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1502] In this invention, the server includes means for converting received data into an analyzable format and extracting facial and movement features, means for analyzing the tone and strength of voice from the audio data, and means for analyzing and estimating the emotions of the pet and the user from the preprocessed data using a generative AI model. This enables comprehensive analysis of the emotional and health states of the pet and the owner, and if an abnormality is detected, it is possible to quickly and accurately generate an alert and notify the user in real time.

[1503] A "terminal" is a device that starts a data collection mode by user operation, collects images, sounds, and biometric data of the pet and the user, and transmits the collected data to a server via the Internet.

[1504] "Camera means" refers to an optical sensor device for capturing image data of the pet and the user.

[1505] The "microphone means" is an acoustic sensor device for acquiring voice data of the pet and the user.

[1506] The "sensor means" is a biometric device for measuring the pet's biological data, specifically, body temperature, heart rate, etc.

[1507] The "data transmission means" is a communication function for transmitting data collected by the terminal to a server via a network.

[1508] "Data analysis means" refers to software and algorithms that convert data received on the server into an analyzable format, extract facial and movement features, and analyze voice tone and strength from audio data.

[1509] "Emotion Analysis Means" means software and algorithms for analyzing and inferring pet and user emotions from data preprocessed using a generative AI model.

[1510] The "abnormality detection means" is software and a system for monitoring the health of the pet and the user based on the analyzed emotional data and biometric data, and generating an alert if an abnormality is detected.

[1511] The "notification means" is a function for visually or audibly notifying the user of the generated alert or the results of emotion analysis.

[1512] The "user interface" is a display and operating environment that displays analysis results in text and graph format, and works in conjunction with a voice assistant to provide real-time notifications.

[1513] A "generative AI model" is an artificial intelligence model used to analyze pet and user emotions from preprocessed data, and includes, for example, GPT-3, image recognition models, and speech recognition models.

[1514] This invention is a system that comprehensively monitors the emotions and health status of pets, and also recognizes the emotions of the owner (user) to optimize the response. This allows the user to quickly and accurately grasp the condition of their pet and take the optimal response. This system is realized by the following components and their operations.

[1515] Components

[1516] 1. Terminal: This terminal is equipped with a camera means for acquiring image data of the pet, a microphone means for acquiring audio data, and a sensor means for acquiring biometric data. It also includes a camera means for acquiring image data of the user and a microphone means for acquiring audio data.

[1517] 2. Server: The server has a data analysis means for receiving and analyzing the acquired data, an emotion analysis means for recognizing and estimating the emotions of the pet and the user, and an anomaly detection means for detecting anomalies and generating alerts.

[1518] 3. User: Owners and pet caregivers use the system to monitor and be notified of their pets' and their own emotional states.

[1519] Program processing

[1520] Data collection

[1521] The device enters data collection mode when the user operates it, activating the camera, microphone, and sensors to collect images, audio, and biometric data of the pet, as well as the user's image and audio data.

[1522] Example: The device's camera captures video of the pet and the user at 30 frames per second, the microphone records audio from both parties at a sampling rate of 44.1 kHz, and the sensor measures the pet's temperature once per second and its heart rate in real time.

[1523] Data transmission

[1524] The device sends the collected data to a server via the Internet. The data is sent in real time, but if a network failure occurs, the data is temporarily stored in the device's internal storage and will be sent again once the network is restored.

[1525] Example: The terminal sends data to the server through the REST API using the HTTP protocol. For example, it sends a JSON formatted data packet every 30 seconds. The data sent also includes a timestamp to ensure data integrity.

[1526] Data analysis

[1527] The server converts the received data into an analyzable format, extracting facial and movement features from the video data and analyzing the tone and strength of the voice from the audio data. Biometric data is also converted into easy-to-read graphs and numbers.

[1528] Example: The server uses Python's OpenCV library to detect facial landmarks from video data. For audio data, the Librosa library is used to extract audio features and analyze the frequency spectrum. Biometric data is organized into a data frame using the Pandas library and visualized as time-series data.

[1529] Emotion analysis

[1530] The server uses a generative AI model to analyze the emotions of the pet and the user from preprocessed data. It analyzes the movements of the pet's tail and ears, the tone of its cries, and other factors to determine whether the pet is happy, and the user's facial expression and tone of voice to determine whether the owner is at ease.

[1531] Example: Using the generative AI model GPT-3, emotions are expressed in text format based on data obtained from image recognition models (e.g., ResNet) and speech recognition models (e.g., DeepSpeech). For example, the most appropriate emotion for a given situation can be estimated based on the width and frequency of a pet's tail wagging or the degree of a user's smile.

[1532] Anomaly detection

[1533] The server monitors the pet's health based on the analysis of its emotional and biological data, and generates an alert if an abnormality is detected. The alert is generated taking into account the pet's emotional state.

[1534] Example: The server sets a threshold based on the analysis results and detects abnormalities. For example, if the pet's body temperature exceeds 39 degrees or the heart rate exceeds 200 bpm, an alert will be generated stating "possible high fever." It will also alert the user if the emotional state of the pet and the user do not match.

[1535] notification

[1536] The server sends the generated alerts and the results of emotion analysis to the user's device.

[1537] The terminal notifies the user visually or audibly.

[1538] Example: The server uses WebSocket or Push Notification services to send notifications to the device in real time. The device displays a message on the screen saying "Your pet's temperature is high. Please take action immediately" and uses a Text-to-Speech (TTS) engine to notify the user by voice, "Your pet is feeling anxious."

[1539] Interface and voice assistant integration

[1540] The device provides a user interface that displays the pet and user's emotional analysis and health data in graphs and text, and also provides real-time notifications via a voice assistant.

[1541] Example: The device uses front-end frameworks such as React and Vue.js to visualize the analysis results in the browser. It also works with a voice assistant to notify users in real time that their pet is hungry, while simultaneously providing advice such as, "Your smile will make your pet feel at ease."

[1542] Prompt Sentence Examples

[1543] Below are some examples of prompt sentences to input to the generative AI model.

[1544] Prompt Sentence Examples

[1545] Analyze the health and emotions of your pet, as well as your emotions. The following data will be collected: pet video, audio, and biometric data, and user video and audio data.

[1546] This system allows users to closely monitor their pets' emotions and health. By analyzing the collected data and providing appropriate alerts and notifications based on the results, it is possible to detect and respond to abnormalities in both pets and users early on. This makes it a useful tool for helping both pets and users live happy and healthy lives.

[1547] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1548] Step 1: Data collection

[1549] The device starts data collection mode when the user operates it. The camera (image data), microphone (audio data), and sensor (biometric data) are activated to collect images, audio, and biometric data of the pet. At the same time, the user's image and audio data are also collected.

[1550] Input: The user operates the terminal to start data collection mode.

[1551] Output: A set of video data, audio data, and biometric data collected by the device.

[1552] How it works: The device's camera captures video of the pet and the user at 30 frames per second, the microphone records audio from both parties at 44.1kHz, and the sensor measures the pet's temperature every second and heart rate in real time.

[1553] Step 2: Send data

[1554] The terminal transmits the collected data to a server via the Internet in real time, but in the event of a network failure, the data is temporarily stored in the terminal.

[1555] Input: A set of collected image, audio, and biometric data.

[1556] Output: A set of image data, audio data, and biometric data sent to the server.

[1557] Specific operation: The terminal uses the HTTP protocol to send data to the server via the REST API every 30 seconds. The data is in JSON format and each data item is time-stamped.

[1558] Step 3: Data analysis

[1559] The server converts the received data into an analyzable format, extracting facial and movement features from the video data and analyzing the tone and strength of the voice from the audio data. The biometric data is then organized into graphs and numerical values.

[1560] Input: A set of image data, audio data, and biometric data sent to the server.

[1561] Output: Analyzed facial and movement characteristics, voice tone and strength, and organized biometric data.

[1562] Specific operation: The server uses Python's OpenCV library to detect facial landmarks from video data, and the Librosa library to extract audio features. The biometric data is organized into a data frame using the Pandas library and visualized.

[1563] Step 4: Sentiment Analysis

[1564] The server uses a generative AI model to analyze the emotions of the pet and the user from the pre-processed data.

[1565] Input: Analyzed facial and movement characteristics, voice tone and volume, and organized biometric data.

[1566] Output: Estimated pet and user emotions.

[1567] Specific operation: The server uses image recognition models (e.g., ResNet) and speech recognition models (e.g., DeepSpeech) as generative AI models to express emotions from data in text format. For example, by linking the behavior of a pet wagging its tail with the user's smile, it can infer that "the pet is happy" and "the owner is at ease."

[1568] Step 5: Anomaly detection

[1569] The server monitors the pet's health based on the analysis of emotional and biometric data, and generates an alert if an abnormality is detected.

[1570] Input: Estimated pet and user emotions, and organised biometric data.

[1571] Output: The generated alert.

[1572] Specific operation: The server sets a threshold and generates an alert saying "possible high fever" if, for example, the pet's body temperature exceeds 39 degrees or the heart rate exceeds 200 bpm. If an anxious expression on the user's face is also detected at the same time, the server issues a warning.

[1573] Step 6: Notification

[1574] The server sends the generated alerts and the results of emotion analysis to the user's device.

[1575] The terminal notifies the user visually or audibly.

[1576] Input: Generated alerts, sentiment analysis results.

[1577] Output: A visual or audio notification.

[1578] Specific operation: The server sends a real-time notification to the device using WebSocket or Push Notification services. The device displays the message "Your pet's temperature is high. Please take action immediately" on the screen and uses a Text-to-Speech (TTS) engine to notify the user by voice, "Your pet is anxious."

[1579] Step 7: Interface and Voice Assistant Integration

[1580] The terminal provides a user interface that displays the emotional analysis results and health data of the pet and the user in graphs and text.

[1581] The device will provide real-time notifications via the voice assistant.

[1582] Input: Sentiment analysis results, health data, generated alerts.

[1583] Output: Data displayed in graphs and text, notifications via voice assistant.

[1584] How it works: The device visualizes the analysis results in a browser using front-end frameworks such as React and Vue.js. It also works with a voice assistant to notify users in real time that their pet is hungry, while simultaneously providing advice such as, "Your smile will make your pet feel at ease."

[1585] (Application example 2)

[1586] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1587] Conventional systems exist that comprehensively monitor a pet's emotions and health status, and also recognize the owner's emotions to optimize responses. However, a system that utilizes such a system in an autonomous vehicle to monitor the status of the pet and owner in real time and automatically optimize the in-vehicle environment has not yet been realized. This has made it difficult for owners to grasp the pet's condition in detail and respond appropriately while on the move. The present invention aims to solve this problem.

[1588] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a data analysis means for analyzing acquired data, an emotion analysis means for estimating the pet's emotion and health state, an abnormality detection means for detecting an abnormality based on the analysis result and generating an alarm, and an environment adjustment means for optimizing the in-vehicle environment. This makes it possible to monitor the status of the pet and its owner in real time inside the autonomously driving vehicle and automatically adjust the in-vehicle environment based on the analysis result.

[1589] The "photography means" is a device for acquiring image data of a pet.

[1590] The "recording means" is a device for acquiring voice data of a pet.

[1591] "Measuring means" refers to a device for acquiring biological data of a pet.

[1592] The "emotion analysis means" is a device that analyzes the acquired image data, audio data, and biometric data to estimate the pet's emotions.

[1593] The "data transmission means" is a device for transmitting the acquired data to the server.

[1594] The "data analysis means" is a device for analyzing the emotions and health condition of pets on the server.

[1595] The "abnormality detection means" is a device that detects abnormalities in the pet based on the analysis results and generates an alarm.

[1596] The "notification means" is a device for notifying the user of the generated alert.

[1597] The "environment adjustment means" is a device that monitors the condition of pets and their owners inside an autonomous vehicle and optimizes the in-vehicle environment based on the analysis results.

[1598] This invention relates to a system that monitors the status of pets and their owners in an autonomous vehicle and optimizes the in-vehicle environment. This system is realized by a number of components and their cooperation.

[1599] Components

[1600] 1. Photography method: Image data of the pet is obtained using a camera installed in the vehicle.

[1601] Specifically, a general webcam (e.g., Logitech C920) is used.

[1602] 2. Recording method: Record your pet's voice using a microphone in the vehicle.

[1603] Specifically, a high-performance microphone (e.g., Blue Yeti) is used.

[1604] 3. Measurement methods: Use sensors to obtain your pet's biological data.

[1605] Specifically, biosensors for pets (e.g., Fitbit pet sensors) are used.

[1606] 4. Emotion analysis method: Analyze the acquired data to estimate the pet's emotions.

[1607] Specifically, video and audio data are analyzed using an AI model (e.g., a generative AI model using Keras).

[1608] 5. Data transmission means: Transmits the acquired data to the server.

[1609] In particular, a communication device is utilized to transmit data over a network connection.

[1610] 6. Data analysis means: The data received on the server is analyzed to determine the pet's emotions and health condition.

[1611] Specifically, data analysis software (e.g., Python scripts) that runs on the server side is used.

[1612] 7. Anomaly detection means: Detects anomalies based on the analysis results and generates an alert.

[1613] The server generates an alert if it detects an abnormality in the health status.

[1614] 8. Notification means: Notifies the user of the generated alert.

[1615] Specifically, audio notifications and screen displays are used.

[1616] 9. Environmental control means: A device for optimizing the in-vehicle environment.

[1617] This includes devices that automatically control temperature, lighting, music, etc. inside the vehicle.

[1618] Explaining program processing in natural language

[1619] The server first collects data through photography, audio recording, and measurement. It then transmits this data in real time to the server. The server processes the data using a generative AI model using Keras to analyze the emotions and health of the pet and owner. Based on the analysis results, it then generates an alert if necessary and notifies the user. At the same time as the notification, the vehicle's environmental control measures are used to optimize the temperature, lighting, music, and other aspects of the vehicle interior.

[1620] Specific examples

[1621] For example, a camera inside the car captures video of the pet and its owner, and a microphone records audio. Measurements are made to measure the pet's heart rate and body temperature. This data is sent to a server in real time, and a generative AI model using Keras analyzes the data. If the server detects that the pet is in an unstable state, it generates an alarm and notifies the user with an audio message saying, "Your pet is in an unstable state," while simultaneously adjusting the in-car environment to make it more comfortable for the pet.

[1622] Prompt Sentence Examples

[1623] "Measure the pet's temperature and analyze the video data captured by the camera. Also, analyze the owner's voice data and infer the emotions of both parties. Based on the analysis results, play appropriate music."

[1624] In this way, an environment can be provided in which both pets and owners can move around with peace of mind.

[1625] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1626] Step 1:

[1627] The terminal activates the photographing means, recording means, and measuring means to collect data on the pet and its owner. Image data acquired by the photographing means, audio data acquired by the recording means, and biometric data acquired by the measuring means (e.g., the pet's body temperature and heart rate) are collected in real time. Input data includes image data, audio data, and biometric data.

[1628] Step 2:

[1629] The terminal temporarily stores the collected data and then transmits it to the server via the network. At this time, transmission may be temporarily delayed depending on the network status, but this data transmission method ensures stable transmission. The input is the image data, voice data, and biometric data collected in step 1, and the output is that these data are transmitted to the server.

[1630] Step 3:

[1631] The server converts the received image data into a processable format. Specifically, this involves extracting facial and movement features of the pet and owner from the image data. This process uses an image processing library such as OpenCV. The input is the transmitted image data, and the output is the data with the extracted features.

[1632] Step 4:

[1633] The server analyzes the received voice data and analyzes the tone and stress of the voice. This process uses a voice analysis library such as Librosa. The input is the transmitted voice data, and the output is the analyzed voice feature data.

[1634] Step 5:

[1635] The server organizes and analyzes the received biometric data, including converting the pet's heart rate and temperature data into an easily readable format. The input is the transmitted biometric data, and the output is the analyzed biometric data.

[1636] Step 6:

[1637] The server uses emotion analysis means to analyze the emotions of the pet and owner from the preprocessed image data, audio data, and biometric data based on the generative AI model. For example, it analyzes the pet's tail movements and the tone of its cries, as well as the owner's facial expressions and tone of voice. The input is the feature-extracted data and analyzed audio and biometric data, and the output is an estimation of the emotional state of the pet and owner.

[1638] Step 7:

[1639] The server monitors the pet's health using anomaly detection means based on the analysis results. If an abnormality is detected, an emergency alert is generated. For example, if the pet's body temperature is high and the user looks anxious, an alert is generated. The input is the emotion analysis result and biometric data, and the output is the alert generation.

[1640] Step 8:

[1641] The device notifies the user of the generated alert and the results of the emotion analysis by audio or visual notification. For example, the device may display "Your pet's temperature is high. Please take action immediately" on the screen and simultaneously notify the user by audio notification, "Your pet is feeling anxious." The input is the alert and the results of the emotion analysis, and the output is the notification to the user.

[1642] Step 9:

[1643] The device adjusts the in-car environment in real time, optimizing the temperature, lighting, music, etc. based on the analysis results. For example, if the pet is relaxed and the owner feels at ease, relaxing music will be played and the lighting will be softened. The input is the results of emotion analysis and health status, and the output is the adjustment of the in-car environment.

[1644] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1645] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1646] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1647] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1648] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1649] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1650] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1651] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1652] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1653] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1654] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1655] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1656] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1657] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1658] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1659] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1660] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1661] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1662] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1663] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1664] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1665] The following is further disclosed regarding the above embodiment.

[1666] (Claim 1)

[1667] camera means for acquiring image data of the pet;

[1668] a microphone means for acquiring voice data of the pet;

[1669] a sensor means for acquiring biometric data of the pet;

[1670] emotion analysis means for analyzing the acquired image data, audio data, and biological data to estimate the emotions of the pet;

[1671] data transmission means for transmitting the acquired data to a server;

[1672] A data analysis method that analyzes the pet's emotions and health status on the server,

[1673] an anomaly detection means for detecting an abnormality in the pet based on the analysis result and generating an alert;

[1674] a notification means for notifying a user of the generated alert;

[1675] A system including:

[1676] (Claim 2)

[1677] The system according to claim 1, wherein the data analysis means analyzes the pet's health condition based on the amount of food eaten, the amount of water consumed, excrement data, body temperature data, and heart rate data, and generates an alert if an abnormality is detected.

[1678] (Claim 3)

[1679] 10. The system of claim 1, wherein the notification means provides the user with real-time audio notifications regarding the pet's current emotional and health state.

[1680] "Example 1"

[1681] (Claim 1)

[1682] camera means for acquiring image data of the pet;

[1683] a microphone means for acquiring voice data of the pet;

[1684] a sensor means for acquiring biometric data of the pet;

[1685] emotion analysis means for analyzing the acquired image data, audio data, and biological data to estimate the emotions of the pet;

[1686] A data analysis method for standardizing the collected data and analyzing it using a generative AI model;

[1687] data transmission means for transmitting the acquired data to a server;

[1688] An anomaly detection means that analyzes the pet's emotions and health condition on the server and detects abnormalities;

[1689] a notification means for generating an alert and notifying a user when an abnormality is detected;

[1690] A means for displaying generated alerts and analysis results in a user interface;

[1691] a voice notification means for providing real-time notifications by voice using a voice assistant;

[1692] A system including:

[1693] (Claim 2)

[1694] The system according to claim 1, wherein the data analysis means analyzes the pet's health condition based on the amount of food eaten, the amount of water consumed, excrement data, body temperature data, and heart rate data, and generates an alert if an abnormality is detected.

[1695] (Claim 3)

[1696] 10. The system of claim 1, wherein the notification means provides the user with real-time audio notifications regarding the pet's current emotional and health state.

[1697] "Application Example 1"

[1698] (Claim 1)

[1699] camera means for acquiring image data of the pet or the working environment;

[1700] a microphone means for acquiring audio data of the pet or the working environment;

[1701] sensor means for acquiring biometric data of a pet or a worker;

[1702] emotion analysis means for analyzing the acquired image data, audio data, and biological data to estimate the emotion of the pet or worker;

[1703] data transmission means for transmitting the acquired data to a server;

[1704] a data analysis means for analyzing the emotions and health conditions of the pet or the worker on the server;

[1705] an anomaly detection means for detecting an abnormality in a pet or a worker based on the analysis result and generating an alert;

[1706] a notification means for notifying a user of the generated alert;

[1707] A system including:

[1708] (Claim 2)

[1709] The system of claim 1, wherein the data analysis means analyzes the health condition of the pet or working environment based on food intake, water intake, excrement data, body temperature data, and heart rate data, and generates an alert if an abnormality is detected.

[1710] (Claim 3)

[1711] 10. The system of claim 1, wherein the notification means provides the user with real-time audio notifications regarding the current emotional and health state of the pet or working environment.

[1712] "Example 2: Combining Emotion Engines"

[1713] (Claim 1)

[1714] a means for starting a data collection mode by a user's operation on the terminal and collecting images, voice, and biological data of the pet and image and voice data of the user;

[1715] means for transmitting the collected image data, voice data, and biometric data to a server via the Internet;

[1716] A means for converting the data received on the server into an analyzable format, extracting facial and movement features, and analyzing the tone and strength of the voice from the audio data;

[1717] A means for analyzing and inferring pet and user emotions from the preprocessed data using a generative AI model; and

[1718] A means for detecting anomalies based on the analyzed emotional data and biometric data and generating an alert;

[1719] A means for visually or audibly notifying the user of the generated alert or the result of the sentiment analysis;

[1720] A means of collecting data, analyzing it, notifying it, and displaying all the results on the user interface, and linking it with a voice assistant to provide real-time notifications.

[1721] A system including:

[1722] (Claim 2)

[1723] The system of claim 1, wherein the data analysis means analyzes the pet's health condition based on the amount of food, water intake, excrement data, body temperature data, and heart rate data collected by the terminal's sensors, and generates an alert if an abnormality is detected.

[1724] (Claim 3)

[1725] 10. The system of claim 1, wherein the notification means provides real-time notifications to the user via the voice assistant regarding the current emotional and health states of the pet and the user.

[1726] "Application example 2 when combining emotion engines"

[1727] (Claim 1)

[1728] a photographing means for acquiring image data of the pet;

[1729] a recording means for acquiring voice data of the pet;

[1730] a measuring means for acquiring biological data of the pet;

[1731] emotion analysis means for analyzing the acquired image data, audio data, and biological data to estimate the emotions of the pet;

[1732] data transmission means for transmitting the acquired data to a server;

[1733] A data analysis method that analyzes the pet's emotions and health status on the server,

[1734] an abnormality detection means for detecting an abnormality in the pet based on the analysis result and generating an alarm;

[1735] a notification means for notifying a user of the generated warning;

[1736] an environmental adjustment means for monitoring the state of the pet and the owner in the self-driving vehicle and optimizing the in-vehicle environment;

[1737] A system including:

[1738] (Claim 2)

[1739] The system according to claim 1, wherein the data analysis means analyzes the pet's health condition based on the amount of food eaten, the amount of water consumed, excrement data, body temperature data, and heart rate data, and generates an alarm if an abnormality is detected.

[1740] (Claim 3)

[1741] 10. The system of claim 1, wherein the notification means provides real-time notifications to the user by voice regarding the pet's current emotional and health state. [Explanation of symbols]

[1742] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. camera means for acquiring image data of the pet; a microphone means for acquiring voice data of the pet; a sensor means for acquiring biometric data of the pet; emotion analysis means for analyzing the acquired image data, audio data, and biological data to estimate the emotions of the pet; data transmission means for transmitting the acquired data to a server; A data analysis method that analyzes the pet's emotions and health status on the server, an anomaly detection means for detecting an abnormality in the pet based on the analysis result and generating an alert; a notification means for notifying a user of the generated alert; A system including:

2. The system according to claim 1, wherein the data analysis means analyzes the pet's health condition based on the amount of food eaten, the amount of water consumed, excrement data, body temperature data, and heart rate data, and generates an alert if an abnormality is detected.

3. 10. The system of claim 1, wherein the notification means provides the user with real-time audio notifications regarding the pet's current emotional and health state.

Images