system

Abstract

We provide the system. [Solution] A data collection means for collecting audio and video data of pets, An analytical means for analyzing the behavior and health status of pets based on collected audio and video data, An alert generation means that notifies the user when an anomaly is detected based on the analysis results, A means of accumulating data on the health status of pets and displaying it in a way that users can check, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, the method including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Pet owners often find it difficult to grasp the daily health status of their pets and are often unable to respond immediately when abnormalities occur. Therefore, in order to maintain the health of pets and prevent diseases, it is necessary to grasp the behavior and physical condition of pets and respond promptly when abnormalities occur. However, at present, the methods and tools for effectively achieving this are limited. Therefore, it is desired to provide a system that supports pet health management by analyzing daily video and audio data of pets and detecting and alerting abnormal behaviors and health risks in real time.

Means for Solving the Problems

[0005] This invention provides a system comprising: data collection means for collecting audio and video data of pets; analysis means for analyzing the behavior and health status of pets based on the collected audio and video data; alert generation means for notifying the user when an abnormality is detected based on the analysis results; and visualization means for accumulating data on the health status of pets and displaying it in a format that can be viewed by the user. This enables early detection of abnormalities by analyzing the behavior and vocalization patterns of pets in real time, allowing users to immediately understand the health status of their pets. As a result, it is possible to reduce health risks for pets and improve the owner's peace of mind.

[0006] "Data collection means" refers to devices and methods for collecting audio and video data obtained from pets.

[0007] "Analysis means" refers to a device and method for analyzing a pet's behavior and health status using collected audio and video data.

[0008] An "alert generation means" is a device and method for notifying a user when an anomaly is detected by an analysis means.

[0009] "Visualization means" refers to a device and method for accumulating data on a pet's health status and presenting it visually to the user.

[0010] A "system" is a combination of a series of processes and devices that include data collection means, analysis means, alert generation means, and visualization means. [Brief explanation of the drawing]

[0011] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3]This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14] This is a sequence diagram showing the processing flow of the data processing system in Application Example 2, which combines an emotion engine. [Modes for carrying out the invention]

[0012] Hereinafter, an example of an embodiment of the system relating to the technology of this disclosure will be described with reference to the attached drawings.

[0013] First, let's explain the terminology used in the following explanation.

[0014] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of a plurality of arithmetic units. Also, they may be a single type of arithmetic unit or a combination of a plurality of types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0015] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0016] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs, various parameters, and the like. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, and the like.

[0017] In the following embodiments, the numbered communication I / F (Interface) is an interface including a communication processor, an antenna, and the like. The communication I / F controls communication between a plurality of computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0018] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0019] [First Embodiment]

[0020] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0021] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0022] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0023] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0024] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0025] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0026] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0027] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0028] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0029] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0030] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0031] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0032] This invention provides a system for monitoring a pet's health in real time and immediately notifying the user if an abnormality is detected. The system is implemented as follows:

[0033] Data collection

[0034] First, the device uses a fixed camera and microphone installed in the pet's living space to acquire video and audio data of the pet in real time. This data is designed to comprehensively cover the pet's daily activities.

[0035] Data transmission and analysis

[0036] Data collected by the device is sent to a server via the internet. The server runs machine learning algorithms to analyze the received data. This analysis extracts characteristics of the pet's walking patterns and vocalizations. This allows the system to determine if the pet is behaving unusually or may be experiencing stress.

[0037] Anomaly detection and notification

[0038] Based on the analysis results, the server generates an alert when it detects an anomaly. The alert is sent to the user's mobile device, and the user can open the app to view detailed information. This allows the user to quickly understand their pet's health condition and take necessary action.

[0039] Recording and Visualizing Health Status

[0040] The server stores pet health data over long periods and visualizes the data in a format accessible to users. By comparing current data with past data, users can identify trends in their pet's health and implement preventative health management. This visualization function is provided in an easy-to-understand format using graphs and charts.

[0041] For example, if a pet barks more frequently than usual, the device captures the audio data and sends it to a server. The server analyzes the audio pattern and, if it determines that the sound is an abnormal sound indicating stress, sends an alert to the user stating, "Your pet may be experiencing stress." The user can then check the details in the app, observe their pet's behavior, and take appropriate measures to reassure them if necessary.

[0042] The following describes the processing flow.

[0043] Step 1:

[0044] The device uses a fixed camera and microphone installed in the pet's living space to capture video and audio of the pet in real time. This allows for the accumulation of data on the pet's behavior and vocalizations.

[0045] Step 2:

[0046] The terminal transmits the collected video and audio data to the server via the internet. Here, the data format is standardized to maintain data continuity and accuracy.

[0047] Step 3:

[0048] The server analyzes the received video data frame by frame. Specifically, it uses image recognition technology to identify the pet's silhouette and movements, and to determine any differences from normal behavior patterns.

[0049] Step 4:

[0050] The server performs frequency spectrum analysis on the audio data to extract the characteristics of pet noises. This is then compared to an existing audio database to detect abnormal patterns and frequencies.

[0051] Step 5:

[0052] Based on the analysis results, the server generates an alert if an anomaly is detected. It then prepares to send the alert via push notification or email, according to the user's settings.

[0053] Step 6:

[0054] The server sends the generated alert to the user's terminal. The alert includes details of the detected anomaly and predictions about possible causes.

[0055] Step 7:

[0056] When a user receives an alert, they open the app to view the details. At this stage, the app helps the user take appropriate action quickly by providing recommended actions and health management information.

[0057] Step 8:

[0058] The server stores pet health data chronologically, managing long-term records of their health status. Users can access this data to review past records and understand their pet's health trends.

[0059] (Example 1)

[0060] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0061] Managing a pet's health is a crucial issue for pet owners, requiring them to respond quickly when their pet experiences stress or illness. However, it is difficult for owners to constantly monitor their pet's condition, and abnormalities may go unnoticed for a long time. Traditional methods have not provided an effective solution to this problem.

[0062] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0063] In this invention, the server includes acquisition means for acquiring acoustic and video information of a pet, analysis means for analyzing the pet's behavior and health status based on the acquired acoustic and video information, and notification generation means for notifying the user if an abnormality is detected based on the analysis results. This allows the user to monitor the pet's health status in real time and immediately identify any abnormalities.

[0064] "Means of acquisition" refers to a system for collecting acoustic and visual information about pets.

[0065] "Analysis means" refers to a system for analyzing a pet's behavior and health condition based on acquired acoustic and video information.

[0066] A "notification generation mechanism" refers to a system that informs users of the details of an anomaly detected as a result of analysis.

[0067] "Display means" refers to a system for providing information about a pet's health status in a way that users can verify.

[0068] A "fixed recording device" refers to cameras and other equipment installed to acquire video information of pets in real time.

[0069] "Acoustic equipment" refers to microphones and other devices used to acquire acoustic information about pets in real time.

[0070] "Machine learning techniques" refer to algorithms that extract features and patterns from data to determine a pet's behavior and health status.

[0071] The system of this invention monitors the health status of pets in real time and promptly notifies the user if an abnormality is detected.

[0072] System Configuration

[0073] The terminal uses a fixed camera and sound system installed in the pet's living space to acquire real-time audio and video information of the pet. This data is temporarily stored by the terminal and transmitted to a server via the internet.

[0074] Data Analysis

[0075] The server analyzes the received audio and video information using machine learning techniques. This analysis extracts information about the pet's behavior and health status, and detects anomalies. The server can improve the accuracy of its analysis by using software libraries such as TENSORFLOW® and PyTorch.

[0076] Notifications and displays

[0077] If an anomaly is detected during the analysis, the server will send an alert to the user's device using a notification generation mechanism. The user can then open the application on their mobile device to check the pet's detailed health status and the nature of the anomaly. Historical data is stored on the server and visualized using interactive graphs and charts, allowing users to easily understand their pet's health trends.

[0078] Specific example

[0079] As a concrete example, consider a case where a pet barks more frequently than usual. The device captures the sound data and sends it to a server. The server analyzes the sound pattern and, if it determines that the sound is an abnormal sound indicating stress, sends an alert to the user saying, "Your pet may be stressed." The user can then check the details in the app, observe their pet's behavior, and take appropriate measures to reassure them.

[0080] Examples of input prompts for a generative AI model

[0081] "Please explain the mechanism that generates an alert when unusual behavior in a pet is detected."

[0082] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0083] Step 1:

[0084] The device uses a fixed camera and audio system to acquire real-time audio and video information of the pet. The camera captures high-resolution video, and the microphone collects ambient sound data. The input is the pet's movements and vocalizations, and the output is digital data of these sounds. This allows for detailed recording of the pet's daily activities.

[0085] Step 2:

[0086] The terminal temporarily stores the acquired audio and video information in memory and then transmits it to the server via the internet. The input is digital data stored in the terminal, which is compressed before transmission. The output is the compressed data transferred to the server. Error checking is performed during this process to maintain data consistency.

[0087] Step 3:

[0088] The server decodes the received audio and video information and prepares it for analysis. The input is compressed digital data, and the decoded raw data is output. Here, data preprocessing such as noise reduction and normalization is performed.

[0089] Step 4:

[0090] The server uses machine learning techniques to analyze raw data. The input is pre-processed data, from which features related to pet behavior patterns and health status are extracted. The output is a judgment result regarding the pet's behavior and health status. Software such as TensorFlow and PyTorch are used for the analysis.

[0091] Step 5:

[0092] If the server detects an anomaly based on the analysis results, it sends an alert to the user's terminal using a notification generation mechanism. The input is the analysis result, and the output is an alert message. The notification is sent in real time, prompting the user to take immediate action.

[0093] Step 6:

[0094] The user opens the app on their mobile device to view the notification. The input is an alert message sent from the server, and the output is detailed information about the pet's health status and any abnormalities. This allows the user to take necessary actions quickly.

[0095] (Application Example 1)

[0096] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0097] Conventional animal health monitoring systems struggle to monitor animals' health in real time, potentially leading to missed abnormalities. Furthermore, they lack the means to prompt human intervention quickly when abnormalities are detected. There is also a need to effectively accumulate detailed data on animals' health over long periods and visualize it in a user-friendly format.

[0098] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0099] In this invention, the server includes data acquisition means for collecting acoustic and visual information to monitor the animal's condition in real time; an analysis mechanism for evaluating the animal's behavioral characteristics and health status based on the collected acoustic and visual information; a warning generation means for generating a notification to the user when an abnormality is detected based on the analysis results; and a display device linkage means for displaying the abnormality notification on a visual information display device worn by a person. This makes it possible to quickly and accurately detect abnormalities in animals and prompt immediate human intervention.

[0100] A "data acquisition means" is a mechanism that collects acoustic and visual information from animals in real time and provides it for necessary analysis.

[0101] An "analysis mechanism" is a system that evaluates the behavioral characteristics and health status of animals based on collected acoustic and visual information, and determines whether or not there are any abnormalities.

[0102] A "warning generation mechanism" is a system that, based on analysis results, quickly generates notifications to users when an anomaly is detected and appropriately transmits that information.

[0103] "Visualization means" refers to a device or software for storing information about an animal's health status and displaying the data in a format that is easily understandable to users.

[0104] The "display device linkage means" is an interface means that presents abnormality notifications to a visual information display device worn by a person, enabling the user to track the information in real time.

[0105] The system implementing this invention enables real-time monitoring of an animal's health status and rapid notification if an abnormality is detected. The system consists of the following elements:

[0106] First, the terminal uses fixed imaging and acoustic devices installed in the animal's living space to collect acoustic and visual information about the animal in real time. The aim of this data is to comprehensively cover the animal's living environment.

[0107] Next, the data collected by the terminal is transmitted to a server via the internet. The server has a powerful analytical mechanism that evaluates the animal's behavioral characteristics and health status based on the collected acoustic and visual information. This process involves using machine learning algorithms to identify the animal's movement patterns and vocal characteristics and to analyze data to detect any abnormal conditions.

[0108] If the server detects an anomaly, it uses a warning generation mechanism to quickly generate a notification to the user, which is then displayed on a visual information display device worn by the user. This allows the user to check the animal's condition in real time and take necessary actions. In addition, the visualization mechanism accumulates data on the animal's health status and displays it to the user in graph and chart format, allowing for a clear understanding of health trends by comparing it with past data.

[0109] For example, if a pet parakeet chirps more frequently than usual, the system captures and analyzes the sound, and if an abnormal pattern is detected, it notifies the user. The user can then check the details through the application and take steps to observe their pet and reassure it.

[0110] Examples of prompts to be input into the generating AI model include, "Design a system that monitors the health status of animals in real time, detects abnormalities, and provides rapid notification."

[0111] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0112] Step 1:

[0113] The terminal uses imaging and acoustic devices installed in the animal's living space to acquire acoustic and visual information of the animal in real time. The input is raw image and sound data collected by the imaging and acoustic devices, and the output is digital data converted into a format that the system can process. In terms of operation, these devices are constantly running, maintaining a state of continuous monitoring of the environment.

[0114] Step 2:

[0115] The terminal transmits the collected digital data to the server via the internet. The input is the digital data obtained in step 1, and the output is confirmation information indicating that the secure data transfer to the server was successful. The data is transmitted securely using security protocols.

[0116] Step 3:

[0117] The server analyzes the received digital data. The input is digital video and audio data transferred from the terminal, and the output is an evaluation result regarding the animal's behavioral characteristics and health status. Machine learning algorithms are used for data analysis to identify the animal's movement patterns and vocal characteristics.

[0118] Step 4:

[0119] Based on the analysis results, the server generates a warning if an anomaly is detected and notifies the user. The input is the evaluation result obtained in step 3, and the output is the warning content depending on whether an anomaly was detected or not. The warning is generated as a digital message and sent to a visual information display device worn by the user.

[0120] Step 5:

[0121] The user receives and confirms warning messages via a visual information display device they wear. The input is the warning message generated in step 4, and the output is the user's informed action. The user can understand the animal's condition in detail in real time and take prompt action as needed.

[0122] Step 6:

[0123] The server stores information on the health status of animals and provides it to users using visualization tools. Input is historical and current analytical data, and output is health trend data visualized in graphs and charts. This helps users understand long-term trends in animal health and facilitates preventative health management.

[0124] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0125] This invention relates to a system for monitoring the health status of pets and providing information tailored to the user's emotional state. This system combines a series of processes—collecting and analyzing pet audio and video data, detecting and notifying of abnormalities based on the results—with an emotion engine that recognizes the user's emotions. The system is implemented as follows:

[0126] Data collection and analysis

[0127] The device uses a fixed camera and microphone to collect video and audio data in real time within the pet's living space. This data is immediately sent to a server, which uses machine learning algorithms to analyze the pet's behavior patterns and health status. For example, it can detect irregular gait or an unusual frequency of barking.

[0128] User emotion recognition by an emotion engine

[0129] The server activates an emotion engine through the user's voice input. This engine identifies emotions such as joy, sadness, and surprise from the user's voice and further evaluates the intensity of those emotions. Based on the emotional state detected by the emotion engine, the system can adjust the information presented to the user.

[0130] Anomaly detection and alert notifications

[0131] If the server detects an abnormality in the pet, it notifies the user of the analysis results. Taking into account the output of the emotion engine, the server optimizes the content and tone of the message included in the alert based on the user's current emotional state. For example, if the user is feeling stressed, the notification regarding the pet's health will be reassuring.

[0132] Visualization of health status and emotions

[0133] The server continuously collects pet health data and user emotional history, and visualizes it in a user-accessible format. This allows users to understand not only their pet's health trends but also changes in their own emotions. This feature provides users with information to help them better interact with their pets.

[0134] For example, if a pet barks at an unusually high frequency, the device captures the audio data and sends it to a server for analysis. If the server detects an anomaly and determines from the user's voice that they are feeling anxious, it will notify the user with a message such as, "Your pet seems a little stressed. Spending some time together might help them feel more at ease." This allows the user to receive information that takes into account not only the pet's condition but also their own emotions.

[0135] The following describes the processing flow.

[0136] Step 1:

[0137] The device uses a fixed camera and microphone to collect video and audio of the space where the pet is located in real time. This data, including the pet's behavior and vocalizations, is continuously recorded.

[0138] Step 2:

[0139] The terminal transmits the collected data to the server via the internet. The data is immediately converted into a format that can be processed for real-time analysis.

[0140] Step 3:

[0141] The server breaks down the video data frame by frame and uses image recognition technology to analyze the pet's movements. It detects the pet's posture and movement patterns and compares them to normal behavioral patterns.

[0142] Step 4:

[0143] The server performs frequency spectrum analysis on the audio data to extract the characteristics of the pet's barks. It then compares the barking patterns with a database of normal barks to detect abnormal patterns.

[0144] Step 5:

[0145] The server analyzes the voice data acquired from the user's terminal using an emotion engine. This engine identifies the user's emotional state, such as joy or anxiety.

[0146] Step 6:

[0147] The server comprehensively evaluates the pet's analysis results and the user's emotional state, and generates an alert if an anomaly is detected. This alert is adjusted to take the user's emotional state into consideration and optimized for appropriate message content.

[0148] Step 7:

[0149] An alert is sent to the user's device, allowing them to open the application and view detailed information about their pet's health. This information is presented in a tone that matches the user's emotional state.

[0150] Step 8:

[0151] The server stores pet health data and user emotional history in a database and provides it to users in a visualized format that they can access. This visualization allows users to comprehensively analyze their pet's health trends and changes in their own emotions.

[0152] (Example 2)

[0153] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0154] Efficiently monitoring a pet's health while providing information that considers the owner's emotional state is a challenging task. Current technology allows for systems that simply detect and notify of health abnormalities in pets, but they cannot appropriately adjust the information content to reflect the user's emotional state. Therefore, there is a need to establish a comprehensive information provision system that enables owners to understand their pet's health and take appropriate action.

[0155] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0156] In this invention, the server includes information gathering means for collecting audio and video information of pets, analysis means for analyzing the pet's behavior and health status based on the collected audio and video information, and emotion recognition means for identifying the user's emotional state and adjusting the content of the information provided according to the identification result. This makes it possible to monitor the pet's health status in real time and provide appropriate information according to the user's emotions.

[0157] "Information gathering means" refers to devices and methods for collecting audio and video information about pets.

[0158] "Analysis means" refers to devices and methods for analyzing a pet's behavior and health condition based on collected audio and video information.

[0159] "Alert generation means" refers to a device or method for notifying the user of information when an anomaly is detected in the analysis results.

[0160] "Emotion recognition means" refers to devices or methods for identifying a user's emotional state from input information such as voice, and adjusting the information accordingly.

[0161] "Visualization means" refers to devices and methods for displaying pet health information and user emotional history in a way that is easy for the user to understand.

[0162] A "machine learning model" refers to a set of algorithms used to identify pet behavior and vocalization patterns and detect differences from normal behavior.

[0163] This invention provides specific methods necessary to build a system that monitors a pet's health and provides appropriate information based on the user's emotional state. Details are provided below.

[0164] First, the device uses a fixed camera and microphone to collect audio and video information of the pet's living space in real time. This data is temporarily stored on the device, and after unwanted noise is removed, it is sent to the server using a secure protocol. The data is transferred periodically in batches, which reduces the network load.

[0165] The server analyzes the received data using a machine learning model. Specifically, it analyzes the pet's walking patterns, vocalization frequencies, and tones to detect unusual behavior or sounds. Any detected anomalies are saved as flags indicating that the pet may be unwell.

[0166] Simultaneously, the server activates an emotion recognition engine to analyze the user's voice. This engine identifies emotions from the user's voice, determines emotions such as joy, sadness, and surprise, and quantifies their intensity.

[0167] The server detects any anomalies related to the pet and, if it determines the user's emotional state, generates an alert message based on that information. The message is tailored to reassure the user and may say something like, "Your pet is a little upset; please try some activities to help calm them down."

[0168] The user-accessible dashboard visualizes pet health information and user sentiment history accumulated by the server. This visualization shows pet health trends and user sentiment changes over time.

[0169] For example, if a pet barks or barks at an unusual frequency, the device captures the audio, and the server analyzes it. If the server detects an anomaly and further analyzes that the user is experiencing stress, it sends an alert such as, "Your pet may be stressed. You might want to try playing with them."

[0170] An example of a prompt to a generative AI model might be: "I want to design a system that monitors a pet's health in real time. Please tell me specifically how to provide information that takes the user's emotions into consideration."

[0171] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0172] Step 1:

[0173] The device uses a fixed camera and microphone to collect audio and video information of pets in real time. This information is temporarily stored on the device. The input is raw data from the camera and microphone, and the output is filtered, clear audio and video data. During this process, noise reduction technology is used to clean the data.

[0174] Step 2:

[0175] The terminal transfers formatted audio and video data to the server using a secure protocol. The data is sent in batches at regular time intervals to reduce network load. In this configuration, the input is filtered data, and the output is the completion of data transfer to the server.

[0176] Step 3:

[0177] The server runs a machine learning model to analyze the received audio and video data. The input is data sent from the terminal, and the output is the analysis results regarding the pet's health. The server performs calculations to detect anomalies based on walking patterns and vocalization tones. Specifically, the model automatically recognizes abnormal behavioral and vocalization patterns.

[0178] Step 4:

[0179] The server takes in the user's voice data and activates the emotion recognition engine. The input here is the user's voice, and the output is an evaluation of the user's emotional state and its intensity. The server analyzes the voice and determines the type of emotion (e.g., joy, sadness) and its intensity.

[0180] Step 5:

[0181] The server generates alerts based on the pet's analysis results and the user's emotional state. Inputs are abnormal pet data and the user's emotional assessment, while output is a customized alert message for the user. If the pet is exhibiting abnormal behavior and the user is experiencing stress, a reassuring message is generated.

[0182] Step 6:

[0183] The server visualizes pet health data and user emotional history in an easy-to-understand format. This visualization is provided in a dashboard format and can be accessed by users through a browser or mobile app. The input is accumulated data, and the output is a clear visual representation of past health trends and emotional fluctuations. Graphs and charts are used in the visualization, making it possible to track the status of both the pet and the user over time.

[0184] (Application Example 2)

[0185] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0186] For many animal lovers today, properly managing their pets' health and quickly detecting abnormalities is crucial. However, communicating with pets and accurately understanding their emotions is also challenging. This invention aims to solve these problems by monitoring the animal's health and behavior while providing appropriate information tailored to the user's emotional state.

[0187] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0188] In this invention, the server includes information gathering means for collecting animal audio and video information, analysis means for analyzing the animal's behavior and health status based on the collected information, and emotion recognition means for recognizing emotions from the user's voice and adjusting the information provided based on that emotional state. This enables appropriate monitoring of the animal's health status and the provision of information that responds to the user's emotions.

[0189] "Animals" refer to creatures that often live alongside humans, such as dogs and cats.

[0190] "Auditory information" refers to data related to sounds and voices emitted by animals.

[0191] "Visual information" refers to visual data that records the appearance and movements of animals.

[0192] "Information gathering means" is a general term for equipment and technologies used to acquire auditory and visual information about animals.

[0193] "Analysis means" refers to technologies and algorithms used to analyze the health status and behavioral patterns of animals using collected audio and video information.

[0194] A "warning generation means" refers to a mechanism or device that notifies the user in some way when an abnormality is detected in an animal's behavior or health condition.

[0195] "Visualization methods" refer to technologies and methods for displaying animal health status and behavioral data in a way that is easy for users to understand.

[0196] "Emotion recognition means" refers to technologies and devices that recognize and analyze a user's emotional state from their voice or other sources.

[0197] A "machine learning algorithm" is a mathematical method that learns specific patterns and features based on large amounts of data, and then uses that knowledge to analyze new data.

[0198] "Real-time" refers to a technology that processes information with virtually no delay from the moment it is generated.

[0199] This invention is a system that collects animal audio and video information and provides users with appropriate information based on the analysis results. The system is configured as follows:

[0200] First, the terminal is equipped with a fixed video camera and an audio acquisition device. These devices are used to collect animal behavior and sounds in real time. The video and audio data are immediately transmitted to the server.

[0201] The server analyzes the received data using machine learning algorithms. During this process, it analyzes patterns in animal gait and vocalizations to detect abnormalities in health or behavior. Machine learning libraries such as TensorFlow are used for the analysis. If a warning is necessary based on the analysis results, the user is notified through a warning generation system. The content of this notification is adjusted based on the user's emotional state, as determined by their emotion recognition system.

[0202] Furthermore, the server performs emotion recognition based on the user's voice information. Software called EmotionRecognizer is used to detect emotions such as joy, sadness, and surprise. Based on this emotion data, the server provides the user with the most relevant information. For example, if the user is feeling stressed, notifications regarding their pet's health will be as reassuring as possible.

[0203] For example, if a pet shows signs of anxiety in the store, the device immediately sends that information to the server. When the server detects an anomaly and determines that the user is in an agitated state, it sends a tailored message such as, "Your pet seems stressed. We recommend taking a short break to help it relax."

[0204] Examples of prompts to input into a generative AI model are as follows:

[0205] "Observe your pet's behavior and health in real time, and provide advice to help them relax."

[0206] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0207] Step 1:

[0208] The terminal uses a fixed video and audio acquisition device to collect animal audio and video information in real time. The input for this step is audio and video from the animal, and the output is audio and video information as data packets. The terminal then prepares this data to be immediately transmitted to the server.

[0209] Step 2:

[0210] The server receives audio and video information sent from the terminal. The input for this step is audio and video information as data packets, and the output is data converted into a format that can be processed on the server. The server prepares to call machine learning algorithms in order to analyze the data. Specifically, it uses TensorFlow to preprocess the data.

[0211] Step 3:

[0212] The server uses machine learning algorithms to analyze audio and video data. The input for this step is pre-processed audio and video data, and the output is the analysis results of the animal's health status and behavioral patterns. The server checks for anomalies and, if an anomaly is detected, notifies the user through a warning generation mechanism. Specifically, it searches for anomalies based on the animal's walking and vocalization patterns.

[0213] Step 4:

[0214] The server collects the user's voice information and analyzes it using emotion recognition technology. The input for this step is the user's voice information, and the output is the user's emotional state. The server uses EmotionRecognizer to evaluate emotions such as joy, sadness, and surprise. The content of subsequent notifications is adjusted based on the intensity of the emotion.

[0215] Step 5:

[0216] The server considers the analysis results and the user's emotional state to generate an optimized warning message. The input for this step is the animal's analysis results and the user's emotional state, and the output is the adjusted warning message. Specifically, a message like "Your pet is stressed" is adjusted to "Your pet seems a little stressed. Let's provide a place where it can relax."

[0217] Step 6:

[0218] The server sends a pre-configured warning message to the user's device. The input for this step is the generated warning message, and the output is the notification information displayed on the user's device. The user can then take appropriate action regarding the animal based on this notification.

[0219] Example prompts for generative AI models:

[0220] "Observe your pet's behavior and health in real time, and provide advice to help them relax."

[0221] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0222] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0223] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0224] [Second Embodiment]

[0225] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0226] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0227] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0228] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0229] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0230] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0231] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0232] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0233] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0234] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0235] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0236] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0237] This invention provides a system for monitoring a pet's health in real time and immediately notifying the user if an abnormality is detected. The system is implemented as follows:

[0238] Data collection

[0239] First, the device uses a fixed camera and microphone installed in the pet's living space to acquire video and audio data of the pet in real time. This data is designed to comprehensively cover the pet's daily activities.

[0240] Data transmission and analysis

[0241] Data collected by the device is sent to a server via the internet. The server runs machine learning algorithms to analyze the received data. This analysis extracts characteristics of the pet's walking patterns and vocalizations. This allows the system to determine if the pet is behaving unusually or may be experiencing stress.

[0242] Anomaly detection and notification

[0243] Based on the analysis results, the server generates an alert when it detects an anomaly. The alert is sent to the user's mobile device, and the user can open the app to view detailed information. This allows the user to quickly understand their pet's health condition and take necessary action.

[0244] Recording and Visualizing Health Status

[0245] The server stores pet health data over long periods and visualizes the data in a format accessible to users. By comparing current data with past data, users can identify trends in their pet's health and implement preventative health management. This visualization function is provided in an easy-to-understand format using graphs and charts.

[0246] For example, if a pet barks more frequently than usual, the device captures the audio data and sends it to a server. The server analyzes the audio pattern and, if it determines that the sound is an abnormal sound indicating stress, sends an alert to the user stating, "Your pet may be experiencing stress." The user can then check the details in the app, observe their pet's behavior, and take appropriate measures to reassure them if necessary.

[0247] The following describes the processing flow.

[0248] Step 1:

[0249] The device uses a fixed camera and microphone installed in the pet's living space to capture video and audio of the pet in real time. This allows for the accumulation of data on the pet's behavior and vocalizations.

[0250] Step 2:

[0251] The terminal transmits the collected video and audio data to the server via the internet. Here, the data format is standardized to maintain data continuity and accuracy.

[0252] Step 3:

[0253] The server analyzes the received video data frame by frame. Specifically, it uses image recognition technology to identify the pet's silhouette and movements, and to determine any differences from normal behavior patterns.

[0254] Step 4:

[0255] The server performs frequency spectrum analysis on the audio data to extract the characteristics of pet noises. This is then compared to an existing audio database to detect abnormal patterns and frequencies.

[0256] Step 5:

[0257] Based on the analysis results, the server generates an alert if an anomaly is detected. It then prepares to send the alert via push notification or email, according to the user's settings.

[0258] Step 6:

[0259] The server sends the generated alert to the user's terminal. The alert includes details of the detected anomaly and predictions about possible causes.

[0260] Step 7:

[0261] When a user receives an alert, they open the app to view the details. At this stage, the app helps the user take appropriate action quickly by providing recommended actions and health management information.

[0262] Step 8:

[0263] The server stores pet health data chronologically, managing long-term records of their health status. Users can access this data to review past records and understand their pet's health trends.

[0264] (Example 1)

[0265] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0266] Managing a pet's health is a crucial issue for pet owners, requiring them to respond quickly when their pet experiences stress or illness. However, it is difficult for owners to constantly monitor their pet's condition, and abnormalities may go unnoticed for a long time. Traditional methods have not provided an effective solution to this problem.

[0267] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0268] In this invention, the server includes acquisition means for acquiring acoustic and video information of a pet, analysis means for analyzing the pet's behavior and health status based on the acquired acoustic and video information, and notification generation means for notifying the user if an abnormality is detected based on the analysis results. This allows the user to monitor the pet's health status in real time and immediately identify any abnormalities.

[0269] "Means of acquisition" refers to a system for collecting acoustic and visual information about pets.

[0270] "Analysis means" refers to a system for analyzing a pet's behavior and health condition based on acquired acoustic and video information.

[0271] A "notification generation mechanism" refers to a system that informs users of the details of an anomaly detected as a result of analysis.

[0272] "Display means" refers to a system for providing information about a pet's health status in a way that users can verify.

[0273] A "fixed recording device" refers to cameras and other equipment installed to acquire video information of pets in real time.

[0274] "Acoustic equipment" refers to microphones and other devices used to acquire acoustic information about pets in real time.

[0275] "Machine learning techniques" refer to algorithms that extract features and patterns from data to determine a pet's behavior and health status.

[0276] The system of this invention monitors the health status of pets in real time and promptly notifies the user if an abnormality is detected.

[0277] System Configuration

[0278] The terminal uses a fixed camera and sound system installed in the pet's living space to acquire real-time audio and video information of the pet. This data is temporarily stored by the terminal and transmitted to a server via the internet.

[0279] Data Analysis

[0280] The server analyzes the received acoustic and video information using machine learning techniques. Through this analysis, information regarding the behavior and health status of the pet is extracted to detect abnormalities. The server can improve the analysis accuracy by using software libraries such as TensorFlow and PyTorch.

[0281] Notification and Display

[0282] If an abnormality is detected as a result of the analysis, the server uses notification generation means to send an alert to the user's terminal. The user can open the application on their mobile terminal to check the detailed health status of the pet and the content of the abnormality. Past data is accumulated on the server and visualized using interactive graphs and charts, so the user can easily grasp the pet's health trends.

[0283] Specific Example

[0284] As a specific example, consider the case where the pet keeps barking more frequently than usual. The terminal captures the acoustic data and sends it to the server. When the server analyzes the acoustic pattern and determines it as an abnormal sound indicating stress, it sends an alert to the user saying "There is a possibility that the pet is feeling stressed." The user can check the details in the app and observe the pet's condition or take appropriate measures to comfort it.

[0285] Example of Input Prompt Sentences for the Generated AI Model

[0286] "Please explain the mechanism for generating an alert when an abnormal behavior different from the pet's normal behavior is detected."

[0287] The flow of the specific process in Example 1 will be described using FIG. 11.

[0288] Step 1:

[0289] The device uses a fixed camera and audio system to acquire real-time audio and video information of the pet. The camera captures high-resolution video, and the microphone collects ambient sound data. The input is the pet's movements and vocalizations, and the output is digital data of these sounds. This allows for detailed recording of the pet's daily activities.

[0290] Step 2:

[0291] The terminal temporarily stores the acquired audio and video information in memory and then transmits it to the server via the internet. The input is digital data stored in the terminal, which is compressed before transmission. The output is the compressed data transferred to the server. Error checking is performed during this process to maintain data consistency.

[0292] Step 3:

[0293] The server decodes the received audio and video information and prepares it for analysis. The input is compressed digital data, and the decoded raw data is output. Here, data preprocessing such as noise reduction and normalization is performed.

[0294] Step 4:

[0295] The server uses machine learning techniques to analyze raw data. The input is pre-processed data, from which features related to pet behavior patterns and health status are extracted. The output is a judgment result regarding the pet's behavior and health status. Software such as TensorFlow and PyTorch are used for the analysis.

[0296] Step 5:

[0297] If the server detects an anomaly based on the analysis results, it sends an alert to the user's terminal using a notification generation mechanism. The input is the analysis result, and the output is an alert message. The notification is sent in real time, prompting the user to take immediate action.

[0298] Step 6:

[0299] The user opens the app on their mobile device to view the notification. The input is an alert message sent from the server, and the output is detailed information about the pet's health status and any abnormalities. This allows the user to take necessary actions quickly.

[0300] (Application Example 1)

[0301] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0302] Conventional animal health monitoring systems struggle to monitor animals' health in real time, potentially leading to missed abnormalities. Furthermore, they lack the means to prompt human intervention quickly when abnormalities are detected. There is also a need to effectively accumulate detailed data on animals' health over long periods and visualize it in a user-friendly format.

[0303] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0304] In this invention, the server includes data acquisition means for collecting acoustic and visual information to monitor the animal's condition in real time; an analysis mechanism for evaluating the animal's behavioral characteristics and health status based on the collected acoustic and visual information; a warning generation means for generating a notification to the user when an abnormality is detected based on the analysis results; and a display device linkage means for displaying the abnormality notification on a visual information display device worn by a person. This makes it possible to quickly and accurately detect abnormalities in animals and prompt immediate human intervention.

[0305] A "data acquisition means" is a mechanism that collects acoustic and visual information from animals in real time and provides it for necessary analysis.

[0306] The "analysis mechanism" is a mechanism for evaluating the behavioral characteristics and health status of animals based on the collected acoustic and visual information and determining the presence or absence of abnormalities.

[0307] The "warning generation means" is a mechanism for quickly generating a notification to the user when an abnormality is detected based on the analysis result and appropriately transmitting the information.

[0308] The "visualization means" is a device or software for accumulating information on the health status of animals and displaying data in a form that can be easily understood by the user.

[0309] The "display device cooperation means" is an interface means for presenting an abnormality notification to a visual information display device worn by a person so that the user can track it in real time.

[0310] The system for implementing this invention enables real-time monitoring of the health status of animals and rapid notification when an abnormality is detected. The system is composed of the following elements.

[0311] First, the terminal uses fixed imaging devices and acoustic devices installed in the animal's living space to collect acoustic and visual information of the animal in real time. These data are intended to comprehensively cover the animal's living environment.

[0312] Next, the data collected by the terminal is transmitted to the server via the Internet. The server has a powerful analysis mechanism and evaluates the behavioral characteristics and health status of animals based on the collected acoustic and visual information. This process includes data analysis using machine learning algorithms to identify the movement patterns and voice characteristics of animals and detect abnormal states.

[0313] If the server detects an anomaly, it uses a warning generation mechanism to quickly generate a notification to the user, which is then displayed on a visual information display device worn by the user. This allows the user to check the animal's condition in real time and take necessary actions. In addition, the visualization mechanism accumulates data on the animal's health status and displays it to the user in graph and chart format, allowing for a clear understanding of health trends by comparing it with past data.

[0314] For example, if a pet parakeet chirps more frequently than usual, the system captures and analyzes the sound, and if an abnormal pattern is detected, it notifies the user. The user can then check the details through the application and take steps to observe their pet and reassure it.

[0315] Examples of prompts to be input into the generating AI model include, "Design a system that monitors the health status of animals in real time, detects abnormalities, and provides rapid notification."

[0316] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0317] Step 1:

[0318] The terminal uses imaging and acoustic devices installed in the animal's living space to acquire acoustic and visual information of the animal in real time. The input is raw image and sound data collected by the imaging and acoustic devices, and the output is digital data converted into a format that the system can process. In terms of operation, these devices are constantly running, maintaining a state of continuous monitoring of the environment.

[0319] Step 2:

[0320] The terminal transmits the collected digital data to the server via the internet. The input is the digital data obtained in step 1, and the output is confirmation information indicating that the secure data transfer to the server was successful. The data is transmitted securely using security protocols.

[0321] Step 3:

[0322] The server analyzes the received digital data. The input is digital video and audio data transferred from the terminal, and the output is an evaluation result regarding the animal's behavioral characteristics and health status. Machine learning algorithms are used for data analysis to identify the animal's movement patterns and vocal characteristics.

[0323] Step 4:

[0324] Based on the analysis results, the server generates a warning if an anomaly is detected and notifies the user. The input is the evaluation result obtained in step 3, and the output is the warning content depending on whether an anomaly was detected or not. The warning is generated as a digital message and sent to a visual information display device worn by the user.

[0325] Step 5:

[0326] The user receives and confirms warning messages via a visual information display device they wear. The input is the warning message generated in step 4, and the output is the user's informed action. The user can understand the animal's condition in detail in real time and take prompt action as needed.

[0327] Step 6:

[0328] The server stores information on the health status of animals and provides it to users using visualization tools. Input is historical and current analytical data, and output is health trend data visualized in graphs and charts. This helps users understand long-term trends in animal health and facilitates preventative health management.

[0329] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0330] This invention relates to a system for monitoring the health status of pets and providing information tailored to the user's emotional state. This system combines a series of processes—collecting and analyzing pet audio and video data, detecting and notifying of abnormalities based on the results—with an emotion engine that recognizes the user's emotions. The system is implemented as follows:

[0331] Data collection and analysis

[0332] The device uses a fixed camera and microphone to collect video and audio data in real time within the pet's living space. This data is immediately sent to a server, which uses machine learning algorithms to analyze the pet's behavior patterns and health status. For example, it can detect irregular gait or an unusual frequency of barking.

[0333] User emotion recognition by an emotion engine

[0334] The server activates an emotion engine through the user's voice input. This engine identifies emotions such as joy, sadness, and surprise from the user's voice and further evaluates the intensity of those emotions. Based on the emotional state detected by the emotion engine, the system can adjust the information presented to the user.

[0335] Anomaly detection and alert notifications

[0336] If the server detects an abnormality in the pet, it notifies the user of the analysis results. Taking into account the output of the emotion engine, the server optimizes the content and tone of the message included in the alert based on the user's current emotional state. For example, if the user is feeling stressed, the notification regarding the pet's health will be reassuring.

[0337] Visualization of health status and emotions

[0338] The server continuously collects pet health data and user emotional history, and visualizes it in a user-accessible format. This allows users to understand not only their pet's health trends but also changes in their own emotions. This feature provides users with information to help them better interact with their pets.

[0339] For example, if a pet barks at an unusually high frequency, the device captures the audio data and sends it to a server for analysis. If the server detects an anomaly and determines from the user's voice that they are feeling anxious, it will notify the user with a message such as, "Your pet seems a little stressed. Spending some time together might help them feel more at ease." This allows the user to receive information that takes into account not only the pet's condition but also their own emotions.

[0340] The following describes the processing flow.

[0341] Step 1:

[0342] The device uses a fixed camera and microphone to collect video and audio of the space where the pet is located in real time. This data, including the pet's behavior and vocalizations, is continuously recorded.

[0343] Step 2:

[0344] The terminal transmits the collected data to the server via the internet. The data is immediately converted into a format that can be processed for real-time analysis.

[0345] Step 3:

[0346] The server breaks down the video data frame by frame and uses image recognition technology to analyze the pet's movements. It detects the pet's posture and movement patterns and compares them to normal behavioral patterns.

[0347] Step 4:

[0348] The server performs frequency spectrum analysis on the audio data to extract the characteristics of the pet's barks. It then compares the barking patterns with a database of normal barks to detect abnormal patterns.

[0349] Step 5:

[0350] The server analyzes the voice data acquired from the user's terminal using an emotion engine. This engine identifies the user's emotional state, such as joy or anxiety.

[0351] Step 6:

[0352] The server comprehensively evaluates the pet's analysis results and the user's emotional state, and generates an alert if an anomaly is detected. This alert is adjusted to take the user's emotional state into consideration and optimized for appropriate message content.

[0353] Step 7:

[0354] An alert is sent to the user's device, allowing them to open the application and view detailed information about their pet's health. This information is presented in a tone that matches the user's emotional state.

[0355] Step 8:

[0356] The server stores pet health data and user emotional history in a database and provides it to users in a visualized format that they can access. This visualization allows users to comprehensively analyze their pet's health trends and changes in their own emotions.

[0357] (Example 2)

[0358] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0359] Efficiently monitoring a pet's health while providing information that considers the owner's emotional state is a challenging task. Current technology allows for systems that simply detect and notify of health abnormalities in pets, but they cannot appropriately adjust the information content to reflect the user's emotional state. Therefore, there is a need to establish a comprehensive information provision system that enables owners to understand their pet's health and take appropriate action.

[0360] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0361] In this invention, the server includes information gathering means for collecting audio and video information of pets, analysis means for analyzing the pet's behavior and health status based on the collected audio and video information, and emotion recognition means for identifying the user's emotional state and adjusting the content of the information provided according to the identification result. This makes it possible to monitor the pet's health status in real time and provide appropriate information according to the user's emotions.

[0362] "Information gathering means" refers to devices and methods for collecting audio and video information about pets.

[0363] "Analysis means" refers to devices and methods for analyzing a pet's behavior and health condition based on collected audio and video information.

[0364] "Alert generation means" refers to a device or method for notifying the user of information when an anomaly is detected in the analysis results.

[0365] "Emotion recognition means" refers to devices or methods for identifying a user's emotional state from input information such as voice, and adjusting the information accordingly.

[0366] "Visualization means" refers to devices and methods for displaying pet health information and user emotional history in a way that is easy for the user to understand.

[0367] A "machine learning model" refers to a set of algorithms used to identify pet behavior and vocalization patterns and detect differences from normal behavior.

[0368] This invention provides specific methods necessary to build a system that monitors a pet's health and provides appropriate information based on the user's emotional state. Details are provided below.

[0369] First, the device uses a fixed camera and microphone to collect audio and video information of the pet's living space in real time. This data is temporarily stored on the device, and after unwanted noise is removed, it is sent to the server using a secure protocol. The data is transferred periodically in batches, which reduces the network load.

[0370] The server analyzes the received data using a machine learning model. Specifically, it analyzes the pet's walking patterns, vocalization frequencies, and tones to detect unusual behavior or sounds. Any detected anomalies are saved as flags indicating that the pet may be unwell.

[0371] Simultaneously, the server activates an emotion recognition engine to analyze the user's voice. This engine identifies emotions from the user's voice, determines emotions such as joy, sadness, and surprise, and quantifies their intensity.

[0372] The server detects any anomalies related to the pet and, if it determines the user's emotional state, generates an alert message based on that information. The message is tailored to reassure the user and may say something like, "Your pet is a little upset; please try some activities to help calm them down."

[0373] The user-accessible dashboard visualizes pet health information and user sentiment history accumulated by the server. This visualization shows pet health trends and user sentiment changes over time.

[0374] For example, if a pet barks or barks at an unusual frequency, the device captures the audio, and the server analyzes it. If the server detects an anomaly and further analyzes that the user is experiencing stress, it sends an alert such as, "Your pet may be stressed. You might want to try playing with them."

[0375] An example of a prompt to a generative AI model might be: "I want to design a system that monitors a pet's health in real time. Please tell me specifically how to provide information that takes the user's emotions into consideration."

[0376] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0377] Step 1:

[0378] The device uses a fixed camera and microphone to collect audio and video information of pets in real time. This information is temporarily stored on the device. The input is raw data from the camera and microphone, and the output is filtered, clear audio and video data. During this process, noise reduction technology is used to clean the data.

[0379] Step 2:

[0380] The terminal transfers formatted audio and video data to the server using a secure protocol. The data is sent in batches at regular time intervals to reduce network load. In this configuration, the input is filtered data, and the output is the completion of data transfer to the server.

[0381] Step 3:

[0382] The server runs a machine learning model to analyze the received audio and video data. The input is data sent from the terminal, and the output is the analysis results regarding the pet's health. The server performs calculations to detect anomalies based on walking patterns and vocalization tones. Specifically, the model automatically recognizes abnormal behavioral and vocalization patterns.

[0383] Step 4:

[0384] The server takes in the user's voice data and activates the emotion recognition engine. The input here is the user's voice, and the output is an evaluation of the user's emotional state and its intensity. The server analyzes the voice and determines the type of emotion (e.g., joy, sadness) and its intensity.

[0385] Step 5:

[0386] The server generates alerts based on the pet's analysis results and the user's emotional state. Inputs are abnormal pet data and the user's emotional assessment, while output is a customized alert message for the user. If the pet is exhibiting abnormal behavior and the user is experiencing stress, a reassuring message is generated.

[0387] Step 6:

[0388] The server visualizes pet health data and user emotional history in an easy-to-understand format. This visualization is provided in a dashboard format and can be accessed by users through a browser or mobile app. The input is accumulated data, and the output is a clear visual representation of past health trends and emotional fluctuations. Graphs and charts are used in the visualization, making it possible to track the status of both the pet and the user over time.

[0389] (Application Example 2)

[0390] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the smart glasses 214 as the "terminal".

[0391] For many animal lovers today, properly managing their pets' health and quickly detecting abnormalities is crucial. However, communicating with pets and accurately understanding their emotions is also challenging. This invention aims to solve these problems by monitoring the animal's health and behavior while providing appropriate information tailored to the user's emotional state.

[0392] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0393] In this invention, the server includes information gathering means for collecting animal audio and video information, analysis means for analyzing the animal's behavior and health status based on the collected information, and emotion recognition means for recognizing emotions from the user's voice and adjusting the information provided based on that emotional state. This enables appropriate monitoring of the animal's health status and the provision of information that responds to the user's emotions.

[0394] "Animals" refer to creatures that often live alongside humans, such as dogs and cats.

[0395] "Auditory information" refers to data related to sounds and voices emitted by animals.

[0396] "Visual information" refers to visual data that records the appearance and movements of animals.

[0397] "Information gathering means" is a general term for equipment and technologies used to acquire auditory and visual information about animals.

[0398] "Analysis means" refers to technologies and algorithms used to analyze the health status and behavioral patterns of animals using collected audio and video information.

[0399] A "warning generation means" refers to a mechanism or device that notifies the user in some way when an abnormality is detected in an animal's behavior or health condition.

[0400] "Visualization methods" refer to technologies and methods for displaying animal health status and behavioral data in a way that is easy for users to understand.

[0401] "Emotion recognition means" refers to technologies and devices that recognize and analyze a user's emotional state from their voice or other sources.

[0402] A "machine learning algorithm" is a mathematical method that learns specific patterns and features based on large amounts of data, and then uses that knowledge to analyze new data.

[0403] "Real-time" refers to a technology that processes information with virtually no delay from the moment it is generated.

[0404] This invention is a system that collects animal audio and video information and provides users with appropriate information based on the analysis results. The system is configured as follows:

[0405] First, the terminal is equipped with a fixed video camera and an audio acquisition device. These devices are used to collect animal behavior and sounds in real time. The video and audio data are immediately transmitted to the server.

[0406] The server analyzes the received data using machine learning algorithms. During this process, it analyzes patterns in animal gait and vocalizations to detect abnormalities in health or behavior. Machine learning libraries such as TensorFlow are used for the analysis. If a warning is necessary based on the analysis results, the user is notified through a warning generation system. The content of this notification is adjusted based on the user's emotional state, as determined by their emotion recognition system.

[0407] Furthermore, the server performs emotion recognition based on the user's voice information. Software called EmotionRecognizer is used to detect emotions such as joy, sadness, and surprise. Based on this emotion data, the server provides the user with the most relevant information. For example, if the user is feeling stressed, notifications regarding their pet's health will be as reassuring as possible.

[0408] For example, if a pet shows signs of anxiety in the store, the device immediately sends that information to the server. When the server detects an anomaly and determines that the user is in an agitated state, it sends a tailored message such as, "Your pet seems stressed. We recommend taking a short break to help it relax."

[0409] Examples of prompts to input into a generative AI model are as follows:

[0410] "Observe your pet's behavior and health in real time, and provide advice to help them relax."

[0411] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0412] Step 1:

[0413] The terminal uses a fixed video and audio acquisition device to collect animal audio and video information in real time. The input for this step is audio and video from the animal, and the output is audio and video information as data packets. The terminal then prepares this data to be immediately transmitted to the server.

[0414] Step 2:

[0415] The server receives audio and video information sent from the terminal. The input for this step is audio and video information as data packets, and the output is data converted into a format that can be processed on the server. The server prepares to call machine learning algorithms in order to analyze the data. Specifically, it uses TensorFlow to preprocess the data.

[0416] Step 3:

[0417] The server uses machine learning algorithms to analyze audio and video data. The input for this step is pre-processed audio and video data, and the output is the analysis results of the animal's health status and behavioral patterns. The server checks for anomalies and, if an anomaly is detected, notifies the user through a warning generation mechanism. Specifically, it searches for anomalies based on the animal's walking and vocalization patterns.

[0418] Step 4:

[0419] The server collects the user's voice information and analyzes it using emotion recognition technology. The input for this step is the user's voice information, and the output is the user's emotional state. The server uses EmotionRecognizer to evaluate emotions such as joy, sadness, and surprise. The content of subsequent notifications is adjusted based on the intensity of the emotion.

[0420] Step 5:

[0421] The server considers the analysis results and the user's emotional state to generate an optimized warning message. The input for this step is the animal's analysis results and the user's emotional state, and the output is the adjusted warning message. Specifically, a message like "Your pet is stressed" is adjusted to "Your pet seems a little stressed. Let's provide a place where it can relax."

[0422] Step 6:

[0423] The server sends a pre-configured warning message to the user's device. The input for this step is the generated warning message, and the output is the notification information displayed on the user's device. The user can then take appropriate action regarding the animal based on this notification.

[0424] Example prompts for generative AI models:

[0425] "Observe your pet's behavior and health in real time, and provide advice to help them relax."

[0426] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0427] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0428] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0429] [Third Embodiment]

[0430] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0431] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0432] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0433] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0434] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0435] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0436] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0437] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0438] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0439] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0440] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0441] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0442] This invention provides a system for monitoring a pet's health in real time and immediately notifying the user if an abnormality is detected. The system is implemented as follows:

[0443] Data collection

[0444] First, the device uses a fixed camera and microphone installed in the pet's living space to acquire video and audio data of the pet in real time. This data is designed to comprehensively cover the pet's daily activities.

[0445] Data transmission and analysis

[0446] Data collected by the device is sent to a server via the internet. The server runs machine learning algorithms to analyze the received data. This analysis extracts characteristics of the pet's walking patterns and vocalizations. This allows the system to determine if the pet is behaving unusually or may be experiencing stress.

[0447] Anomaly detection and notification

[0448] Based on the analysis results, the server generates an alert when it detects an anomaly. The alert is sent to the user's mobile device, and the user can open the app to view detailed information. This allows the user to quickly understand their pet's health condition and take necessary action.

[0449] Recording and Visualizing Health Status

[0450] The server stores pet health data over long periods and visualizes the data in a format accessible to users. By comparing current data with past data, users can identify trends in their pet's health and implement preventative health management. This visualization function is provided in an easy-to-understand format using graphs and charts.

[0451] For example, if a pet barks more frequently than usual, the device captures the audio data and sends it to a server. The server analyzes the audio pattern and, if it determines that the sound is an abnormal sound indicating stress, sends an alert to the user stating, "Your pet may be experiencing stress." The user can then check the details in the app, observe their pet's behavior, and take appropriate measures to reassure them if necessary.

[0452] The following describes the processing flow.

[0453] Step 1:

[0454] The device uses a fixed camera and microphone installed in the pet's living space to capture video and audio of the pet in real time. This allows for the accumulation of data on the pet's behavior and vocalizations.

[0455] Step 2:

[0456] The terminal transmits the collected video and audio data to the server via the internet. Here, the data format is standardized to maintain data continuity and accuracy.

[0457] Step 3:

[0458] The server analyzes the received video data frame by frame. Specifically, it uses image recognition technology to identify the pet's silhouette and movements, and to determine any differences from normal behavior patterns.

[0459] Step 4:

[0460] The server performs frequency spectrum analysis on the audio data to extract the characteristics of pet noises. This is then compared to an existing audio database to detect abnormal patterns and frequencies.

[0461] Step 5:

[0462] Based on the analysis results, the server generates an alert if an anomaly is detected. It then prepares to send the alert via push notification or email, according to the user's settings.

[0463] Step 6:

[0464] The server sends the generated alert to the user's terminal. The alert includes details of the detected anomaly and predictions about possible causes.

[0465] Step 7:

[0466] When a user receives an alert, they open the app to view the details. At this stage, the app helps the user take appropriate action quickly by providing recommended actions and health management information.

[0467] Step 8:

[0468] The server stores pet health data chronologically, managing long-term records of their health status. Users can access this data to review past records and understand their pet's health trends.

[0469] (Example 1)

[0470] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0471] Managing a pet's health is a crucial issue for pet owners, requiring them to respond quickly when their pet experiences stress or illness. However, it is difficult for owners to constantly monitor their pet's condition, and abnormalities may go unnoticed for a long time. Traditional methods have not provided an effective solution to this problem.

[0472] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0473] In this invention, the server includes acquisition means for acquiring acoustic and video information of a pet, analysis means for analyzing the pet's behavior and health status based on the acquired acoustic and video information, and notification generation means for notifying the user if an abnormality is detected based on the analysis results. This allows the user to monitor the pet's health status in real time and immediately identify any abnormalities.

[0474] "Means of acquisition" refers to a system for collecting acoustic and visual information about pets.

[0475] "Analysis means" refers to a system for analyzing a pet's behavior and health condition based on acquired acoustic and video information.

[0476] A "notification generation mechanism" refers to a system that informs users of the details of an anomaly detected as a result of analysis.

[0477] "Display means" refers to a system for providing information about a pet's health status in a way that users can verify.

[0478] A "fixed recording device" refers to cameras and other equipment installed to acquire video information of pets in real time.

[0479] "Acoustic equipment" refers to microphones and other devices used to acquire acoustic information about pets in real time.

[0480] "Machine learning techniques" refer to algorithms that extract features and patterns from data to determine a pet's behavior and health status.

[0481] The system of this invention monitors the health status of pets in real time and promptly notifies the user if an abnormality is detected.

[0482] System Configuration

[0483] The terminal uses a fixed camera and sound system installed in the pet's living space to acquire real-time audio and video information of the pet. This data is temporarily stored by the terminal and transmitted to a server via the internet.

[0484] Data Analysis

[0485] The server analyzes the received audio and video information using machine learning techniques. This analysis extracts information about the pet's behavior and health status, and detects anomalies. The server can improve the accuracy of its analysis by using software libraries such as TensorFlow and PyTorch.

[0486] Notifications and displays

[0487] If an anomaly is detected during the analysis, the server will send an alert to the user's device using a notification generation mechanism. The user can then open the application on their mobile device to check the pet's detailed health status and the nature of the anomaly. Historical data is stored on the server and visualized using interactive graphs and charts, allowing users to easily understand their pet's health trends.

[0488] Specific example

[0489] As a concrete example, consider a case where a pet barks more frequently than usual. The device captures the sound data and sends it to a server. The server analyzes the sound pattern and, if it determines that the sound is an abnormal sound indicating stress, sends an alert to the user saying, "Your pet may be stressed." The user can then check the details in the app, observe their pet's behavior, and take appropriate measures to reassure them.

[0490] Examples of input prompts for a generative AI model

[0491] "Please explain the mechanism that generates an alert when unusual behavior in a pet is detected."

[0492] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0493] Step 1:

[0494] The device uses a fixed camera and audio system to acquire real-time audio and video information of the pet. The camera captures high-resolution video, and the microphone collects ambient sound data. The input is the pet's movements and vocalizations, and the output is digital data of these sounds. This allows for detailed recording of the pet's daily activities.

[0495] Step 2:

[0496] The terminal temporarily stores the acquired audio and video information in memory and then transmits it to the server via the internet. The input is digital data stored in the terminal, which is compressed before transmission. The output is the compressed data transferred to the server. Error checking is performed during this process to maintain data consistency.

[0497] Step 3:

[0498] The server decodes the received audio and video information and prepares it for analysis. The input is compressed digital data, and the decoded raw data is output. Here, data preprocessing such as noise reduction and normalization is performed.

[0499] Step 4:

[0500] The server uses machine learning techniques to analyze raw data. The input is pre-processed data, from which features related to pet behavior patterns and health status are extracted. The output is a judgment result regarding the pet's behavior and health status. Software such as TensorFlow and PyTorch are used for the analysis.

[0501] Step 5:

[0502] If the server detects an anomaly based on the analysis results, it sends an alert to the user's terminal using a notification generation mechanism. The input is the analysis result, and the output is an alert message. The notification is sent in real time, prompting the user to take immediate action.

[0503] Step 6:

[0504] The user opens the app on their mobile device to view the notification. The input is an alert message sent from the server, and the output is detailed information about the pet's health status and any abnormalities. This allows the user to take necessary actions quickly.

[0505] (Application Example 1)

[0506] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0507] Conventional animal health monitoring systems struggle to monitor animals' health in real time, potentially leading to missed abnormalities. Furthermore, they lack the means to prompt human intervention quickly when abnormalities are detected. There is also a need to effectively accumulate detailed data on animals' health over long periods and visualize it in a user-friendly format.

[0508] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0509] In this invention, the server includes data acquisition means for collecting acoustic and visual information to monitor the animal's condition in real time; an analysis mechanism for evaluating the animal's behavioral characteristics and health status based on the collected acoustic and visual information; a warning generation means for generating a notification to the user when an abnormality is detected based on the analysis results; and a display device linkage means for displaying the abnormality notification on a visual information display device worn by a person. This makes it possible to quickly and accurately detect abnormalities in animals and prompt immediate human intervention.

[0510] A "data acquisition means" is a mechanism that collects acoustic and visual information from animals in real time and provides it for necessary analysis.

[0511] An "analysis mechanism" is a system that evaluates the behavioral characteristics and health status of animals based on collected acoustic and visual information, and determines whether or not there are any abnormalities.

[0512] A "warning generation mechanism" is a system that, based on analysis results, quickly generates notifications to users when an anomaly is detected and appropriately transmits that information.

[0513] "Visualization means" refers to a device or software for storing information about an animal's health status and displaying the data in a format that is easily understandable to users.

[0514] The "display device linkage means" is an interface means that presents abnormality notifications to a visual information display device worn by a person, enabling the user to track the information in real time.

[0515] The system implementing this invention enables real-time monitoring of an animal's health status and rapid notification if an abnormality is detected. The system consists of the following elements:

[0516] First, the terminal uses fixed imaging and acoustic devices installed in the animal's living space to collect acoustic and visual information about the animal in real time. The aim of this data is to comprehensively cover the animal's living environment.

[0517] Next, the data collected by the terminal is transmitted to a server via the internet. The server has a powerful analytical mechanism that evaluates the animal's behavioral characteristics and health status based on the collected acoustic and visual information. This process involves using machine learning algorithms to identify the animal's movement patterns and vocal characteristics and to analyze data to detect any abnormal conditions.

[0518] If the server detects an anomaly, it uses a warning generation mechanism to quickly generate a notification to the user, which is then displayed on a visual information display device worn by the user. This allows the user to check the animal's condition in real time and take necessary actions. In addition, the visualization mechanism accumulates data on the animal's health status and displays it to the user in graph and chart format, allowing for a clear understanding of health trends by comparing it with past data.

[0519] For example, if a pet parakeet chirps more frequently than usual, the system captures and analyzes the sound, and if an abnormal pattern is detected, it notifies the user. The user can then check the details through the application and take steps to observe their pet and reassure it.

[0520] Examples of prompts to be input into the generating AI model include, "Design a system that monitors the health status of animals in real time, detects abnormalities, and provides rapid notification."

[0521] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0522] Step 1:

[0523] The terminal uses imaging and acoustic devices installed in the animal's living space to acquire acoustic and visual information of the animal in real time. The input is raw image and sound data collected by the imaging and acoustic devices, and the output is digital data converted into a format that the system can process. In terms of operation, these devices are constantly running, maintaining a state of continuous monitoring of the environment.

[0524] Step 2:

[0525] The terminal transmits the collected digital data to the server via the internet. The input is the digital data obtained in step 1, and the output is confirmation information indicating that the secure data transfer to the server was successful. The data is transmitted securely using security protocols.

[0526] Step 3:

[0527] The server analyzes the received digital data. The input is digital video and audio data transferred from the terminal, and the output is an evaluation result regarding the animal's behavioral characteristics and health status. Machine learning algorithms are used for data analysis to identify the animal's movement patterns and vocal characteristics.

[0528] Step 4:

[0529] Based on the analysis results, the server generates a warning if an anomaly is detected and notifies the user. The input is the evaluation result obtained in step 3, and the output is the warning content depending on whether an anomaly was detected or not. The warning is generated as a digital message and sent to a visual information display device worn by the user.

[0530] Step 5:

[0531] The user receives and confirms warning messages via a visual information display device they wear. The input is the warning message generated in step 4, and the output is the user's informed action. The user can understand the animal's condition in detail in real time and take prompt action as needed.

[0532] Step 6:

[0533] The server stores information on the health status of animals and provides it to users using visualization tools. Input is historical and current analytical data, and output is health trend data visualized in graphs and charts. This helps users understand long-term trends in animal health and facilitates preventative health management.

[0534] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0535] This invention relates to a system for monitoring the health status of pets and providing information tailored to the user's emotional state. This system combines a series of processes—collecting and analyzing pet audio and video data, detecting and notifying of abnormalities based on the results—with an emotion engine that recognizes the user's emotions. The system is implemented as follows:

[0536] Data collection and analysis

[0537] The device uses a fixed camera and microphone to collect video and audio data in real time within the pet's living space. This data is immediately sent to a server, which uses machine learning algorithms to analyze the pet's behavior patterns and health status. For example, it can detect irregular gait or an unusual frequency of barking.

[0538] User emotion recognition by an emotion engine

[0539] The server activates an emotion engine through the user's voice input. This engine identifies emotions such as joy, sadness, and surprise from the user's voice and further evaluates the intensity of those emotions. Based on the emotional state detected by the emotion engine, the system can adjust the information presented to the user.

[0540] Anomaly detection and alert notifications

[0541] If the server detects an abnormality in the pet, it notifies the user of the analysis results. Taking into account the output of the emotion engine, the server optimizes the content and tone of the message included in the alert based on the user's current emotional state. For example, if the user is feeling stressed, the notification regarding the pet's health will be reassuring.

[0542] Visualization of health status and emotions

[0543] The server continuously collects pet health data and user emotional history, and visualizes it in a user-accessible format. This allows users to understand not only their pet's health trends but also changes in their own emotions. This feature provides users with information to help them better interact with their pets.

[0544] For example, if a pet barks at an unusually high frequency, the device captures the audio data and sends it to a server for analysis. If the server detects an anomaly and determines from the user's voice that they are feeling anxious, it will notify the user with a message such as, "Your pet seems a little stressed. Spending some time together might help them feel more at ease." This allows the user to receive information that takes into account not only the pet's condition but also their own emotions.

[0545] The following describes the processing flow.

[0546] Step 1:

[0547] The device uses a fixed camera and microphone to collect video and audio of the space where the pet is located in real time. This data, including the pet's behavior and vocalizations, is continuously recorded.

[0548] Step 2:

[0549] The terminal transmits the collected data to the server via the internet. The data is immediately converted into a format that can be processed for real-time analysis.

[0550] Step 3:

[0551] The server breaks down the video data frame by frame and uses image recognition technology to analyze the pet's movements. It detects the pet's posture and movement patterns and compares them to normal behavioral patterns.

[0552] Step 4:

[0553] The server performs frequency spectrum analysis on the audio data to extract the characteristics of the pet's barks. It then compares the barking patterns with a database of normal barks to detect abnormal patterns.

[0554] Step 5:

[0555] The server analyzes the voice data acquired from the user's terminal using an emotion engine. This engine identifies the user's emotional state, such as joy or anxiety.

[0556] Step 6:

[0557] The server comprehensively evaluates the pet's analysis results and the user's emotional state, and generates an alert if an anomaly is detected. This alert is adjusted to take the user's emotional state into consideration and optimized for appropriate message content.

[0558] Step 7:

[0559] An alert is sent to the user's device, allowing them to open the application and view detailed information about their pet's health. This information is presented in a tone that matches the user's emotional state.

[0560] Step 8:

[0561] The server stores pet health data and user emotional history in a database and provides it to users in a visualized format that they can access. This visualization allows users to comprehensively analyze their pet's health trends and changes in their own emotions.

[0562] (Example 2)

[0563] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0564] Efficiently monitoring a pet's health while providing information that considers the owner's emotional state is a challenging task. Current technology allows for systems that simply detect and notify of health abnormalities in pets, but they cannot appropriately adjust the information content to reflect the user's emotional state. Therefore, there is a need to establish a comprehensive information provision system that enables owners to understand their pet's health and take appropriate action.

[0565] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0566] In this invention, the server includes information gathering means for collecting audio and video information of pets, analysis means for analyzing the pet's behavior and health status based on the collected audio and video information, and emotion recognition means for identifying the user's emotional state and adjusting the content of the information provided according to the identification result. This makes it possible to monitor the pet's health status in real time and provide appropriate information according to the user's emotions.

[0567] "Information gathering means" refers to devices and methods for collecting audio and video information about pets.

[0568] "Analysis means" refers to devices and methods for analyzing a pet's behavior and health condition based on collected audio and video information.

[0569] "Alert generation means" refers to a device or method for notifying the user of information when an anomaly is detected in the analysis results.

[0570] "Emotion recognition means" refers to devices or methods for identifying a user's emotional state from input information such as voice, and adjusting the information accordingly.

[0571] "Visualization means" refers to devices and methods for displaying pet health information and user emotional history in a way that is easy for the user to understand.

[0572] A "machine learning model" refers to a set of algorithms used to identify pet behavior and vocalization patterns and detect differences from normal behavior.

[0573] This invention provides specific methods necessary to build a system that monitors a pet's health and provides appropriate information based on the user's emotional state. Details are provided below.

[0574] First, the device uses a fixed camera and microphone to collect audio and video information of the pet's living space in real time. This data is temporarily stored on the device, and after unwanted noise is removed, it is sent to the server using a secure protocol. The data is transferred periodically in batches, which reduces the network load.

[0575] The server analyzes the received data using a machine learning model. Specifically, it analyzes the pet's walking patterns, vocalization frequencies, and tones to detect unusual behavior or sounds. Any detected anomalies are saved as flags indicating that the pet may be unwell.

[0576] Simultaneously, the server activates an emotion recognition engine to analyze the user's voice. This engine identifies emotions from the user's voice, determines emotions such as joy, sadness, and surprise, and quantifies their intensity.

[0577] The server detects any anomalies related to the pet and, if it determines the user's emotional state, generates an alert message based on that information. The message is tailored to reassure the user and may say something like, "Your pet is a little upset; please try some activities to help calm them down."

[0578] The user-accessible dashboard visualizes pet health information and user sentiment history accumulated by the server. This visualization shows pet health trends and user sentiment changes over time.

[0579] For example, if a pet barks or barks at an unusual frequency, the device captures the audio, and the server analyzes it. If the server detects an anomaly and further analyzes that the user is experiencing stress, it sends an alert such as, "Your pet may be stressed. You might want to try playing with them."

[0580] An example of a prompt to a generative AI model might be: "I want to design a system that monitors a pet's health in real time. Please tell me specifically how to provide information that takes the user's emotions into consideration."

[0581] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0582] Step 1:

[0583] The device uses a fixed camera and microphone to collect audio and video information of pets in real time. This information is temporarily stored on the device. The input is raw data from the camera and microphone, and the output is filtered, clear audio and video data. During this process, noise reduction technology is used to clean the data.

[0584] Step 2:

[0585] The terminal transfers formatted audio and video data to the server using a secure protocol. The data is sent in batches at regular time intervals to reduce network load. In this configuration, the input is filtered data, and the output is the completion of data transfer to the server.

[0586] Step 3:

[0587] The server runs a machine learning model to analyze the received audio and video data. The input is data sent from the terminal, and the output is the analysis results regarding the pet's health. The server performs calculations to detect anomalies based on walking patterns and vocalization tones. Specifically, the model automatically recognizes abnormal behavioral and vocalization patterns.

[0588] Step 4:

[0589] The server takes in the user's voice data and activates the emotion recognition engine. The input here is the user's voice, and the output is an evaluation of the user's emotional state and its intensity. The server analyzes the voice and determines the type of emotion (e.g., joy, sadness) and its intensity.

[0590] Step 5:

[0591] The server generates alerts based on the pet's analysis results and the user's emotional state. Inputs are abnormal pet data and the user's emotional assessment, while output is a customized alert message for the user. If the pet is exhibiting abnormal behavior and the user is experiencing stress, a reassuring message is generated.

[0592] Step 6:

[0593] The server visualizes pet health data and user emotional history in an easy-to-understand format. This visualization is provided in a dashboard format and can be accessed by users through a browser or mobile app. The input is accumulated data, and the output is a clear visual representation of past health trends and emotional fluctuations. Graphs and charts are used in the visualization, making it possible to track the status of both the pet and the user over time.

[0594] (Application Example 2)

[0595] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0596] For many animal lovers today, properly managing their pets' health and quickly detecting abnormalities is crucial. However, communicating with pets and accurately understanding their emotions is also challenging. This invention aims to solve these problems by monitoring the animal's health and behavior while providing appropriate information tailored to the user's emotional state.

[0597] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0598] In this invention, the server includes information gathering means for collecting animal audio and video information, analysis means for analyzing the animal's behavior and health status based on the collected information, and emotion recognition means for recognizing emotions from the user's voice and adjusting the information provided based on that emotional state. This enables appropriate monitoring of the animal's health status and the provision of information that responds to the user's emotions.

[0599] "Animals" refer to creatures that often live alongside humans, such as dogs and cats.

[0600] "Auditory information" refers to data related to sounds and voices emitted by animals.

[0601] "Visual information" refers to visual data that records the appearance and movements of animals.

[0602] "Information gathering means" is a general term for equipment and technologies used to acquire auditory and visual information about animals.

[0603] "Analysis means" refers to technologies and algorithms used to analyze the health status and behavioral patterns of animals using collected audio and video information.

[0604] A "warning generation means" refers to a mechanism or device that notifies the user in some way when an abnormality is detected in an animal's behavior or health condition.

[0605] "Visualization methods" refer to technologies and methods for displaying animal health status and behavioral data in a way that is easy for users to understand.

[0606] "Emotion recognition means" refers to technologies and devices that recognize and analyze a user's emotional state from their voice or other sources.

[0607] A "machine learning algorithm" is a mathematical method that learns specific patterns and features based on large amounts of data, and then uses that knowledge to analyze new data.

[0608] "Real-time" refers to a technology that processes information with virtually no delay from the moment it is generated.

[0609] This invention is a system that collects animal audio and video information and provides users with appropriate information based on the analysis results. The system is configured as follows:

[0610] First, the terminal is equipped with a fixed video camera and an audio acquisition device. These devices are used to collect animal behavior and sounds in real time. The video and audio data are immediately transmitted to the server.

[0611] The server analyzes the received data using machine learning algorithms. During this process, it analyzes patterns in animal gait and vocalizations to detect abnormalities in health or behavior. Machine learning libraries such as TensorFlow are used for the analysis. If a warning is necessary based on the analysis results, the user is notified through a warning generation system. The content of this notification is adjusted based on the user's emotional state, as determined by their emotion recognition system.

[0612] Furthermore, the server performs emotion recognition based on the user's voice information. Software called EmotionRecognizer is used to detect emotions such as joy, sadness, and surprise. Based on this emotion data, the server provides the user with the most relevant information. For example, if the user is feeling stressed, notifications regarding their pet's health will be as reassuring as possible.

[0613] For example, if a pet shows signs of anxiety in the store, the device immediately sends that information to the server. When the server detects an anomaly and determines that the user is in an agitated state, it sends a tailored message such as, "Your pet seems stressed. We recommend taking a short break to help it relax."

[0614] Examples of prompts to input into a generative AI model are as follows:

[0615] "Observe your pet's behavior and health in real time, and provide advice to help them relax."

[0616] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0617] Step 1:

[0618] The terminal uses a fixed video and audio acquisition device to collect animal audio and video information in real time. The input for this step is audio and video from the animal, and the output is audio and video information as data packets. The terminal then prepares this data to be immediately transmitted to the server.

[0619] Step 2:

[0620] The server receives audio and video information sent from the terminal. The input for this step is audio and video information as data packets, and the output is data converted into a format that can be processed on the server. The server prepares to call machine learning algorithms in order to analyze the data. Specifically, it uses TensorFlow to preprocess the data.

[0621] Step 3:

[0622] The server uses machine learning algorithms to analyze audio and video data. The input for this step is pre-processed audio and video data, and the output is the analysis results of the animal's health status and behavioral patterns. The server checks for anomalies and, if an anomaly is detected, notifies the user through a warning generation mechanism. Specifically, it searches for anomalies based on the animal's walking and vocalization patterns.

[0623] Step 4:

[0624] The server collects the user's voice information and analyzes it using emotion recognition technology. The input for this step is the user's voice information, and the output is the user's emotional state. The server uses EmotionRecognizer to evaluate emotions such as joy, sadness, and surprise. The content of subsequent notifications is adjusted based on the intensity of the emotion.

[0625] Step 5:

[0626] The server considers the analysis results and the user's emotional state to generate an optimized warning message. The input for this step is the animal's analysis results and the user's emotional state, and the output is the adjusted warning message. Specifically, a message like "Your pet is stressed" is adjusted to "Your pet seems a little stressed. Let's provide a place where it can relax."

[0627] Step 6:

[0628] The server sends a pre-configured warning message to the user's device. The input for this step is the generated warning message, and the output is the notification information displayed on the user's device. The user can then take appropriate action regarding the animal based on this notification.

[0629] Example prompts for generative AI models:

[0630] "Observe your pet's behavior and health in real time, and provide advice to help them relax."

[0631] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0632] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0633] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0634] [Fourth Embodiment]

[0635] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0636] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0637] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0638] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0639] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0640] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0641] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0642] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0643] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0644] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0645] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0646] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0647] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0648] This invention provides a system for monitoring a pet's health in real time and immediately notifying the user if an abnormality is detected. The system is implemented as follows:

[0649] Data collection

[0650] First, the device uses a fixed camera and microphone installed in the pet's living space to acquire video and audio data of the pet in real time. This data is designed to comprehensively cover the pet's daily activities.

[0651] Data transmission and analysis

[0652] Data collected by the device is sent to a server via the internet. The server runs machine learning algorithms to analyze the received data. This analysis extracts characteristics of the pet's walking patterns and vocalizations. This allows the system to determine if the pet is behaving unusually or may be experiencing stress.

[0653] Anomaly detection and notification

[0654] Based on the analysis results, the server generates an alert when it detects an anomaly. The alert is sent to the user's mobile device, and the user can open the app to view detailed information. This allows the user to quickly understand their pet's health condition and take necessary action.

[0655] Recording and Visualizing Health Status

[0656] The server stores pet health data over long periods and visualizes the data in a format accessible to users. By comparing current data with past data, users can identify trends in their pet's health and implement preventative health management. This visualization function is provided in an easy-to-understand format using graphs and charts.

[0657] For example, if a pet barks more frequently than usual, the device captures the audio data and sends it to a server. The server analyzes the audio pattern and, if it determines that the sound is an abnormal sound indicating stress, sends an alert to the user stating, "Your pet may be experiencing stress." The user can then check the details in the app, observe their pet's behavior, and take appropriate measures to reassure them if necessary.

[0658] The following describes the processing flow.

[0659] Step 1:

[0660] The device uses a fixed camera and microphone installed in the pet's living space to capture video and audio of the pet in real time. This allows for the accumulation of data on the pet's behavior and vocalizations.

[0661] Step 2:

[0662] The terminal transmits the collected video and audio data to the server via the internet. Here, the data format is standardized to maintain data continuity and accuracy.

[0663] Step 3:

[0664] The server analyzes the received video data frame by frame. Specifically, it uses image recognition technology to identify the pet's silhouette and movements, and to determine any differences from normal behavior patterns.

[0665] Step 4:

[0666] The server performs frequency spectrum analysis on the audio data to extract the characteristics of pet noises. This is then compared to an existing audio database to detect abnormal patterns and frequencies.

[0667] Step 5:

[0668] Based on the analysis results, the server generates an alert if an anomaly is detected. It then prepares to send the alert via push notification or email, according to the user's settings.

[0669] Step 6:

[0670] The server sends the generated alert to the user's terminal. The alert includes details of the detected anomaly and predictions about possible causes.

[0671] Step 7:

[0672] When a user receives an alert, they open the app to view the details. At this stage, the app helps the user take appropriate action quickly by providing recommended actions and health management information.

[0673] Step 8:

[0674] The server stores pet health data chronologically, managing long-term records of their health status. Users can access this data to review past records and understand their pet's health trends.

[0675] (Example 1)

[0676] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0677] Managing a pet's health is a crucial issue for pet owners, requiring them to respond quickly when their pet experiences stress or illness. However, it is difficult for owners to constantly monitor their pet's condition, and abnormalities may go unnoticed for a long time. Traditional methods have not provided an effective solution to this problem.

[0678] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0679] In this invention, the server includes acquisition means for acquiring acoustic and video information of a pet, analysis means for analyzing the pet's behavior and health status based on the acquired acoustic and video information, and notification generation means for notifying the user if an abnormality is detected based on the analysis results. This allows the user to monitor the pet's health status in real time and immediately identify any abnormalities.

[0680] "Means of acquisition" refers to a system for collecting acoustic and visual information about pets.

[0681] "Analysis means" refers to a system for analyzing a pet's behavior and health condition based on acquired acoustic and video information.

[0682] A "notification generation mechanism" refers to a system that informs users of the details of an anomaly detected as a result of analysis.

[0683] "Display means" refers to a system for providing information about a pet's health status in a way that users can verify.

[0684] A "fixed recording device" refers to cameras and other equipment installed to acquire video information of pets in real time.

[0685] "Acoustic equipment" refers to microphones and other devices used to acquire acoustic information about pets in real time.

[0686] "Machine learning techniques" refer to algorithms that extract features and patterns from data to determine a pet's behavior and health status.

[0687] The system of this invention monitors the health status of pets in real time and promptly notifies the user if an abnormality is detected.

[0688] System Configuration

[0689] The terminal uses a fixed camera and sound system installed in the pet's living space to acquire real-time audio and video information of the pet. This data is temporarily stored by the terminal and transmitted to a server via the internet.

[0690] Data Analysis

[0691] The server analyzes the received audio and video information using machine learning techniques. This analysis extracts information about the pet's behavior and health status, and detects anomalies. The server can improve the accuracy of its analysis by using software libraries such as TensorFlow and PyTorch.

[0692] Notifications and displays

[0693] If an anomaly is detected during the analysis, the server will send an alert to the user's device using a notification generation mechanism. The user can then open the application on their mobile device to check the pet's detailed health status and the nature of the anomaly. Historical data is stored on the server and visualized using interactive graphs and charts, allowing users to easily understand their pet's health trends.

[0694] Specific example

[0695] As a concrete example, consider a case where a pet barks more frequently than usual. The device captures the sound data and sends it to a server. The server analyzes the sound pattern and, if it determines that the sound is an abnormal sound indicating stress, sends an alert to the user saying, "Your pet may be stressed." The user can then check the details in the app, observe their pet's behavior, and take appropriate measures to reassure them.

[0696] Examples of input prompts for a generative AI model

[0697] "Please explain the mechanism that generates an alert when unusual behavior in a pet is detected."

[0698] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0699] Step 1:

[0700] The device uses a fixed camera and audio system to acquire real-time audio and video information of the pet. The camera captures high-resolution video, and the microphone collects ambient sound data. The input is the pet's movements and vocalizations, and the output is digital data of these sounds. This allows for detailed recording of the pet's daily activities.

[0701] Step 2:

[0702] The terminal temporarily stores the acquired audio and video information in memory and then transmits it to the server via the internet. The input is digital data stored in the terminal, which is compressed before transmission. The output is the compressed data transferred to the server. Error checking is performed during this process to maintain data consistency.

[0703] Step 3:

[0704] The server decodes the received audio and video information and prepares it for analysis. The input is compressed digital data, and the decoded raw data is output. Here, data preprocessing such as noise reduction and normalization is performed.

[0705] Step 4:

[0706] The server uses machine learning techniques to analyze raw data. The input is pre-processed data, from which features related to pet behavior patterns and health status are extracted. The output is a judgment result regarding the pet's behavior and health status. Software such as TensorFlow and PyTorch are used for the analysis.

[0707] Step 5:

[0708] If the server detects an anomaly based on the analysis results, it sends an alert to the user's terminal using a notification generation mechanism. The input is the analysis result, and the output is an alert message. The notification is sent in real time, prompting the user to take immediate action.

[0709] Step 6:

[0710] The user opens the app on their mobile device to view the notification. The input is an alert message sent from the server, and the output is detailed information about the pet's health status and any abnormalities. This allows the user to take necessary actions quickly.

[0711] (Application Example 1)

[0712] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0713] Conventional animal health monitoring systems struggle to monitor animals' health in real time, potentially leading to missed abnormalities. Furthermore, they lack the means to prompt human intervention quickly when abnormalities are detected. There is also a need to effectively accumulate detailed data on animals' health over long periods and visualize it in a user-friendly format.

[0714] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0715] In this invention, the server includes data acquisition means for collecting acoustic and visual information to monitor the animal's condition in real time; an analysis mechanism for evaluating the animal's behavioral characteristics and health status based on the collected acoustic and visual information; a warning generation means for generating a notification to the user when an abnormality is detected based on the analysis results; and a display device linkage means for displaying the abnormality notification on a visual information display device worn by a person. This makes it possible to quickly and accurately detect abnormalities in animals and prompt immediate human intervention.

[0716] A "data acquisition means" is a mechanism that collects acoustic and visual information from animals in real time and provides it for necessary analysis.

[0717] An "analysis mechanism" is a system that evaluates the behavioral characteristics and health status of animals based on collected acoustic and visual information, and determines whether or not there are any abnormalities.

[0718] A "warning generation mechanism" is a system that, based on analysis results, quickly generates notifications to users when an anomaly is detected and appropriately transmits that information.

[0719] "Visualization means" refers to a device or software for storing information about an animal's health status and displaying the data in a format that is easily understandable to users.

[0720] The "display device linkage means" is an interface means that presents abnormality notifications to a visual information display device worn by a person, enabling the user to track the information in real time.

[0721] The system implementing this invention enables real-time monitoring of an animal's health status and rapid notification if an abnormality is detected. The system consists of the following elements:

[0722] First, the terminal uses fixed imaging and acoustic devices installed in the animal's living space to collect acoustic and visual information about the animal in real time. The aim of this data is to comprehensively cover the animal's living environment.

[0723] Next, the data collected by the terminal is transmitted to a server via the internet. The server has a powerful analytical mechanism that evaluates the animal's behavioral characteristics and health status based on the collected acoustic and visual information. This process involves using machine learning algorithms to identify the animal's movement patterns and vocal characteristics and to analyze data to detect any abnormal conditions.

[0724] If the server detects an anomaly, it uses a warning generation mechanism to quickly generate a notification to the user, which is then displayed on a visual information display device worn by the user. This allows the user to check the animal's condition in real time and take necessary actions. In addition, the visualization mechanism accumulates data on the animal's health status and displays it to the user in graph and chart format, allowing for a clear understanding of health trends by comparing it with past data.

[0725] For example, if a pet parakeet chirps more frequently than usual, the system captures and analyzes the sound, and if an abnormal pattern is detected, it notifies the user. The user can then check the details through the application and take steps to observe their pet and reassure it.

[0726] Examples of prompts to be input into the generating AI model include, "Design a system that monitors the health status of animals in real time, detects abnormalities, and provides rapid notification."

[0727] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0728] Step 1:

[0729] The terminal uses imaging and acoustic devices installed in the animal's living space to acquire acoustic and visual information of the animal in real time. The input is raw image and sound data collected by the imaging and acoustic devices, and the output is digital data converted into a format that the system can process. In terms of operation, these devices are constantly running, maintaining a state of continuous monitoring of the environment.

[0730] Step 2:

[0731] The terminal transmits the collected digital data to the server via the internet. The input is the digital data obtained in step 1, and the output is confirmation information indicating that the secure data transfer to the server was successful. The data is transmitted securely using security protocols.

[0732] Step 3:

[0733] The server analyzes the received digital data. The input is digital video and audio data transferred from the terminal, and the output is an evaluation result regarding the animal's behavioral characteristics and health status. Machine learning algorithms are used for data analysis to identify the animal's movement patterns and vocal characteristics.

[0734] Step 4:

[0735] Based on the analysis results, the server generates a warning if an anomaly is detected and notifies the user. The input is the evaluation result obtained in step 3, and the output is the warning content depending on whether an anomaly was detected or not. The warning is generated as a digital message and sent to a visual information display device worn by the user.

[0736] Step 5:

[0737] The user receives and confirms warning messages via a visual information display device they wear. The input is the warning message generated in step 4, and the output is the user's informed action. The user can understand the animal's condition in detail in real time and take prompt action as needed.

[0738] Step 6:

[0739] The server stores information on the health status of animals and provides it to users using visualization tools. Input is historical and current analytical data, and output is health trend data visualized in graphs and charts. This helps users understand long-term trends in animal health and facilitates preventative health management.

[0740] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0741] This invention relates to a system for monitoring the health status of pets and providing information tailored to the user's emotional state. This system combines a series of processes—collecting and analyzing pet audio and video data, detecting and notifying of abnormalities based on the results—with an emotion engine that recognizes the user's emotions. The system is implemented as follows:

[0742] Data collection and analysis

[0743] The device uses a fixed camera and microphone to collect video and audio data in real time within the pet's living space. This data is immediately sent to a server, which uses machine learning algorithms to analyze the pet's behavior patterns and health status. For example, it can detect irregular gait or an unusual frequency of barking.

[0744] User emotion recognition by an emotion engine

[0745] The server activates an emotion engine through the user's voice input. This engine identifies emotions such as joy, sadness, and surprise from the user's voice and further evaluates the intensity of those emotions. Based on the emotional state detected by the emotion engine, the system can adjust the information presented to the user.

[0746] Anomaly detection and alert notifications

[0747] If the server detects an abnormality in the pet, it notifies the user of the analysis results. Taking into account the output of the emotion engine, the server optimizes the content and tone of the message included in the alert based on the user's current emotional state. For example, if the user is feeling stressed, the notification regarding the pet's health will be reassuring.

[0748] Visualization of health status and emotions

[0749] The server continuously collects pet health data and user emotional history, and visualizes it in a user-accessible format. This allows users to understand not only their pet's health trends but also changes in their own emotions. This feature provides users with information to help them better interact with their pets.

[0750] For example, if a pet barks at an unusually high frequency, the device captures the audio data and sends it to a server for analysis. If the server detects an anomaly and determines from the user's voice that they are feeling anxious, it will notify the user with a message such as, "Your pet seems a little stressed. Spending some time together might help them feel more at ease." This allows the user to receive information that takes into account not only the pet's condition but also their own emotions.

[0751] The following describes the processing flow.

[0752] Step 1:

[0753] The device uses a fixed camera and microphone to collect video and audio of the space where the pet is located in real time. This data, including the pet's behavior and vocalizations, is continuously recorded.

[0754] Step 2:

[0755] The terminal transmits the collected data to the server via the internet. The data is immediately converted into a format that can be processed for real-time analysis.

[0756] Step 3:

[0757] The server breaks down the video data frame by frame and uses image recognition technology to analyze the pet's movements. It detects the pet's posture and movement patterns and compares them to normal behavioral patterns.

[0758] Step 4:

[0759] The server performs frequency spectrum analysis on the audio data to extract the characteristics of the pet's barks. It then compares the barking patterns with a database of normal barks to detect abnormal patterns.

[0760] Step 5:

[0761] The server analyzes the voice data acquired from the user's terminal using an emotion engine. This engine identifies the user's emotional state, such as joy or anxiety.

[0762] Step 6:

[0763] The server comprehensively evaluates the pet's analysis results and the user's emotional state, and generates an alert if an anomaly is detected. This alert is adjusted to take the user's emotional state into consideration and optimized for appropriate message content.

[0764] Step 7:

[0765] An alert is sent to the user's device, allowing them to open the application and view detailed information about their pet's health. This information is presented in a tone that matches the user's emotional state.

[0766] Step 8:

[0767] The server stores pet health data and user emotional history in a database and provides it to users in a visualized format that they can access. This visualization allows users to comprehensively analyze their pet's health trends and changes in their own emotions.

[0768] (Example 2)

[0769] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0770] Efficiently monitoring a pet's health while providing information that considers the owner's emotional state is a challenging task. Current technology allows for systems that simply detect and notify of health abnormalities in pets, but they cannot appropriately adjust the information content to reflect the user's emotional state. Therefore, there is a need to establish a comprehensive information provision system that enables owners to understand their pet's health and take appropriate action.

[0771] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0772] In this invention, the server includes information gathering means for collecting audio and video information of pets, analysis means for analyzing the pet's behavior and health status based on the collected audio and video information, and emotion recognition means for identifying the user's emotional state and adjusting the content of the information provided according to the identification result. This makes it possible to monitor the pet's health status in real time and provide appropriate information according to the user's emotions.

[0773] "Information gathering means" refers to devices and methods for collecting audio and video information about pets.

[0774] "Analysis means" refers to devices and methods for analyzing a pet's behavior and health condition based on collected audio and video information.

[0775] "Alert generation means" refers to a device or method for notifying the user of information when an anomaly is detected in the analysis results.

[0776] "Emotion recognition means" refers to devices or methods for identifying a user's emotional state from input information such as voice, and adjusting the information accordingly.

[0777] "Visualization means" refers to devices and methods for displaying pet health information and user emotional history in a way that is easy for the user to understand.

[0778] A "machine learning model" refers to a set of algorithms used to identify pet behavior and vocalization patterns and detect differences from normal behavior.

[0779] This invention provides specific methods necessary to build a system that monitors a pet's health and provides appropriate information based on the user's emotional state. Details are provided below.

[0780] First, the device uses a fixed camera and microphone to collect audio and video information of the pet's living space in real time. This data is temporarily stored on the device, and after unwanted noise is removed, it is sent to the server using a secure protocol. The data is transferred periodically in batches, which reduces the network load.

[0781] The server analyzes the received data using a machine learning model. Specifically, it analyzes the pet's walking patterns, vocalization frequencies, and tones to detect unusual behavior or sounds. Any detected anomalies are saved as flags indicating that the pet may be unwell.

[0782] Simultaneously, the server activates an emotion recognition engine to analyze the user's voice. This engine identifies emotions from the user's voice, determines emotions such as joy, sadness, and surprise, and quantifies their intensity.

[0783] The server detects any anomalies related to the pet and, if it determines the user's emotional state, generates an alert message based on that information. The message is tailored to reassure the user and may say something like, "Your pet is a little upset; please try some activities to help calm them down."

[0784] The user-accessible dashboard visualizes pet health information and user sentiment history accumulated by the server. This visualization shows pet health trends and user sentiment changes over time.

[0785] For example, if a pet barks or barks at an unusual frequency, the device captures the audio, and the server analyzes it. If the server detects an anomaly and further analyzes that the user is experiencing stress, it sends an alert such as, "Your pet may be stressed. You might want to try playing with them."

[0786] An example of a prompt to a generative AI model might be: "I want to design a system that monitors a pet's health in real time. Please tell me specifically how to provide information that takes the user's emotions into consideration."

[0787] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0788] Step 1:

[0789] The device uses a fixed camera and microphone to collect audio and video information of pets in real time. This information is temporarily stored on the device. The input is raw data from the camera and microphone, and the output is filtered, clear audio and video data. During this process, noise reduction technology is used to clean the data.

[0790] Step 2:

[0791] The terminal transfers formatted audio and video data to the server using a secure protocol. The data is sent in batches at regular time intervals to reduce network load. In this configuration, the input is filtered data, and the output is the completion of data transfer to the server.

[0792] Step 3:

[0793] The server runs a machine learning model to analyze the received audio and video data. The input is data sent from the terminal, and the output is the analysis results regarding the pet's health. The server performs calculations to detect anomalies based on walking patterns and vocalization tones. Specifically, the model automatically recognizes abnormal behavioral and vocalization patterns.

[0794] Step 4:

[0795] The server takes in the user's voice data and activates the emotion recognition engine. The input here is the user's voice, and the output is an evaluation of the user's emotional state and its intensity. The server analyzes the voice and determines the type of emotion (e.g., joy, sadness) and its intensity.

[0796] Step 5:

[0797] The server generates alerts based on the pet's analysis results and the user's emotional state. Inputs are abnormal pet data and the user's emotional assessment, while output is a customized alert message for the user. If the pet is exhibiting abnormal behavior and the user is experiencing stress, a reassuring message is generated.

[0798] Step 6:

[0799] The server visualizes pet health data and user emotional history in an easy-to-understand format. This visualization is provided in a dashboard format and can be accessed by users through a browser or mobile app. The input is accumulated data, and the output is a clear visual representation of past health trends and emotional fluctuations. Graphs and charts are used in the visualization, making it possible to track the status of both the pet and the user over time.

[0800] (Application Example 2)

[0801] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0802] For many animal lovers today, properly managing their pets' health and quickly detecting abnormalities is crucial. However, communicating with pets and accurately understanding their emotions is also challenging. This invention aims to solve these problems by monitoring the animal's health and behavior while providing appropriate information tailored to the user's emotional state.

[0803] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0804] In this invention, the server includes information gathering means for collecting animal audio and video information, analysis means for analyzing the animal's behavior and health status based on the collected information, and emotion recognition means for recognizing emotions from the user's voice and adjusting the information provided based on that emotional state. This enables appropriate monitoring of the animal's health status and the provision of information that responds to the user's emotions.

[0805] "Animals" refer to creatures that often live alongside humans, such as dogs and cats.

[0806] "Auditory information" refers to data related to sounds and voices emitted by animals.

[0807] "Visual information" refers to visual data that records the appearance and movements of animals.

[0808] "Information gathering means" is a general term for equipment and technologies used to acquire auditory and visual information about animals.

[0809] "Analysis means" refers to technologies and algorithms used to analyze the health status and behavioral patterns of animals using collected audio and video information.

[0810] A "warning generation means" refers to a mechanism or device that notifies the user in some way when an abnormality is detected in an animal's behavior or health condition.

[0811] "Visualization methods" refer to technologies and methods for displaying animal health status and behavioral data in a way that is easy for users to understand.

[0812] "Emotion recognition means" refers to technologies and devices that recognize and analyze a user's emotional state from their voice or other sources.

[0813] A "machine learning algorithm" is a mathematical method that learns specific patterns and features based on large amounts of data, and then uses that knowledge to analyze new data.

[0814] "Real-time" refers to a technology that processes information with virtually no delay from the moment it is generated.

[0815] This invention is a system that collects animal audio and video information and provides users with appropriate information based on the analysis results. The system is configured as follows:

[0816] First, the terminal is equipped with a fixed video camera and an audio acquisition device. These devices are used to collect animal behavior and sounds in real time. The video and audio data are immediately transmitted to the server.

[0817] The server analyzes the received data using machine learning algorithms. During this process, it analyzes patterns in animal gait and vocalizations to detect abnormalities in health or behavior. Machine learning libraries such as TensorFlow are used for the analysis. If a warning is necessary based on the analysis results, the user is notified through a warning generation system. The content of this notification is adjusted based on the user's emotional state, as determined by their emotion recognition system.

[0818] Furthermore, the server performs emotion recognition based on the user's voice information. Software called EmotionRecognizer is used to detect emotions such as joy, sadness, and surprise. Based on this emotion data, the server provides the user with the most relevant information. For example, if the user is feeling stressed, notifications regarding their pet's health will be as reassuring as possible.

[0819] For example, if a pet shows signs of anxiety in the store, the device immediately sends that information to the server. When the server detects an anomaly and determines that the user is in an agitated state, it sends a tailored message such as, "Your pet seems stressed. We recommend taking a short break to help it relax."

[0820] Examples of prompts to input into a generative AI model are as follows:

[0821] "Observe your pet's behavior and health in real time, and provide advice to help them relax."

[0822] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0823] Step 1:

[0824] The terminal uses a fixed video and audio acquisition device to collect animal audio and video information in real time. The input for this step is audio and video from the animal, and the output is audio and video information as data packets. The terminal then prepares this data to be immediately transmitted to the server.

[0825] Step 2:

[0826] The server receives audio and video information sent from the terminal. The input for this step is audio and video information as data packets, and the output is data converted into a format that can be processed on the server. The server prepares to call machine learning algorithms in order to analyze the data. Specifically, it uses TensorFlow to preprocess the data.

[0827] Step 3:

[0828] The server uses machine learning algorithms to analyze audio and video data. The input for this step is pre-processed audio and video data, and the output is the analysis results of the animal's health status and behavioral patterns. The server checks for anomalies and, if an anomaly is detected, notifies the user through a warning generation mechanism. Specifically, it searches for anomalies based on the animal's walking and vocalization patterns.

[0829] Step 4:

[0830] The server collects the user's voice information and analyzes it using emotion recognition technology. The input for this step is the user's voice information, and the output is the user's emotional state. The server uses EmotionRecognizer to evaluate emotions such as joy, sadness, and surprise. The content of subsequent notifications is adjusted based on the intensity of the emotion.

[0831] Step 5:

[0832] The server considers the analysis results and the user's emotional state to generate an optimized warning message. The input for this step is the animal's analysis results and the user's emotional state, and the output is the adjusted warning message. Specifically, a message like "Your pet is stressed" is adjusted to "Your pet seems a little stressed. Let's provide a place where it can relax."

[0833] Step 6:

[0834] The server sends a pre-configured warning message to the user's device. The input for this step is the generated warning message, and the output is the notification information displayed on the user's device. The user can then take appropriate action regarding the animal based on this notification.

[0835] Example prompts for generative AI models:

[0836] "Observe your pet's behavior and health in real time, and provide advice to help them relax."

[0837] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0838] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0839] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0840] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0841] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0842] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0843] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0844] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0845] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0846] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0847] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0848] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0849] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0850] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0851] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0852] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0853] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0854] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0855] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0856] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0857] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted as being incorporated by reference.

[0858] The following is further disclosed regarding the embodiments described above.

[0859] (Claim 1)

[0860] A data collection means for collecting audio and video data of pets,

[0861] An analytical means for analyzing the behavior and health status of pets based on collected audio and video data,

[0862] An alert generation means that notifies the user when an anomaly is detected based on the analysis results,

[0863] A means of accumulating data on the health status of pets and displaying it in a way that users can check,

[0864] A system that includes this.

[0865] (Claim 2)

[0866] The system according to claim 1, which uses a fixed camera and microphone to acquire audio and video data of a pet in real time.

[0867] (Claim 3)

[0868] The system according to claim 1, wherein the analysis means uses a machine learning algorithm to identify the pet's walking and barking patterns and detect differences from normal behavior.

[0869] "Example 1"

[0870] (Claim 1)

[0871] A means for acquiring acoustic and video information of a pet,

[0872] An analysis means for analyzing the behavior and health status of a pet based on acquired acoustic and video information,

[0873] A notification generation means that notifies the user when an anomaly is detected based on the analysis results,

[0874] A means of storing information about a pet's health status and displaying it in a format that users can review,

[0875] A system that includes this.

[0876] (Claim 2)

[0877] The system according to claim 1, which uses a fixed camera and an acoustic device to acquire acoustic and video information of a pet in real time.

[0878] (Claim 3)

[0879] The system according to claim 1, wherein the analysis means uses machine learning techniques to identify the movement and vocalization patterns of a pet and detects differences from normal behavior.

[0880] "Application Example 1"

[0881] (Claim 1)

[0882] A data acquisition means for collecting acoustic and visual information to monitor the animal's condition in real time,

[0883] An analytical mechanism for evaluating the behavioral characteristics and health status of animals based on collected acoustic and visual information,

[0884] A warning generation means that generates a notification to the user when an anomaly is detected based on the analysis results,

[0885] A visualization means for accumulating information related to the health status of animals and displaying it in a form accessible to users,

[0886] A means for coordinating a display device to present an abnormality notification to a visual information display device worn by a person,

[0887] A system that includes this.

[0888] (Claim 2)

[0889] The system according to claim 1, which acquires acoustic and visual information of an animal in real time using a fixed imaging device and an acoustic device.

[0890] (Claim 3)

[0891] The system according to claim 1, wherein the analysis mechanism uses machine learning techniques to identify the movement patterns and vocal characteristics of animals and detects differences from the normal state.

[0892] "Example 2 of combining an emotion engine"

[0893] (Claim 1)

[0894] Information gathering means for collecting audio and video information of pets,

[0895] An analysis means for analyzing the behavior and health status of pets based on collected audio and video information,

[0896] An alert generation means that notifies the user when an anomaly is detected based on the analysis results,

[0897] An emotion recognition means that identifies the user's emotional state and adjusts the content of information provided according to the identification result,

[0898] A means of visualizing pet health information and user emotional history, and displaying it in a format that the user can review.

[0899] A system that includes this.

[0900] (Claim 2)

[0901] The system according to claim 1, which uses a fixed camera and an audio input device to acquire audio and video information of a pet in real time.

[0902] (Claim 3)

[0903] The system according to claim 1, wherein the analysis means uses a machine learning model to identify the movement and vocalization patterns of a pet and detects differences from normal behavior.

[0904] "Application example 2 of combining emotional engines"

[0905] (Claim 1)

[0906] Information gathering means for collecting animal sound and video information,

[0907] An analytical means for analyzing the behavior and health status of animals based on collected audio and video information,

[0908] A warning generation means that notifies the user when an anomaly is detected based on the analysis results,

[0909] A means of accumulating information on the health status of animals and displaying it in a form that users can check,

[0910] An emotion recognition means that recognizes emotions from the user's voice and adjusts the information provided based on that emotional state,

[0911] A system that includes this.

[0912] (Claim 2)

[0913] The system according to claim 1, which acquires animal audio and video information in real time using a fixed video device and an audio acquisition device.

[0914] (Claim 3)

[0915] The system according to claim 1, wherein the analysis means uses a machine learning algorithm to identify the walking and vocalization patterns of animals, detects differences from normal conditions, and further adjusts the notification content based on the user's emotions. [Explanation of Symbols]

[0916] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A data collection means for collecting audio and video data of pets, An analytical means for analyzing the behavior and health status of pets based on collected audio and video data, An alert generation means that notifies the user when an anomaly is detected based on the analysis results, A means of accumulating data on the health status of pets and displaying it in a way that users can check, A system that includes this.

2. The system according to claim 1, which uses a fixed camera and microphone to acquire audio and video data of a pet in real time.

3. The system according to claim 1, wherein the analysis means uses a machine learning algorithm to identify the walking and barking patterns of a pet and detects differences from normal behavior.

Images